Sguil: Difference between revisions

From ICO wiki
Jump to navigationJump to search
(Created page with "User talk:Lphanvan int power(long base, long exponent) { int counter; int result = 1; for (counter = 0; counter < exponent; counter++) result *= base; ret...")
 
 
(135 intermediate revisions by one other user not shown)
Line 1: Line 1:
User talk:Lphanvan
[[File:Sguil_logo_h.png‎‎|320px|right|text-top]]
[[File:Sguil_main.png‎‎|320px|right|text-top]]


int power(long base, long exponent) {


  int counter;
Author: Kustas Kurval
  int result = 1;
  for (counter = 0; counter < exponent; counter++)
      result *= base;
  return result;


}
Cyber Security Engineering C11


Written 06.06.2016


/* When processor enters the function body the arguments are already placed in registers r0=5 (base), r1=3 (exponent) */


cmp r1, #0 /* Compare exponent to 0 */ mov r2, #1 /* Place constant 1 in register r2,
=Introduction=


                                    this corresponds to result = 1 in C code */
This tutorial was made to make an introduction to Sguil. Sguil (pronounced sgweel) is built by network security analysts for network security analysts.
It is a collection of free software components for Network Security Monitoring (NSM) and event driven analysis of [https://en.wikipedia.org/wiki/Intrusion_prevention_system IDS] alerts. Sguil's main component is an intuitive GUI that provides access to realtime events, session data, and raw packet captures. Sguil facilitates the practice of Network Security Monitoring and event driven analysis.


ble .L2 /* Exponent was not less than 0, so no jump to L2 mov r3, #0 /* Place constant 0 in register r3,
The Sguil client is written in [https://en.wikipedia.org/wiki/Tcl tcl] / [https://en.wikipedia.org/wiki/Tk_(software) tk] and can be run on any operating system that supports [https://en.wikipedia.org/wiki/Tcl tcl] / [https://en.wikipedia.org/wiki/Tk_(software) tk] (including Linux, *BSD, Solaris, MacOS, and Win32).


                                    this corresponds to variable counter */
It is provided by [https://en.wikipedia.org/wiki/Q_Public_License Q Public License]


add r3, r3, #1 /* Perform r3 = 0 + 1 which results in 1 being stored to r3
Sguil integrates alert data from Snort, session data from SANCP, and full content data from a second instance of Snort running in packet logger mode.


                                    this corresponds to first invocation of counter++ in C code */
In this introduction I will be covering Sguil in [https://en.wikipedia.org/wiki/Xubuntu Xbuntu]. You will need to know basic Linux syntax and terminology also some terminology concerning overall [https://en.wikipedia.org/wiki/Intrusion_prevention_system intrusion detection and prevention systems (IDPS)] and overall basic networking.


cmp r3, r1 /* Compare counter (1 in this case) to exponent (3), this will be used by bne instruction below */ mul r2, r0, r2 /* Perform r2 = r0 * r2 which results in 1 * 5 = 5 being placed in r2
==Software architecture==
A sguil system is composed of a single sguil server and an arbitrary number of sguil network sensors. The sensors perform all the security monitoring tasks and feed information back to the server on a regular basis. The server coordinates this information, stores it in a database and communicates with sguil clients running on administrators' desktop machines. It can also issue requests for specific information from the sensors.


                                    this corresponds to first invocation of result *= base in C code */
Each sensor monitors a single network link (although you can have multiple sensors on one physical machine). They collect several different types of information:


bne .L3 /* The comparison resulted in counter being not equal to exponent, so we jump back to L3
                                            |<---------> Sensor #1
            Client  ==>  Server <=========> |<---------> Sensor #2
                                            |<---------> Sensor #3


                                    this corresponds to first invocation of counter < exponent in C code */
# Snort monitors the link for security events, and logs them to a file on the local disk.
# Barnyard takes events from the snort log file and sends them to the sensor agent, which inserts them into database running on the sguil server in near real-time
# A separate instance of snort logs the full content of all network packets to the local disk (this typically requires a large separate data partition)
# SANCP records TCP/IP sessions and forwards them to the database on the sguil server
# The sguil agent also listens for commands from the sguil server. These commands are typically requests for packet data previously logged by Snort.


add r3, r3, #1 /* Perform r3 = 1 + 1 which results in 2 being stored to r3
=Contents=
Since Sguil is dependant on many other types of software to gather, facilitate, store ,decode and analyze I will be using [https://en.wikipedia.org/wiki/Xubuntu Xbuntu] based distribution [https://security-onion-solutions.github.io/security-onion/ Security Onion] which saves massive amount of time to set up the entire environment. [https://security-onion-solutions.github.io/security-onion/ Security Onion] has all this and more build in and is able to quickly configure which software to tie to Sguil.
==Dependencies==
===Hardware===
The hardware necessary to run sguil depends entirely upon the amount of network traffic you plan to monitor, how much of it you store and how long you store it.  The public demonstration server at demo.sguil.net gets by with a 133MHz Pentium system, though for production systems you'll want something beefier.


                                    this corresponds to second invocation of counter++ in C code */
For the sguil server, try to find something with a lot of memory and a fast disk.  The MySQL database will need a lot of RAM for caching results from its queries, and a fast disk will speed up the access time while doing sequential searches.


cmp r3, r1 /* Compare counter (2 in this case) to exponent (3), this will be used by bne instruction below */ mul r2, r0, r2 /* Perform r2 = r0 * r2 which results 5 * 5 = 25 being placed in r2
You may find it helpful to place the MySQL storage on a separate disk partition, or even a separate disk. It doesn't matter where you mount this storage area, but for convenience, I'll assume that that you're going to mount it on the same directory path you're using for the sensor data directory (see below), referred to in this document as ''$NSM''.


                                    this corresponds to second invocation of result *= base in C code */
The following table is a useful guideline for a good server hardware configuration.  Feel free to adjust it up or down depending on the size of your installation.


bne .L3 /* The comparison resulted in counter being not equal to exponent, so we jump back to L3
{| border="1" align="center"
|+ Recommended Server Hardware
! CPU !! RAM !! Disk Storage
|-
| 3.0GHz || 2GB || 150GB (Consider a RAID5 array for speed & reliability)
|}


                                    this corresponds to second invocation of counter < exponent in C code */
===Software===
Tools that usually make up Sguil
{| class="wikitable"
|-
! Tool !! Purpose
|-
| [[MySQL]] 4.x or 5.x  || Data storage and retrieval
|-
| [[Snort (software)|Snort]] 2.x / [[Suricata (software)|Suricata]]  || Intrusion detection alerts, scan detection, packet logging
|-
| Barnyard / Barnyard2  || Decodes IDS alerts and sends them to sguil
|-
| SANCP  || TCP/IP session records
|-
| [[Tcpflow]] || Extract an ASCII dump of a given TCP session
|-
| [[p0f]]  || Operating system fingerprinting
|-
| [[tcpdump]] || Extracts individual sessions from packet logs 
|-
| [[Wireshark]]  || Packet analysis tool (used to be called Ethereal)
|}


add r3, r3, #1 /* Perform r3 = 2 + 1 which results in 3 being stored to r3
==Setup==
*  First, review the [https://github.com/Security-Onion-Solutions/security-onion/wiki/Hardware Hardware Requirements] page.
*  Review the [https://github.com/Security-Onion-Solutions/security-onion/wiki/Security-Onion-14.04-Release-Notes Release Notes] page.
*  [https://github.com/Security-Onion-Solutions/security-onion/blob/master/Verify_ISO.md Download and verify our Security Onion ISO image].
* Boot the ISO image and select the Install option.


                                    this corresponds to third invocation of counter++ in C code */
I Installed this on Oracle Virutalbox as a 64 bit Ubuntu operating system, with 4GB of memory and a single processor. I set the network adapter as bridged with promiscuous mode allowed. This ensured that I am able to capture network traffic from the host machine
*  Follow the prompts in the Xubuntu installer. If prompted with an encrypt home folder or encrypt partition option, DO NOT enable this feature. If asked about *  *  *  automatic updates, DO NOT enable automatic updates. Reboot into your new installation. Login using the username/password you specified during installation.
*  Verify that you have Internet connectivity. If necessary, configure your proxy settings.
*  [https://github.com/Security-Onion-Solutions/security-onion/wiki/Upgrade Install updates and reboot].
*  Double-click the Setup icon on the desktop. The Setup wizard will walk you through configuring /etc/network/interfaces and will then reboot.
* After rebooting, log back in and start the Setup wizard again. It will detect that you have already configured /etc/network/interfaces and will walk you through the rest of the configuration. When prompted for Evaluation Mode or Production Mode, choose Evaluation Mode.


cmp r3, r1 /* Compare counter (3 in this case) to exponent (3), this will be used by bne instruction below */ mul r2, r0, r2 /* Perform r2 = r0 * r2 which results 25 * 5 = 125 being placed in r2
Security Onion usually expects at least two networking interfaces. One for monitoring the other for management. Since I only had access to a single interface on the virtual machine I set it as management. I used the static IP address 192.168.1.111 with regular /24 [https://en.wikipedia.org/wiki/Subnetwork subnet mask] for ease of use. After this I was prompted for a gateway address and [https://en.wikipedia.org/wiki/Domain_Name_System DNS] server.
* Once you've completed the Setup wizard, use the Desktop icons to login to Sguil.


                                    this corresponds to third invocation of result *= base in C code */
==Post Installation==
Verify services are running:
sudo service nsm status


bne .L3 /* The comparison resulted in counter being equal to exponent, so we DO NOT jump back to L3 */
If any services are not running, try starting them:
sudo service nsm start


mov r0, r2 /* Copy register r2 contents (125) to register r0 */ bx lr /* Jump back to caller */ /* Function returns with 125 placed in r0 this is where caller function should expect the return value */ /* The other registers will still hold whatever values were left there: r1 = 3, r2 = 125, r3 = 3 */  
====Tuning / Miscellaneous====
Storage abstractions


 What is a block device?
*  Are you monitoring network traffic that has VLAN tags? If so, take a look at our [https://github.com/Security-Onion-Solutions/security-onion/wiki/VLAN-Traffic VLAN] page.
*  If you’re monitoring IP address ranges other than private RFC1918 address space (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12), you should update your sensor configuration with the correct IP ranges. Sensor configuration files can be found in <code>/etc/nsm/$HOSTNAME-$INTERFACE/</code>. Modify either <code>snort.conf</code> or <code>suricata.yaml</code> (depending on which IDS engine you chose during <code>sosetup</code>) and update the <code>HOME_NET</code> variable. Also update the home_nets variable in <code>prads.conf</code>. Then update Bro’s network configuration in <code>/opt/bro/etc/networks.cfg</code>. Restart the sensor processes:
sudo nsm_sensor_ps-restart
* If you have Internet access, create an IDS alert by typing the following at a terminal:
<code>curl http://testmyids.com</code>
* As of securityonion-setup - 20120912-0ubuntu0securityonion201, Setup now defaults to only opening port 22 in the firewall. If you need to connect OSSEC agents, syslog devices, or analyst VMs, you can run the new <code>so-allow</code> utility which will walk you through creating firewall rules to allow these devices to connect. For more information, please see the [https://github.com/Security-Onion-Solutions/security-onion/wiki/Firewall firewall] page.
* Full-time analysts should install Security Onion in a VM on their workstation (run through the Ubuntu installer, but do not run our Setup wizard). This gives you a local copy of Wireshark, NetworkMiner, and our customized Sguil client. Launch the Sguil client and connect to the IP/hostname of your production Sguil sensor (you may need to run so-allow as described in the previous step). This allows you to investigate pcaps without fear of impacting your production server/sensors. To change the resolution of your Security Onion VM, install the Virtual Tools for your virtualization solution or use xrandr. For a list of available screen resolutions, simply execute “xrandr”. To set the screen resolution (replace W and H with the actual Width and Height desired):
<code>xrandr -s WxH</code>
* Login to Sguil and review your IDS alerts. Squert and ELSA can be accessed by visiting https://server/ for additional in-depth analysis.
* Run the following to see how your sensor is coping with the load. You should check this on a daily basis to make sure your sensor is not dropping packets. Consider adding it to a cronjob and having it emailed to you (see the “configure email” link below).
<code>sudo sostat | less</code>
* Please note that any IDS/NSM system needs to be tuned for the network it’s monitoring. Please see [https://github.com/Security-Onion-Solutions/security-onion/wiki/ManagingAlerts ManagingAlerts]. You should only run the signatures you really care about.
* Also note that you should be looking at and categorizing events every day with the goal being to categorize all events every day. Even if you don’t use the Sguil console for your primary analysis, you need to log into it periodically and F8 old events to keep the real time queue from getting too big. Neglecting to do so will result in database/Sguil issues as the number of uncategorized events continues to increase on a daily basis. Please see the [http://nsmwiki.org/Sguil_Client Sguil client page on NSMwiki].
* On the server running the Sguil database, set the <code>DAYSTOKEEP</code> variable in <code>/etc/nsm/securityonion.conf</code> to however many days you want to keep in your archive. The default is 30, but you may need to adjust it based on your organization’s detection/response policy and your available disk space.
* If you enabled [https://github.com/Security-Onion-Solutions/security-onion/wiki/http_agent http_agent], you should tune it using http_agent.conf. If you're running ELSA, you already have all the Bro HTTP logs available there, so you might want to disable http_agent to avoid duplicating those logs in the Sguil database:
<code># Terminate the running http_agent</code>


 In computing (specifically data transmission and data storage), a block, sometimes called a physical record, is a sequence of bytes or bits, usually containing some whole number of records, having a maximum length, a block size.[1] Data thus structured are said to be blocked. The process of putting data into blocks is called blocking, while deblocking is the process of extracting data from blocks. Blocked data is normally stored in a data buffer and read or written a whole block at a time.
<code>sudo nsm_sensor_ps-stop --only-http-agent</code>


 What is logical block addressing and what are the benefits compared to older cylinder-head-sector addressing method in terms of harddisks?
<code># Disable http_agent</code>


Logical block addressing (LBA) is a common scheme used for specifying the location of blocks of data stored on computer storage devices, generally secondary storage systems such as hard disk drives. LBA is a particularly simple linear addressing scheme; blocks are located by an integer index, with the first block being LBA 0, the second LBA 1, and so on. Cylinder-head-sector, also known as CHS, is an early method for giving addresses to each physical block of data on a hard disk drive. In the case of floppy drives, for which the same exact diskette medium can be truly low-level formatted to different capacities, this is still true.
<code>sudo sed -i 's|HTTP_AGENT_ENABLED="yes"|HTTP_AGENT_ENABLED="no"|g' /etc/nsm//sensor.conf</code>
* [https://github.com/Security-Onion-Solutions/security-onion/wiki/DisablingProcesses Disable any unneeded sensor processes]
* Tune the number of PF_RING instances for Snort/Suricata and Bro: [https://github.com/Security-Onion-Solutions/security-onion/wiki/PF_RING PF_RING]
* Optional:* exclude unnecessary traffic from your monitoring using [https://github.com/Security-Onion-Solutions/security-onion/wiki/BPF BPF].
* Optional: add new Sguil user accounts with the following:
<code>sudo nsm_server_user-add</code>
* Optional, but highly recommended: configure [https://github.com/Security-Onion-Solutions/security-onion/wiki/email Email] for alerting and reporting.
* Optional, but highly recommended: place /etc under version control. If your organization doesn't already have a standard version control tool, you can use [https://help.ubuntu.com/12.04/serverguide/bazaar.html bazaar], [https://git-scm.com/ git], [https://help.ubuntu.com/12.04/serverguide/etckeeper.html etckeeper]:
<code>sudo apt-get install etckeeper</code>
* Optional: need “remote desktop” access to your Security Onion sensor or server? We recommend SSH X-Forwarding as shown above, but if you want something more rdp-like, you can install FreeNX or xrdp:
<code>sudo apt-get install xrdp</code>


 What is a disk partition? Disk partitioning is the creation of one or more regions on a hard disk or other secondary storage, so that an operating system can manage information in each region separately.[1] Partitioning is typically the first step of preparing a newly manufactured disk, before any files or directories have been created
Please note that we do not support [https://github.com/Security-Onion-Solutions/security-onion/wiki/FreeNX FreeNX] or [http://www.xrdp.org xrdp].
* Read more about the tools contained in Security Onion: [https://github.com/Security-Onion-Solutions/security-onion/wiki/Tools Tools]


 What is a file system?
==Using Sguil==


In computing, a file system (or filesystem) is used to control how data is stored and retrieved. Without a file system, information placed in a storage area would be one large body of data with no way to tell where one piece of information stops and the next begins. By separating the data into individual pieces, and giving each piece a name, the information is easily separated and identified. Taking its name from the way paper-based information systems are named, each group of data is called a "file". The structure and logic rules used to manage the groups of information and their names is called a "file system".


 What is journaling in terms of filesystems and what are the benefits? Name some journaled filesystems in use nowadays.
* ''' Double-click the Sguil desktop icon.  Log into Sguil using the username/password you specified in the previous step.  There may already be some alerts in the Sguil console.  If not, open Firefox and click the testmyids.com bookmark and you should then see an alert appear in Sguil.'''


A journaling file system is a file system that keeps track of changes not yet committed to the file system's main part by recording the intentions of such changes in a data structure known as a "journal", which is usually a circular log. In the event of a system crash or power failure, such file systems can be brought back online quicker with lower likelihood of becoming corrupted
[[File:Security_Onion_20110116_Sguil_alert.PNG|frame|none|alt=Alt text|Caption text]]


In the Linux operating system, JFS is supported with the kernel module (since the kernel version 2.4.18pre9-ac4) and the complementary userspace utilities packaged under the name JFSutils. Most Linux distributions support JFS, unless it is specifically removed due to space restrictions or other concerns.
* ''' Double-click the Squert desktop icon.  The Squert main page appears.  Click the "submit" button. Snort alerts appear at the bottom of the page and they should match what you saw in Sguil.'''
Hardware


Computer hardware Jargon: CPU, RAM, ROM, HDD, SSD, PCI, PCI Express, USB 2.0, USB 3.0, VGA, HDMI, DVI, LCD, TFT, LED, OLED, AMOLED, CRT, PWM Lecture recording #1 Lecture recording #2 starting 12:30 Lecture slides Random access memory, permanent storage, buses, input devices, display technologies, networking Potential exam questions:  Different buses and their uses Bus is a system which help to transact the date between each component in computer or between computers. It has 2 types of buses in side computer (Asus socket 7) and outside of computers (Pc card or IEEE-448)
[[File:Security_Onion_20110116_Sguil_alert_(1).PNG|frame|none|alt=Alt text|Caption text]]


 What are the differences between hard disk drive (HDD) and solid state drive (SSD)? [1]
* ''' Go back to Sguil, select an alert, and press the F8 key to expire it.  Notice that the alert disappears from Sguil.'''


 What is the purpose of Flash Translation Layer in terms of solid state drives?
[[File:Security_Onion_20110116_Sguil_alert_expired.PNG|frame|none|alt=Alt text|Caption text]]


A flash translation layer is used to adapt a fully functional file system to the constraints and restrictions imposed by flash memory devices
* ''' Go back to Squert and click the "submit" button again.  Notice that the alert remains in Squert.  Sguil's main console shows events that have not yet been classified, so we need to tell Squert to do the same.  Click the "Status" drop-down box and select "Unclassified".  Click the "submit" button and notice that the alert is now gone. '''


 What are difference between volatile/non-volatile, RAM, ROM, EEPROM and where are they used?
[[File:Security_Onion_20110116_Squert_alert_expired.PNG|frame|none|alt=Alt text|Caption text]]


RAM is Random Access Memory. ROM is Read Only Memory. RAM is the memory available for the operating system, programs and processes to use when the computer is running. ROM is the memory that comes with your computer that is pre-written to hold the instructions for booting-up the computer. RAM requires a flow of electricity to retain data (e.g. the computer powered on). ROM will retain data without the flow of electricity (e.g. when computer is powered off). RAM is a type of volatile memory. Data in RAM is not permanently written. When you power off your computer the data stored in RAM is deleted. ROM is a type of non- volatile memory. Data in ROM is permanently written and is not erased when you power off your computer. There are different types of RAM, including DRAM (Dynamic Random Access Memory) andSRAM (Static Random Access Memory). There are different types of ROM, including PROM (programmable read-only memory) that is manufactured as blank memory (e.g. a CD-ROM) and EPROM (erasable programmable read-only memory). There are many differences between RAM and ROM memory but there are also a couple similarities (and these are very easy to remember). Both types of memory used by a computer, and they are both required for your computer to operate properly and efficiently. EEPROM EEPROM , or electrically erasable programmable read only memory, is another step up from EPROM because EEPROM chips do away with some of the drawbacks. For example, EEPROM chips do not need to be removed to be rewritten. Additionally, a portion of the chip can be changed without erasing the entire chip. Furthermore, it does not require special equipment to rewrite the chip.


Volatile memory Non-volatile memory Requires a power source to retain information. Does not require a power source to retain information. When power source is disconnected, information is lost or deleted. When power source is disconnected, information is not deleted. Often used for temporary retention of data, such as with RAM, or for retention of sensitive data. Often used for long-term retention of data, such as files and folders.
===How do I set up sguil to automatically categorize incoming alerts?===
This is called "automatic categorization", or just "autocat" for short. Take a look at ''/etc/sguild/autocat.conf'', which contains full instructions. Once you edit this file, you'll need to restart sguild in order for the changes to take effect.


 What is data retention? Data retention defines the policies of persistent data and records management for meeting legal and business data archival requirements;
NOTE:  Be sure you are running sguild with the proper "-a" flag!


 What are difference between asynchronous/synchronous, dynamic/static RAM and where are they used? 
===Can sguil page me when it sees a particular alert?===
Yes, using the sguild.email file on the sguild server (for version 0.6 
and higher).  Note that the file is only read on init, and reread on
HUP signals, so if you make changes to it, you'll need to restart
sguild.
Set-up is fairly straightforward, as the file is very well documented.
To activate:
set EMAIL_EVENTS 1
set SMTP_SERVER {your_mail_server}
set EMAIL_RCPT_TO "recipient1@mydomain.com,recipient2@mydomain.com"
set EMAIL_FROM "sguil@mydomain.com"
Modify your notification options to meet your needs:
set EMAIL_CLASSES "successful-admin trojan-activity attempted-admin attempted-user"
set EMAIL_PRIORITIES "0"
Optionally, use the last two parameters, EMAIL_DISABLE_SIDS and  
EMAIL_ENABLE_SIDS to override any specific sids you'd like.
Restart sguild to complete.
By the way, the procedure for 0.5.3 and previous releases is very similar, except that the email configuration is included directly in the sguild.conf file instead.


Synchronous Circuits: These are the class of sequential circuits which are governed by a global clock signal generated by an oscillator. The state of all elements of a synchronous circuit changes only by an application of a distributed clock signal. So, this makes the state of a synchronous circuit predictable. Also, synchronous clock signals are less susceptible to noise, circuit anomalies and hence safer to design and operate. But they are limited in operation of speed by the propagation delay of the clock signal in reaching all the elements of the clock signal. The time period of a clock signal should be long enough to accommodate longest propagation delay. Practically all the circuits today are synchronous circuits, except the part where speed of the circuit operation is crucial.
===How do I expire (purge) old information from the database?===
Sguil 0.5.3 comes with a handy script for this, called ''archive_sguildb.tcl''. Basic usage looks like this:
archive_sguildb.tcl -d 2004-12-27 -p 2004_12_27_ --dbname sguildb \
        --dbhost localhost --dbuser sguil --dbpass password --event \
--session --sancp
This command would expire all event, session and SANCP entries older than "2004-12-27", placing them in new tables called "2004_12_27_event", "2004_12_27_session" and "2004_12_27_sancp". You can drop these tables if you don't want the data, or you can keep them around in case you need to make historical queries. As long as you have the disk space to store them, these older tables do not affect the performance of queries running against the current tables.


Asynchronous Circuits:
After expiring old date, you should also run mysqlcheck to re-optimize the database, reindex and repair tables and to reclaim the space used by the expired data.


Asyncronous circuits change state only through the inputs received by them. So, the operation is quite instantaneous since they dont have to wait for a clock pulse. They are limited by propagation delay of logic gates only. But asynchronous circuits can transition into a wrong state due to incorrect arrival time of 2 inputs. This is called a race condition. Asynchronous circuits are quite difficult to design for a reliable operation. These are used primarily in high speed systems such as Signal Processing hardware.
Be warned that expiring old data may take hours on a large database (especially the sessions and SANCP tables). This can temporarily lock tables in the db, which will interfere with queries and with insertions.  The sensors will queue up their data and try again when the table is unlocked, but interactive use might suffer. It's probably best to run these overnight when no one is using the GUI.


The basic difference between Static and Dynamic RAM lies mainly in structure and work principal.
For sguil 0.5.3, you might also want to try out David Bianco's [http://www.biancorama.com/sguil/sguil_age_db sguil_age_db script], which is a wrapper for [[archive_sguildb.tcl]].  The script's advantage is that it doesn't require you to give an absolute date for the expiration time, and you can specify different thresholds for different tables.  For example:
    sguil_age_db --event "-27 days" --session "-3 weeks" --sancp "-1 month"
This makes it a little more suitable for running out of cron.


•Firstly the main difference in the structure varies due to transistor and capacitor number and setting as just three to four transistors are required for a Dynamic RAM, but six to eight MOS transistors are necessary for a Static RAM.
Sguil 0.6.0 and above changes the database schema extensively, and the archive script is no longer necessary.  This version uses MERGE tables to create "virtual tables" for events, SANCP records and other supporting information.  The virtual tables are comprised of a number of individual tables, one for each day.  The table names look something like "tablename_sensorname_date" (e.g., "sancp_externalnet_20051128", "event_finance_20051031" or "data_finance_20051031").  The sguil server creates the merged tables dynamically, so you'll find "event", "icmphdr", "tcphdr", "udphdr", "data" and "sancp" tables, along with all the individual daily tables that make up these merged tables.


•Secondly Dynamic RAM memory can be deleted and refreshed while running the program, but in case of Static RAM it is not possible to refresh programs.  
Given this, if you want to get rid of old data, simply stop the sguil server, drop the daily tables you don't want, drop the merged tables, then restart the sguil server.  Sguil will recreate the merged tables using the remaining data in the database.


•Data is stored as a charge in a capacitor in Dynamic RAM, where data is stored in flip flop level in Static RAM.
Here is a handy bash shell script that will automate this process and also repairs any remaining tables to keep data corruption to a minumum:


•For refreshing a data another capacitor is required in case of Dynamic capacitor, but no refreshing option is available in Static RAM.
#! /bin/bash
•A Dynamic RAM possesses less space in the chip than a Static RAM.
•Dynamic RAM is used to create larger RAM space system, where Static RAM create speed- sensitive cache.
DATABASE=sguildb
•Static ram is 4 times more expensive than Dynamic RAM. •Dynamic RAM consumes less power than Static RAM.
DB_USER=sguil
DB_PASSWORD=password
DAYSTOKEEP=45
KEEPDAY=`/usr/bin/mysql -u$DB_USER -p$DB_PASSWORD -BN -e "SELECT DATE_FORMAT(DATE_SUB(NOW(), INTERVAL $DAYSTOKEEP DAY), '%Y%m%d');" -D $DATABASE`
/sbin/service sguild stop
for TABLEPREFIX in "data" "event" "icmphdr" "sancp" "tcphdr" "udphdr"
do
/usr/bin/mysql -u$DB_USER -p$DB_PASSWORD -BN -e "DROP TABLE $TABLEPREFIX;" -D $DATABASE
TABLES=(`/usr/bin/mysql -u$DB_USER -p$DB_PASSWORD -BN -e "SHOW TABLES LIKE '$TABLEPREFIX%';" -D $DATABASE`)
for TABLE in "${TABLES[@]}"
do
TABLEDAY=`echo "$TABLE" | awk -F_ '{print($3)}'`
if [ "$TABLEDAY" -lt "$KEEPDAY" ]
then /usr/bin/mysql -u$DB_USER -p$DB_PASSWORD -BN -e "DROP TABLE $TABLE;" -D $DATABASE
else
/usr/bin/mysql -u$DB_USER -p$DB_PASSWORD -BN -e "REPAIR TABLE $TABLE;" -D $DATABASE
fi
done
done
/sbin/service sguild start


•For accessing a data or information, Static RAM takes less time than Dynamic RAM. •Dynamic RAM has higher storage capacity. In fact it can store 4 times than Static RAM.
===What commands are available in the "User Messages" window?===
Most people probably don't realize this, but the client's User Messages window is good for more than just user-to-user chat. It also offers a few simple commands you can use to check the status of the sguil sensors and server. To use one of the commands, simply type it on a line by itself in the User Message tab.  


 What is cache? What is cache coherence?
Version 0.5.3 supports the following commands:


Cache is very fast and small memory that is placed in between the CPU and the main memory. cache coherence is the consistency of shared resource data that ends up stored in multiple local caches. When clients in a system maintain caches of a common memory resource, problems may arise with inconsistent data, which is particularly the case with CPUs in a multiprocessing system.
{| border="1"
! Command || Purpose
|-
| agents || Lists all the sensor agents connected to sguild.  This is deprecated, but still supported.
|-
| healthcheck || Like the "agents" command, but more comprehensive.  It also checks each agent to make sure it is still actively responding to requests.  Unlike the other commands, the output for this is displayed in the "System Messages" tab.  This is also deprecated, since the 0.6.0 client now includes a handy "Sensor Status" panel.
|-
| sensors || An alias for the "agents" command.
|-
| who || List all users connected to sguild.
|}
===I'm not satisfied with the default packet logging subsystem.  Are there any alternatives available?===
Yes.  Two alternatives have already been developed, based on [[DaemonLogger] and [[SANCP]].  See [[Packet Logging in Sguil]] for more information on this subsystem and the alternatives available.
=Summary=
Sguil is a superb software to combine together a very large amount of IDS information while keeping every it sorted and very easily manageable. Additional tools in Security Onion also help to set up custom configuration with all analysis software a few clicks away. This gives a lot of possibility for automation of deep packet analysis. Clean installation and configuration of Sguil with all its dependencies is quite time consuming for a newcomer. But Security Onion can reduce this very to a reasonable time frame.


 What are differences between resistive and capacitive touchscreen? [2]
=See also=
* [https://en.wikipedia.org/wiki/Sagan_(software) Sagan]
* [https://en.wikipedia.org/wiki/Intrusion_detection_system IDS]
* [https://en.wikipedia.org/wiki/Intrusion_prevention_system IPS]
* [https://en.wikipedia.org/wiki/Metasploit_Project Metasploit Project]
* [https://en.wikipedia.org/wiki/Nmap NMAP]
* [https://www.snort.org/ SNORT]
* [https://www.bro.org/ BRO]


Explain how computer mouse works? History of computer mouse.
=References=
 
*[http://sguil.sourceforge.net Sguil Homepage]
Ball mouse and optical mouse How does a mouse like this actually work? As you move it across your desk, the ball rolls under its own weight and pushes against two plastic rollers linked to thin wheels (numbered 6 and 7 in the photo). One of the wheels detects movements in an up-and-down direction (like the y-axis on graph/chart paper); the other detects side-to-side movements (like the x-axis on graph paper).
*[http://nsmwiki.org/Sguil_FAQ Sguil FAQ]
 
*[http://nsmwiki.org/Main_Page NSMWiki]: The official wiki for the Sguil project.
How do the wheels measure your hand movements? As you move the mouse, the ball moves the rollers that turn one or both of the wheels. If you move the mouse straight up, only the y-axis wheel turns; if you move to the right, only the x-axis wheel turns. And if you move the mouse at an angle, the ball turns both wheels at once. Now here's the clever bit. Each wheel is made up of plastic spokes and, as it turns, the spokes repeatedly break a light beam. The more the wheel turns, the more times the beam is broken. So counting the number of times the beam is broken is a way of precisely measuring how far the wheel has turned and how far you've pushed the mouse. The counting and measuring is done by the microchip inside the mouse, which sends details down the cable to your computer. Software in your computer moves the cursor on your screen by a corresponding amount.
*[https://security-onion-solutions.github.io/security-onion/ Security Onion homepage]
 
*[https://en.wikipedia.org/wiki/Sguil Wikipedia Sguil]
An optical mouse works in a completely different way. It shines a bright light down onto your desk from an LED (light-emitting diode) mounted on the bottom of the mouse. The light bounces straight back up off the desk into a photocell (photoelectric cell), also mounted under the mouse, a short distance from the LED. The photocell has a lens in front of it that magnifies the reflected light, so the mouse can respond more precisely to your hand movements. As you push the mouse around your desk, the pattern of reflected light changes, and the chip inside the mouse uses this to figure out how you're moving your hand. The mouse was invented by Douglas Engelbart in 1964 and consisted of a wooden shell, circuit board and two metal wheels that came into contact with the surface it was being used on.
*[https://github.com/Security-Onion-Solutions/security-onion Security Onion Github Wiki]
 
[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]
 Explain how computer keyboard works? HowStuffworks article Explain that Stuff article Keyboard History http://www.explainthatstuff.com/computerkeyboards.html
 
Keyboards and typing technology have come a long way over the past couple centuries. The first typing devices were designed and patented in the 1700s while the first manufactured typing devices came about in the 1870s. These machines featured “blind typing” technology, where characters were printed on upside-down pages that remained unseen until completion. Since then, we have seen several updates in design, layout, technology, and function that are more efficient and user-friendly.
 
 Explain how cathode ray tube (CRT) based screen technology works and name pros/cons. [3]
 
Sort for cathode-ray tubes, CRT monitors were the only choice consumers had for monitor technology for many years. Cathode ray tube (CRT) technology has been in use for more than 100 years, and is found in most televisions and computer monitors. A CRT works by moving an electron beam back and forth across the back of the screen. Each time the beam makes a pass across the screen, it lights up phosphor dots on the inside of the glass tube, thereby illuminating the active portions of the screen. By drawing many such lines from the top to the bottom of the screen, it creates an entire screen of images.
 
Resolution on a CRT is flexible and a newer model will provide you with viewing resolutions of up to 1600 by 1200 and higher, On a CRT the sharpness of the picture can be blemished by soft edges or a flawed focus. A CRT monitor can be viewed from almost any angle Some users of a CRT may notice a bit of an annoying flicker, which is an inherent trait based on a CRTs physical components. Today's graphics cards, however, can provide a high refresh rate signal to the CRT to get rid of this otherwise annoying problem.. Screen (viewable) Size Most people today tend to look at a 17-inch CRT or bigger monitor. When you purchase a 17-inch CRT monitor, you usually get 16.1 inches or a bit more of actual viewing area, depending on the brand and manufacturer of a specific CRT. Physical Size There is no denying that an LCD wins in terms of its physical size and the space it needs. CRT monitors are big, bulky and heavy. They are not a good choice if you're working with limited desk space, or need to move the monitor around (for some odd reason) between computers
 
 Explain how liquid crystal displays (LCD) work and name pros/cons. [4]
 
Short for liquid crystal display, LCD technology can be found in digital watches and computer monitors. LCD displays use two sheets of polarizing material with a liquid crystal solution between them. An electric current passed through the liquid causes the crystals to align so that light cannot pass through them. Each crystal, therefore, is like a shutter, either allowing light to pass through or blocking the light. Color LCD displays use two basic techniques for producing color: Passive matrix is the less expensive of the two technologies. The other technology, calledthin film transistor (TFT) or active-matrix, produces color images that are as sharp as traditional CRT displays, but the technology is expensive.
 
resolution
 
an LCD the resolution is fixed within each monitor (called a native resolution). The resolution on an LCD can be changed, but if you're running it at a resolution other than its native resolution you will notice a drop in performance or quality. Both types of monitors (newer models) provide bright and vibrant color display. However, LCDs cannot display the maximum color range that a CRT can. In terms of image sharpness, when an LCD is running at its native resolution the picture quality is perfectly sharp. On a CRT the sharpness of the picture can be blemished by soft edges or a flawed focus. A CRT monitor can be viewed from almost any angle, but with an LCD this is often a problem. When you use an LCD, your view changes as you move different angles and distances away from the monitor. At some odd angles, you may notice the picture fade, and possibly look as if it will disappear from view.
 
Refresh Rate
 
Some users of a CRT may notice a bit of an annoying flicker, which is an inherent trait based on a CRTs physical components. Today's graphics cards, however, can provide a high refresh rate signal to the CRT to get rid of this otherwise annoying problem. LCDs are flicker-free and as such the refresh rate isn't an important issue with LCDs. Dot Pitch Dot pitch refers to the space between the pixels that make up the images on your screen, and is measured in millimeters. The less space between pixels, the better the image quality. On either type of monitor, smaller dot pitch is better and you're going to want to look at something in the 0.26 mm dot pitch or smaller range. Screen (viewable) Size Most people today tend to look at a 17-inch CRT or bigger monitor. When you purchase a 17-inch CRT monitor, you usually get 16.1 inches or a bit more of actual viewing area, depending on the brand and manufacturer of a specific CRT. The difference between the "monitor size" and the "view area" is due to the large bulky frame of a CRT. If you purchase a 17" LCD monitor, you actually get a full 17" viewable area, or very close to a 17".
 
Physical Size
 
There is no denying that an LCD wins in terms of its physical size and the space it needs. CRT monitors are big, bulky and heavy. They are not a good choice if you're working with limited desk space, or need to move the monitor around (for some odd reason) between computers. An LCD on the other hand is small, compact and lightweight. LCDs are thin, take up far less space and are easy to move around. An average 17-inch CRT monitor could be upwards of 40 pounds, while a 17&-inch LCD would weigh in at around 15 pounds. Price As an individual one-time purchase an LCD monitor is going to be more expensive. Throughout a lifetime, however, LCDs are cheaper as they are known to have a longer lifespan and also a lower power consumption. The cost of both technologies have come down over the past few years, and LCDs are reaching a point where smaller monitors are within many consumers' price range. You will pay more for a 17" LCD compared to a 17" CRT, but since the CRT's actual viewing size is smaller, it does bring the question of price back into proportion. Today, fewer CRT monitors are manufactured as the price on LCDs lowers and they become mainstream.
 
 Name screen technologies making use of thin film transistor (TFT) technology? [5]
 
A thin-film transistor (TFT) is a special kind of field-effect transistor made by depositing thin films of an active semiconductor layer as well as the dielectric layer and metallic contacts over a supporting (but non-conducting) substrate. A common substrate is glass, because the primary application of TFTs is in liquid-crystal displays. This differs from the conventional transistor, where the semiconductor material typically is the substrate, such as a silicon wafer.
 
 Name uses for light polarization filters? [6] [7] Camera, tv, photography….
 
 What are the benefits of twisted pair cabling and differential signalling? twisted pair cabling
  Electrical noise going into or coming from the cable can be prevented.[10]  Cross-talk is minimized differential signalling The technique minimizes electronic crosstalk and electromagnetic interference, both noise emission and noise acceptance, and can achieve a constant or known characteristic impedance, allowing impedance matching techniques important in a high-speed signal transmission line or high qualitybalanced line and balanced circuit audio signal path.
 
 Active matrix vs passive matrix in display technology 
 
Active-matrix display : An active-matrix display, also known as a TFT (thin-film transistor) display, uses a separate transistor to apply charges to each liquid crystal cell and thus displays high-quality color that is viewable from all angles.
 
Passive-matrix display : A passive-matrix display uses fewer transistors, requires less power, and is less expensive than an active-matrix display. The color on a passive-matrix display often is not as bright as an active-matrix display. Users view images on a passive-matrix display best when working directly in front of it.
 
    Compare FAT32 and NTFS
 
NTFS
 
NTFS is the preferred file system for this version of Windows. It has many benefits over the earlier FAT32 file system, including:
 
The capability to recover from some disk-related errors automatically, which FAT32 cannot.
 
Improved support for larger hard disks.
 
Better security because you can use permissions and encryption to restrict access to specific files to approved users.
 
FAT32
 
FAT32, and the lesser-used FAT, were used in earlier versions of Windows operating systems, including Windows 95, Windows 98, and Windows Millennium Edition. FAT32 does not have the security that NTFS provides, so if you have a FAT32 partition or volume on your computer, any user who has access to your computer can read any file on it. FAT32 also has size limitations. You cannot create a FAT32 partition greater than 32GB in this version of Windows, and you cannot store a file larger than 4GB on a FAT32 partition.
Bootloaders, kernels
 
 What is the role of BIOS/UEFI in x86-based machines?  BIOS (Basic Input/Output System) is read from EEPROM and copied to RAM
 
• Processor starts executing the BIOS code in RAM • BIOS sets up the hardware and probes storage, USB etc for bootable media • BIOS reads master boot record of selected bootable media and boot loader takes over
 
 Explain step by step how operating system is booted up, see slides for flowchart.
 
 Turn on the Power button.  • CPU pins are reset and registers are set to specific value.  • CPU jump to address of BIOS (0xFFFF0).  • BIOS run POST (Power-On Self Test) and other necessary checks.  • BIOS jumps to MBR(Master Boot Record).  • Primary Bootloader runs from MBR and jumps to Secondary Bootloader.  • Secondary Bootloaders loads Operating System
 
 Describe the functionality provided by general purpose operating system. See architecture of Windows NT, Android, OS X.
 
User mode in Windows NT is made of subsystems capable of passing I/O requests to the appropriate kernel mode device drivers by using the I/O manager. The user mode layer of Windows NT is made up of the "Environment subsystems," which run applications written for many different types of operating systems, and the "Integral subsystem," which operates system specific functions on behalf of environment subsystems.
 
 What are the main differences between real mode and protected mode of x86-based processor?  If your computer is in real mode, software communicates directly with the computer's ports and devices. For example, when you print a document, the software sends the data stream directly to the port that holds the printer. However, this paradigm doesn't work in a multitasking OS. Imagine what would happen if multiple programs sent data streams to the ports simultaneously. Ports are dumb, and they have no ability to filter or arrange data streams to match the sending programs.  If your computer is in protected mode, the system's ports and devices are protected from the applications that use them. The software thinks it's sending data to a port, but it's a virtual port. The OS is grabbing the data stream and managing it, to ensure that all applications have equal access and to ensure that data from each application is appropriately preserved.
 
 
 What happens during context switch?
 
 In a switch, the state of the first process (assuming that the first process is the process in execution and is to be switched) must be saved somehow, so that, when the scheduler gets back to the execution of the first process, it can restore this state and continue.  The state of the process includes all the registers that the process may be using, especially the program counter, plus any other operating system specific data that may be necessary. This data is usually stored in a data structure called a process control block (PCB), or switchframe.  In order to switch processes, the PCB for the first process must be created and saved. The PCBs are sometimes stored upon a per-process stack in kernel memory (as opposed to the user-mode call stack), or there may be some specific operating system defined data structure for this information.  Since the operating system has effectively suspended the execution of the first process, it can now load the PCB and context of the second process. In doing so, the program counter from the PCB is loaded, and thus execution can continue in the new process. New processes are chosen from a queue or queues (often referred as ready queue). Process and thread priority can influence which process continues execution, with processes of the highest priority checked first for ready threads to execute.
 
 What is the purpose of paged virtual memory?
 
 In computing, virtual memory is a memory management technique that is implemented using both hardware and software. It maps memory addresses used by a program, called virtual addresses, into physical addresses in computer memory. Main storageas seen by a process or task appears as a contiguous address space or collection of contiguous segments. The operating system manages virtual address spaces and the assignment of real memory to virtual memory. Address translation hardware in the CPU, often referred to as a memory management unit or MMU, automatically translates virtual addresses to physical addresses. Software within the operating system may extend these capabilities to provide a virtual address space that can exceed the capacity of real memory and thus reference more memory than is physically present in the computer.  The primary benefits of virtual memory include freeing applications from having to manage a shared memory space, increased security due to memory isolation, and being able to conceptually use more memory than might be physically available, using the technique of paging.

Latest revision as of 00:44, 30 January 2017


Author: Kustas Kurval

Cyber Security Engineering C11

Written 06.06.2016


Introduction

This tutorial was made to make an introduction to Sguil. Sguil (pronounced sgweel) is built by network security analysts for network security analysts. It is a collection of free software components for Network Security Monitoring (NSM) and event driven analysis of IDS alerts. Sguil's main component is an intuitive GUI that provides access to realtime events, session data, and raw packet captures. Sguil facilitates the practice of Network Security Monitoring and event driven analysis.

The Sguil client is written in tcl / tk and can be run on any operating system that supports tcl / tk (including Linux, *BSD, Solaris, MacOS, and Win32).

It is provided by Q Public License

Sguil integrates alert data from Snort, session data from SANCP, and full content data from a second instance of Snort running in packet logger mode.

In this introduction I will be covering Sguil in Xbuntu. You will need to know basic Linux syntax and terminology also some terminology concerning overall intrusion detection and prevention systems (IDPS) and overall basic networking.

Software architecture

A sguil system is composed of a single sguil server and an arbitrary number of sguil network sensors. The sensors perform all the security monitoring tasks and feed information back to the server on a regular basis. The server coordinates this information, stores it in a database and communicates with sguil clients running on administrators' desktop machines. It can also issue requests for specific information from the sensors.

Each sensor monitors a single network link (although you can have multiple sensors on one physical machine). They collect several different types of information: 

                                           |<---------> Sensor #1 
           Client  ==>  Server <=========> |<---------> Sensor #2
                                           |<---------> Sensor #3
  1. Snort monitors the link for security events, and logs them to a file on the local disk.
  2. Barnyard takes events from the snort log file and sends them to the sensor agent, which inserts them into database running on the sguil server in near real-time
  3. A separate instance of snort logs the full content of all network packets to the local disk (this typically requires a large separate data partition)
  4. SANCP records TCP/IP sessions and forwards them to the database on the sguil server
  5. The sguil agent also listens for commands from the sguil server. These commands are typically requests for packet data previously logged by Snort.

Contents

Since Sguil is dependant on many other types of software to gather, facilitate, store ,decode and analyze I will be using Xbuntu based distribution Security Onion which saves massive amount of time to set up the entire environment. Security Onion has all this and more build in and is able to quickly configure which software to tie to Sguil.

Dependencies

Hardware

The hardware necessary to run sguil depends entirely upon the amount of network traffic you plan to monitor, how much of it you store and how long you store it. The public demonstration server at demo.sguil.net gets by with a 133MHz Pentium system, though for production systems you'll want something beefier.

For the sguil server, try to find something with a lot of memory and a fast disk. The MySQL database will need a lot of RAM for caching results from its queries, and a fast disk will speed up the access time while doing sequential searches.

You may find it helpful to place the MySQL storage on a separate disk partition, or even a separate disk. It doesn't matter where you mount this storage area, but for convenience, I'll assume that that you're going to mount it on the same directory path you're using for the sensor data directory (see below), referred to in this document as $NSM.

The following table is a useful guideline for a good server hardware configuration. Feel free to adjust it up or down depending on the size of your installation.

Recommended Server Hardware
CPU RAM Disk Storage
3.0GHz 2GB 150GB (Consider a RAID5 array for speed & reliability)

Software

Tools that usually make up Sguil

Tool Purpose
MySQL 4.x or 5.x Data storage and retrieval
Snort 2.x / Suricata Intrusion detection alerts, scan detection, packet logging
Barnyard / Barnyard2 Decodes IDS alerts and sends them to sguil
SANCP TCP/IP session records
Tcpflow Extract an ASCII dump of a given TCP session
p0f Operating system fingerprinting
tcpdump Extracts individual sessions from packet logs
Wireshark Packet analysis tool (used to be called Ethereal)

Setup

I Installed this on Oracle Virutalbox as a 64 bit Ubuntu operating system, with 4GB of memory and a single processor. I set the network adapter as bridged with promiscuous mode allowed. This ensured that I am able to capture network traffic from the host machine

  • Follow the prompts in the Xubuntu installer. If prompted with an encrypt home folder or encrypt partition option, DO NOT enable this feature. If asked about * * * automatic updates, DO NOT enable automatic updates. Reboot into your new installation. Login using the username/password you specified during installation.
  • Verify that you have Internet connectivity. If necessary, configure your proxy settings.
  • Install updates and reboot.
  • Double-click the Setup icon on the desktop. The Setup wizard will walk you through configuring /etc/network/interfaces and will then reboot.
  • After rebooting, log back in and start the Setup wizard again. It will detect that you have already configured /etc/network/interfaces and will walk you through the rest of the configuration. When prompted for Evaluation Mode or Production Mode, choose Evaluation Mode.

Security Onion usually expects at least two networking interfaces. One for monitoring the other for management. Since I only had access to a single interface on the virtual machine I set it as management. I used the static IP address 192.168.1.111 with regular /24 subnet mask for ease of use. After this I was prompted for a gateway address and DNS server.

  • Once you've completed the Setup wizard, use the Desktop icons to login to Sguil.

Post Installation

Verify services are running: 

sudo service nsm status

If any services are not running, try starting them: 

sudo service nsm start

Tuning / Miscellaneous

  • Are you monitoring network traffic that has VLAN tags? If so, take a look at our VLAN page.
  • If you’re monitoring IP address ranges other than private RFC1918 address space (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12), you should update your sensor configuration with the correct IP ranges. Sensor configuration files can be found in /etc/nsm/$HOSTNAME-$INTERFACE/. Modify either snort.conf or suricata.yaml (depending on which IDS engine you chose during sosetup) and update the HOME_NET variable. Also update the home_nets variable in prads.conf. Then update Bro’s network configuration in /opt/bro/etc/networks.cfg. Restart the sensor processes:
sudo nsm_sensor_ps-restart
  • If you have Internet access, create an IDS alert by typing the following at a terminal:

curl http://testmyids.com

  • As of securityonion-setup - 20120912-0ubuntu0securityonion201, Setup now defaults to only opening port 22 in the firewall. If you need to connect OSSEC agents, syslog devices, or analyst VMs, you can run the new so-allow utility which will walk you through creating firewall rules to allow these devices to connect. For more information, please see the firewall page.
  • Full-time analysts should install Security Onion in a VM on their workstation (run through the Ubuntu installer, but do not run our Setup wizard). This gives you a local copy of Wireshark, NetworkMiner, and our customized Sguil client. Launch the Sguil client and connect to the IP/hostname of your production Sguil sensor (you may need to run so-allow as described in the previous step). This allows you to investigate pcaps without fear of impacting your production server/sensors. To change the resolution of your Security Onion VM, install the Virtual Tools for your virtualization solution or use xrandr. For a list of available screen resolutions, simply execute “xrandr”. To set the screen resolution (replace W and H with the actual Width and Height desired):

xrandr -s WxH

  • Login to Sguil and review your IDS alerts. Squert and ELSA can be accessed by visiting https://server/ for additional in-depth analysis.
  • Run the following to see how your sensor is coping with the load. You should check this on a daily basis to make sure your sensor is not dropping packets. Consider adding it to a cronjob and having it emailed to you (see the “configure email” link below).

sudo sostat | less

  • Please note that any IDS/NSM system needs to be tuned for the network it’s monitoring. Please see ManagingAlerts. You should only run the signatures you really care about.
  • Also note that you should be looking at and categorizing events every day with the goal being to categorize all events every day. Even if you don’t use the Sguil console for your primary analysis, you need to log into it periodically and F8 old events to keep the real time queue from getting too big. Neglecting to do so will result in database/Sguil issues as the number of uncategorized events continues to increase on a daily basis. Please see the Sguil client page on NSMwiki.
  • On the server running the Sguil database, set the DAYSTOKEEP variable in /etc/nsm/securityonion.conf to however many days you want to keep in your archive. The default is 30, but you may need to adjust it based on your organization’s detection/response policy and your available disk space.
  • If you enabled http_agent, you should tune it using http_agent.conf. If you're running ELSA, you already have all the Bro HTTP logs available there, so you might want to disable http_agent to avoid duplicating those logs in the Sguil database:

# Terminate the running http_agent

sudo nsm_sensor_ps-stop --only-http-agent

# Disable http_agent

sudo sed -i 's|HTTP_AGENT_ENABLED="yes"|HTTP_AGENT_ENABLED="no"|g' /etc/nsm//sensor.conf

  • Disable any unneeded sensor processes
  • Tune the number of PF_RING instances for Snort/Suricata and Bro: PF_RING
  • Optional:* exclude unnecessary traffic from your monitoring using BPF.
  • Optional: add new Sguil user accounts with the following:

sudo nsm_server_user-add

  • Optional, but highly recommended: configure Email for alerting and reporting.
  • Optional, but highly recommended: place /etc under version control. If your organization doesn't already have a standard version control tool, you can use bazaar, git, etckeeper:

sudo apt-get install etckeeper

  • Optional: need “remote desktop” access to your Security Onion sensor or server? We recommend SSH X-Forwarding as shown above, but if you want something more rdp-like, you can install FreeNX or xrdp:

sudo apt-get install xrdp

Please note that we do not support FreeNX or xrdp.

  • Read more about the tools contained in Security Onion: Tools

Using Sguil

  • Double-click the Sguil desktop icon. Log into Sguil using the username/password you specified in the previous step. There may already be some alerts in the Sguil console. If not, open Firefox and click the testmyids.com bookmark and you should then see an alert appear in Sguil.
Alt text
Caption text
  • Double-click the Squert desktop icon. The Squert main page appears. Click the "submit" button. Snort alerts appear at the bottom of the page and they should match what you saw in Sguil.
Alt text
Caption text
  • Go back to Sguil, select an alert, and press the F8 key to expire it. Notice that the alert disappears from Sguil.
Alt text
Caption text
  • Go back to Squert and click the "submit" button again. Notice that the alert remains in Squert. Sguil's main console shows events that have not yet been classified, so we need to tell Squert to do the same. Click the "Status" drop-down box and select "Unclassified". Click the "submit" button and notice that the alert is now gone.
Alt text
Caption text


How do I set up sguil to automatically categorize incoming alerts?

This is called "automatic categorization", or just "autocat" for short. Take a look at /etc/sguild/autocat.conf, which contains full instructions. Once you edit this file, you'll need to restart sguild in order for the changes to take effect.

NOTE: Be sure you are running sguild with the proper "-a" flag!

Can sguil page me when it sees a particular alert?

Yes, using the sguild.email file on the sguild server (for version 0.6  
and higher).  Note that the file is only read on init, and reread on 
HUP signals, so if you make changes to it, you'll need to restart 
sguild.

Set-up is fairly straightforward, as the file is very well documented.

To activate:
	set EMAIL_EVENTS 1
	set SMTP_SERVER {your_mail_server} 
	set EMAIL_RCPT_TO "recipient1@mydomain.com,recipient2@mydomain.com"
	set EMAIL_FROM "sguil@mydomain.com"

Modify your notification options to meet your needs:

	set EMAIL_CLASSES "successful-admin trojan-activity attempted-admin attempted-user"
	set EMAIL_PRIORITIES "0"

Optionally, use the last two parameters, EMAIL_DISABLE_SIDS and 
EMAIL_ENABLE_SIDS to override any specific sids you'd like.

Restart sguild to complete.

By the way, the procedure for 0.5.3 and previous releases is very similar, except that the email configuration is included directly in the sguild.conf file instead.

How do I expire (purge) old information from the database?

Sguil 0.5.3 comes with a handy script for this, called archive_sguildb.tcl. Basic usage looks like this: 

archive_sguildb.tcl -d 2004-12-27 -p 2004_12_27_ --dbname sguildb \
       --dbhost localhost --dbuser sguil --dbpass password --event \
	--session --sancp

This command would expire all event, session and SANCP entries older than "2004-12-27", placing them in new tables called "2004_12_27_event", "2004_12_27_session" and "2004_12_27_sancp". You can drop these tables if you don't want the data, or you can keep them around in case you need to make historical queries. As long as you have the disk space to store them, these older tables do not affect the performance of queries running against the current tables.

After expiring old date, you should also run mysqlcheck to re-optimize the database, reindex and repair tables and to reclaim the space used by the expired data.

Be warned that expiring old data may take hours on a large database (especially the sessions and SANCP tables). This can temporarily lock tables in the db, which will interfere with queries and with insertions. The sensors will queue up their data and try again when the table is unlocked, but interactive use might suffer. It's probably best to run these overnight when no one is using the GUI.

For sguil 0.5.3, you might also want to try out David Bianco's sguil_age_db script, which is a wrapper for archive_sguildb.tcl. The script's advantage is that it doesn't require you to give an absolute date for the expiration time, and you can specify different thresholds for different tables. For example: 

   sguil_age_db --event "-27 days" --session "-3 weeks" --sancp "-1 month"

This makes it a little more suitable for running out of cron.

Sguil 0.6.0 and above changes the database schema extensively, and the archive script is no longer necessary. This version uses MERGE tables to create "virtual tables" for events, SANCP records and other supporting information. The virtual tables are comprised of a number of individual tables, one for each day. The table names look something like "tablename_sensorname_date" (e.g., "sancp_externalnet_20051128", "event_finance_20051031" or "data_finance_20051031"). The sguil server creates the merged tables dynamically, so you'll find "event", "icmphdr", "tcphdr", "udphdr", "data" and "sancp" tables, along with all the individual daily tables that make up these merged tables.

Given this, if you want to get rid of old data, simply stop the sguil server, drop the daily tables you don't want, drop the merged tables, then restart the sguil server. Sguil will recreate the merged tables using the remaining data in the database.

Here is a handy bash shell script that will automate this process and also repairs any remaining tables to keep data corruption to a minumum: 

#! /bin/bash

DATABASE=sguildb
DB_USER=sguil
DB_PASSWORD=password
DAYSTOKEEP=45 

KEEPDAY=`/usr/bin/mysql -u$DB_USER -p$DB_PASSWORD -BN -e "SELECT DATE_FORMAT(DATE_SUB(NOW(), INTERVAL $DAYSTOKEEP DAY), '%Y%m%d');" -D $DATABASE` 

/sbin/service sguild stop

for TABLEPREFIX in "data" "event" "icmphdr" "sancp" "tcphdr" "udphdr"
do
	/usr/bin/mysql -u$DB_USER -p$DB_PASSWORD -BN -e "DROP TABLE $TABLEPREFIX;" -D $DATABASE 
	TABLES=(`/usr/bin/mysql -u$DB_USER -p$DB_PASSWORD -BN -e "SHOW TABLES LIKE '$TABLEPREFIX%';" -D $DATABASE`)
	for TABLE in "${TABLES[@]}"
	do
		TABLEDAY=`echo "$TABLE" | awk -F_ '{print($3)}'`
		if [ "$TABLEDAY" -lt "$KEEPDAY" ]
			then /usr/bin/mysql -u$DB_USER -p$DB_PASSWORD -BN -e "DROP TABLE $TABLE;" -D $DATABASE
		else
			/usr/bin/mysql -u$DB_USER -p$DB_PASSWORD -BN -e "REPAIR TABLE $TABLE;" -D $DATABASE
		fi
	done
done

/sbin/service sguild start

What commands are available in the "User Messages" window?

Most people probably don't realize this, but the client's User Messages window is good for more than just user-to-user chat. It also offers a few simple commands you can use to check the status of the sguil sensors and server. To use one of the commands, simply type it on a line by itself in the User Message tab.

Version 0.5.3 supports the following commands:

Command Purpose
agents Lists all the sensor agents connected to sguild. This is deprecated, but still supported.
healthcheck Like the "agents" command, but more comprehensive. It also checks each agent to make sure it is still actively responding to requests. Unlike the other commands, the output for this is displayed in the "System Messages" tab. This is also deprecated, since the 0.6.0 client now includes a handy "Sensor Status" panel.
sensors An alias for the "agents" command.
who List all users connected to sguild.

I'm not satisfied with the default packet logging subsystem. Are there any alternatives available?

Yes. Two alternatives have already been developed, based on [[DaemonLogger] and SANCP. See Packet Logging in Sguil for more information on this subsystem and the alternatives available.

Summary

Sguil is a superb software to combine together a very large amount of IDS information while keeping every it sorted and very easily manageable. Additional tools in Security Onion also help to set up custom configuration with all analysis software a few clicks away. This gives a lot of possibility for automation of deep packet analysis. Clean installation and configuration of Sguil with all its dependencies is quite time consuming for a newcomer. But Security Onion can reduce this very to a reasonable time frame.

See also

References

Retrieved from "https://wiki.itcollege.ee/index.php?title=Sguil&oldid=117339"