Logging - Monitoring C21
Logging and Monitoring with Graylog
Course: Logging and Monitoring - Lecturer: Margus Ernits
Group: Cyber Security Engineering (C21)
Team members: Ender Phan, Kustas Kurval, Sheela Gowry Sumathi Raju, Artur Vincent Kerge
Page created on: October 05, 2016
Abstract
In order to understand how to set up the Graylog service as well as understand its crucial roles. We decided to choose Graylog as our application for logging and monitoring. Below are our objectives which would be expected to achieve later on:
- How to install Graylog on Ubuntu 14.04.
- How to use Graylog to protect servers.
- Upgrading and configuring Graylog at first, and know how to secure Graylog.
- Threats and security during logging.
- How to install Netdata and Monitoring with Netdata.
- How to install Nagios and Monitoring with Nagios.
- Trouble shooting and experiences.
Installation Guide
Ubuntu 14.04
Logging Solutions
Since the Elasticsearch is based on java, we would require to install either openJDK or Oracle JDK. It is recommended to install Oracle JDK, verify the java version by using the following command.
Remove the OpenJDK from the system, if you have it already installed. From now on it is presumed that user has root privileges.
apt-get remove --purge openjdk*
Add repository.
add-apt-repository -y ppa:webupd8team/java
Run the following command to pull the packages information from the newly added repository.
apt-get update
Issue the following command to install Java jdk 1.8.
apt-get -y install oracle-java8-installer
java -version
Output:
Java version "1.8.0_60" Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
Installing extras
Elasticsearch
Let’s install Elasticsearch, it can be downloaded from the official website.
Download and install GPG signing key.
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Note that the -qO argument is lowercase Quebec followed by capital Oscar.
Save the repository definition to /etc/apt/sources.list.d/elasticsearch.list
echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch.list
Update repository cache and install Elasticsearch
apt-get update && apt-get install elasticsearch
Configure Elasticsearch to start during system startup.
update-rc.d elasticsearch defaults
The only important thing is to set a cluster name as “graylog2“, that is being used by graylog. Now edit the configuration file of Elasticsearch.
nano /etc/elasticsearch/elasticsearch.yml
cluster.name: graylog2
In the same file disable dynamic scripts to avoid remote execution. That can be done by adding the following line:
FOLLOWING LINE GOES INTO GRAYLOG CONF!
script.disable_dynamic: true
Once it is done, we are good to go. Before that, restart the Elasticsearch services to load the modified configuration.
service elasticsearch restart
Wait at least a minute to let the Elasticsearch get fully restarted, otherwise testing will fail. Elastisearch should be now listen on 9200 for processing HTTP request, we can use CURL to get the response. Ensure that it returns with cluster name as “graylog2”
curl -X GET http://localhost:9200
{
"status" : 200,
"name" : "Pistol",
"cluster_name" : "graylog2",
"version" : {
"number" : "1.7.1",
"build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
"build_timestamp" : "2015-07-29T09:54:16Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
Optional: Use the following command to check the Elasticsearch cluster health, you must get a cluster status as “green” for graylog to work.
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "graylog2",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
MongoDB
MongoDB is available in deb format and same can be downloaded from the official website. Add the following repository information on the system to install MongoDB. Before that we must import public key.
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
Add repository by creating the /etc/apt/sources.list.d/mongodb-org-3.0.list list file using the command.
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
Update repository cache.
apt-get update
Install MongoDB using the following command.
apt-get install mongodb-org
Start the MongoDB service and enable it to start automatically during the system start-up.
service mongod start
ln -s /usr/bin/mongod /etc/init.d/mongod
Create a symbolic link
update-rc.d mongod defaults
Graylog2
Graylog-server accepts and process the log messages, also spawns the RESTAPI for the requests that comes from graylog-web-interface. Download the latest version of graylog from graylog.org,
Use the following command to install graylog2 repository.
wget https://packages.graylog2.org/repo/packages/graylog-1.2-repository-ubuntu14.04_latest.deb
dpkg -i graylog-1.2-repository-ubuntu14.04_latest.deb
Install https suppport and update the repository cache.
apt-get install apt-transport-https
apt-get update
Install Graylog server using following command.
apt-get install graylog-server
Edit the server.conf file.
nano /etc/graylog/server/server.conf
Configure the following variables in the above file.
Set a secret to secure the user passwords, use the following command to generate a secret, use at least 64 character’s.
pwgen -N 1 -s 96
OH9wXpsNZVBA8R5vJQSnkhTB1qDOjCxAh3aE3LvXddtfDlZlKYEyGS24BJAiIxI0sbSTSPovTTnhLkkrUvhSSxodTlzDi5gP
If you get a “pwgen: command not found“, use the following command to install pwgen.
apt-get install pwgen
Place the secret.
password_secret = OH9wXpsNZVBA8R5vJQSnkhTB1qDOjCxAh3aE3LvXddtfDlZlKYEyGS24BJAiIxI0sbSTSPovTTnhLkkrUvhSSxodTlzDi5gP
Next is to set a hash password for the root user (not to be confused with system user, root user of graylog is admin). You will use this password for login into the web interface, admin’s password can not be changed using web interface, must edit this variable to set.
Replace “yourpassword” with the choice of your’s.
echo -n yourpassword | sha256sum
e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951
Place the hash password.
root_password_sha2 = e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951
You can setup email address root (admin) user.
root_email = "cyber.web@gmail.com"
Set timezone of root (admin) user.
root_timezone = UTC
Graylog will try to find the Elasticsearch nodes automatically, it uses multicast mode for the same. But when it comes to larger network, it is recommended to use unicast mode which is best suited one for production setups. So add the following two entries to graylog server.conf file, replace ipaddress with live hostname or ipaddress. Multiple hosts can be added with comma separated.
elasticsearch_http_enabled = false elasticsearch_discovery_zen_ping_unicast_hosts = ipaddress:9300
Set only one master node by defining the below variable, default setting is true, you must set it as a false to make the particular node as a slave. Master node performs some periodic tasks that slave won’t perform.
is_master = true
The following variable sets the number of log messages to keep per index, it is recommended to have several smaller indices instead of larger ones.
elasticsearch_max_docs_per_index = 20000000
The following parameter defines to have total number of indices, if the this number is reached old index will be deleted.
elasticsearch_max_number_of_indices = 20
Shards setting is really depends on the number of nodes in the Elasticsearch cluster, if you have only one node, set it as 1.
elasticsearch_shards = 1
The number of replicas for your indices, if you have only one node in Elasticsearch cluster; set it as 0.
elasticsearch_replicas = 0
Restart Graylog service.
service graylog-server restart
Enable auto start of graylog server service during system startup.
update-rc.d graylog-server defaults
You can check out the server startup logs, it will be useful for you to troubleshoot graylog in case of any issue.
tailf /var/log/graylog-server/server.log
On successful start of graylog-server, you should get the following message in the log file.
2015-09-17T09:35:22.895+02:00 INFO [ServerBootstrap] Graylog server up and running.
Install Graylog web interface:
To configure graylog-web-interface, you must have at least one graylog-server node. Install Graylog web interface using “apt-get”.
apt-get install graylog-web
Edit the configuration file and set the following parameters.
nano /etc/graylog/web/web.conf
This is the list of graylog-server nodes, you can add multiple nodes, separate by commas.
graylog2-server.uris="http://127.0.0.1:12900/"
Set the application scret and can be generated using pwgen -N 1 -s 96.
application.secret="sNXyFf6B4Au3GqSlZwq7En86xp10JimdxxYiLtpptOejX6tIUpUE4DGRJOrcMj07wcK0wugPaapvzEzCYinEWj7BOtHXVl5Z"
Set Web interface timezone.
Timezone="Europe/Tallinn"
Restart the gralog-web-interface using following command,
service graylog-web restart
Enable auto start of web interface service during system startup.
update-rc.d graylog-web defaults
Add log forwarding to ryslog.
nano /etc/rsyslog.d/50-default.conf *.*@192.168.56.200:5555
Monitoring Solutions
- Netdata
- Nagios
Netdata Monitoring
Linus Distribution: Debian Linux and its derivatives (including Ubuntu, Mint)
Install nessesary packages:
Install the packages for having a basic netdata installation (system monitoring and many applications, without mysql / mariadb, postgres, named, hardware sensors and SNMP):
curl -Ss 'https://raw.githubusercontent.com/firehol/netdata-demo-site/master/install-required-packages.sh' >/tmp/kickstart.sh && bash /tmp/kickstart.sh netdata
Install all the required packages for monitoring everything netdata can monitor:
curl -Ss 'https://raw.githubusercontent.com/firehol/netdata-demo-site/master/install-required-packages.sh' >/tmp/kickstart.sh && bash /tmp/kickstart.sh netdata-all
apt-get install zlib1g-dev uuid-dev libmnl-dev gcc make git autoconf autoconf-archive autogen automake pkg-config curl
Install Netdata
Download it - the directory 'netdata' will be created
git clone https://github.com/firehol/netdata.git --depth=1 cd netdata
Build it, install it, start it
./netdata-installer.sh
starting netdata at boot
Copy the netdata startup file to /etc/init.d
cp system/netdata-lsb /etc/init.d/netdata
Make sure it is executable
chmod +x /etc/init.d/netdata
Enable it
update-rc.d netdata defaults
Access to web interface
http://<ipaddress>:19999
Referenced : https://github.com/firehol/netdata/wiki/Installation
Nagios Monitoring
Choosing Nagios implementation
Nagios core and Nagios XI. Depending on what version of Linux you are running you might only be able to support Nagios core. On Ubuntu 15 and ahead PHP5 is unavailable which means that Installing Nagios XI is not possible since it depends on PHP5. In this case you have to install Nagios core using PHP7.:
For Nagios XI
The following steps are for Nagios XI. For Nagios Core install please go below. Install PHP (version 5):
apt-get install php5-common libapache2-mod-php5 php5-cli
Downloading the Latest Release
Make sure to download the Nagios XI installation package to the /tmp directory of the server on which you wish to install it, as shown in the following commands:
cd /tmp
To download the latest stable release, use the following command:
wget http://assets.nagios.com/downloads/nagiosxi/xi-latest.tar.gz
tar xzf xi-latest.tar.gz
cd /tmp/nagiosxi
./fullinstall
Note: It will take around 5 minutes to finish the installation.
You are access the Nagios XI interface by pointing your web browser to:
http://<ipaddress>/nagiosxi OR http://<ipaddress>/nagiosql/install
Note: An empty file named ENABLE_INSTALLER is necessary to create in install/ directory to continue with installation.
Troubleshooting
I got the problem with the given IP after finishing installation, it gave me wrong IP to access. But just ignore it and go to your correct IP of your web server:
http://<ipaddress>
without "/nagiosxi"
It will show the installation GUI and you just need to follow and login after all.
Referenced: https://assets.nagios.com/downloads/nagiosxi/docs/XI_Manual_Installation_Instructions.pdf
Install Plugins
Locate Your Plugin
Visit this website to download plugins: http://exchange.nagios.org.
Install Plugin
Click the Admin menu --> Manage Plugins
Click the Browse button --> click the Upload Plugin button.
Test Your Plugin From The Command Line
cd /usr/local/nagios/libexec ./plugin_name arg1 arg2 ...
- See the document of plugin to get more detail of its arguments
Define A Command
browse to Configure → Core
Configuration Manager → Commands and click Add New.
Enter a Command Name, usually the same as the plugin filename. Next in the Command Line field, enter $USER1$/plugin_name $ARG1$. The $USER1$ macro is replaced with the path to the plugin directory, while the $ARG1$ macro is a placeholder for an argument you want to specify later on a per-service basis. If you want, you can omit $ARG1$ or add more ($ARG2$, $ARG3$, etc.)
Click Save --> Apply Configuration.
Add A Service
In the left-hand menu
Click Services --> click Add New.
- Config Name field, enter the file name where the
configuration will be stored. Next enter a description of the service in the Description field. Associate the service with a host by clicking the Manage Host button.
- Check command field, select the command you defined
above from the dropdown list. If you had any argument placeholders when you defined the command earlier, enter replacement text in the appropriate argument fields.
- Manage Templates button allows you to apply a template
to this service which will save you from having to fill in all the other configuration options, or you can enter the required Check and Alert settings manually.
Save --> Apply Configuration again
Verify Your Service Is Working
Navigate to
Home → Service Detail
Look for your new service in the list. It will probably take a few minutes before it runs its first check.
Referenced: https://assets.nagios.com/downloads/nagiosxi/docs/Managing-Plugins-in-Nagios-XI.pdf
For Nagios Core
Install Nagios on Ubuntu 16.04
Step 1. First make sure that all your system packages are up-to-date by running these following apt-get commands in the terminal.
sudo apt-get update sudo apt-get upgrade
Step 2. Install LAMP (Linux, Apache, MariaDB, PHP) server.
A Ubuntu 16.04 LAMP server is required. If you do not have LAMP installed, you can follow our guide here. Also install all required PHP modules:
apt-get install php7.0 openssl perl make php7.0-gd libgd2-xpm-dev libapache2-mod-php7.0 libperl-dev libssl-dev daemon wget apache2-utils unzip
Step 3. Create users and groups for Nagios.
Now create a new nagios user account and setup a password to this account:
useradd nagios groupadd nagcmd usermod -a -G nagcmd nagios usermod -a -G nagcmd www-data
Step 4. Installing Nagios and plugins.
First thing to do is to go to Nagios’s download page and download the latest stable version of Nagios, At the moment of writing this article it is version 4.1.1:
wget https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.1.1.tar.gz tar -zxvf /tmp/nagios-4.1.1.tar.gz cd /tmp/nagios-4.1.1/
Perform below steps to compile the Nagios from the source code:
./configure --with-nagios-group=nagios --with-command-group=nagcmd --with- httpd_conf=/etc/apache2/sites-enabled/ make all make install make install-init make install-config make install-commandmode make install-webconf
Next steps, Download latest nagios-plugins source and install using following commands:
wget http://www.nagios-plugins.org/download/nagios-plugins-2.1.1.tar.gz tar xzf nagios-plugins-2.1.1.tar.gz cd nagios-plugins-2.1.1 ./configure --with-nagios-user=nagios --with-nagios-group=nagios make make install
Step 5. Configure Nagios.
Edit the /usr/local/nagios/etc/objects/contacts.cfg config file with your favorite editor and change the email address associated with the nagiosadmin contact definition to the address you’d like to use for receiving alerts.
nano /usr/local/nagios/etc/objects/contacts.cfg
Change the email address field to receive the notification:
[...]
define contact{
contact_name nagiosadmin ; Short name of userus
generic-contact ; Inherit default values from generic-contact template (defined above)
alias Nagios Admin ; Full name of useremail
admin@idroot.net ; <<***** CHANGE THIS TO YOUR EMAIL ADDRESS ******
[...]
Step 6. Configure Apache web server for Nagios.
Now create nagios apache2 configuration file:
nano /etc/apache2/sites-enabled/nagios.conf
Edit the following lines if you want to access nagios administrative console from a particular IP series, Here, I want to allow nagios administrative access from 192.168.1.0/24 series only:
[...]
- Comment the following lines ##
- Order allow,deny
- Allow from all
- Uncomment and Change lines as shown below ##
Order deny,allow Deny from all Allow from 127.0.0.1 192.168.1.0/24 [...]
Enable Apache’s rewrite and cgi modules:
sudo a2enmod rewrite sudo a2enmod cgi
Configure Apache authentication:
We need to setup the password for the user nagiosadmin. This username will be used to access the web interface so it is important to remember the password that you will input here. Set the password running the following command and enter the password twice:
# sudo htpasswd -s -c /usr/local/nagios/etc/htpasswd.users nagiosadmin New password: Re-type new password: Adding password for user nagiosadmin
Restart Apache for the changes to take effect:
systemctl restart apache2
Step 7. Verify and Start Nagios service.
Next we have to make Nagios start at boot time, so first verify that the configuration file has no errors running the following command:
sudo /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
And you should get the output:
[...] Checking objects... Checked 8 services. Checked 1 hosts. Checked 1 host groups. Checked 0 service groups. Checked 1 contacts. Checked 1 contact groups. Checked 24 commands. Checked 5 time periods. Checked 0 host escalations. Checked 0 service escalations. Checking for circular paths... Checked 1 hosts Checked 0 service dependencies Checked 0 host dependencies Checked 5 timeperiods Checking global event handlers... Checking obsessive compulsive processor commands... Checking misc settings...
Total Warnings: 0 Total Errors: 0
Things look okay - No serious problems were detected during the pre-flight check [...]
Ubuntu 16.04 uses systemd for starting / stopping all the services, so, we need to create nagios.service file:
nano /etc/systemd/system/nagios.service
Add the following lines:
[Unit] Description=Nagios BindTo=network.target
[Install] WantedBy=multi-user.target
[Service] User=nagios Group=nagios Type=simple ExecStart=/usr/local/nagios/bin/nagios /usr/local/nagios/etc/nagios.cfg
Enable Nagios to start automatically at system startup:
systemctl enable /etc/systemd/system/nagios.service
Now, start Nagios service:
systemctl start nagios
Step 8. Accessing Nagios.
Nagios will be available on HTTP port 80 by default. Open your favorite browser and navigate to http://yourdomain.com/install.php or http://server-ip/install.php and complete the required the steps to finish the installation. When prompted for username and password you will introduce the username “nagiosadmin” and the password that you entered in step 6.
Experiences
Graylog
In during installation time, we have found some problems which will impact on our your installation as well.
- The version of Ubuntu and Graylog might conflict each other as well as Graylog's packages ( e.g : java )
Nagios
I have faced some of certain problem with installing Plugin for Nagios. First of all, you need to check if the plugins you want to use is supported for the OS of the server you are running, for example the plugin is written for Windows Server but you are running Linux server, thus it will give you an error about its format or bunch of error which is difficult to determine exactly its trouble.
Furthermore, once you wish to add the plugin as a service. Please pay attention on the "Service Name" and "Manage Host" which might effect to your installation as well.
Summary
Graylog
During the installation Graylogs in both versions of Ubuntu ( 14.04 and 16.0 ). We realized that it had many differences between these versions. Listed above our objectives what we expect to achieve after installing and using Graylog, along with its useful information and its interaction between Administrator (users) with Graylog.
its advantages:
+ Free. + Easy to interact with web interface. + Easy to install with the good support from its sources. + Help Administrator (user) to collect information in during logging and monitoring straightforwardly. + Many useful tools ( plugins ) which supports to work on.
Nagios
The reason I choose Nagios as my monitoring solution is because it is quite fully functions which help people can handle the services on the server for example it points out detail notification of crucial services such as manage users, HTTP, SSH..so forth. I am using a trial version, but it's seem pretty enough for beginner to "hangout" with Monitoring tool at first.
This is my opinion of its advantages:
+ Easy to install with many ways with official guide from Nagios Supports. + Good trial version to barely use for beginner. + There are some useful extensions, plugins which are free and easy to install. + Easy to handle with web interface. + Does not take many space of hard disk.
Another hand, it also has disadvantages:
+ Too expensive for license version. + Lack of plugins development, some of them is quite old as well.
References
http://www.itzgeek.com/how-tos/linux/ubuntu-how-tos/how-to-install-graylog2-on-ubuntu-14-04.html
https://assets.nagios.com/downloads/nagiosxi/docs/Managing-Plugins-in-Nagios-XI.pdf
https://assets.nagios.com/downloads/nagiosxi/docs/XI_Manual_Installation_Instructions.pdf
https://github.com/firehol/netdata/wiki/Installation
http://docs.graylog.org/en/2.1/
https://assets.nagios.com/downloads/nagioscore/docs/Installing_Nagios_Core_From_Source.pdf
