Logging - Monitoring C21

Logging and Monitoring with Graylog

Course: Logging and Monitoring - Lecturer: Margus Ernits

Group: Cyber Security Engineering (C21)

Team members: Ender Phan, Kustas Kurval, Sheela Gowry Sumathi Raju, Artur Vincent Kerge

Page created on: October 05, 2016

Abstract

In order to understand how to set up the Graylog service as well as understand its crucial roles. We decided to choose Graylog as our application for logging and monitoring. Below are our objectives which would be expected to achieve later on:

• How to install Graylog on Ubuntu 14.04.

• How to use Graylog to protect servers.

• Upgrading and configuring Graylog at first, and know how to secure Graylog.

• Threats and security during logging.

• How to install Netdata and Monitoring with Netdata.

• How to install Nagios and Monitoring with Nagios.

• Trouble shooting and experiences.

Installation Guide

Ubuntu 14.04

Logging Solutions

Since the Elasticsearch is based on java, we would require to install either openJDK or Oracle JDK. It is recommended to install Oracle JDK, verify the java version by using the following command.

Remove the OpenJDK from the system, if you have it already installed. From now on it is presumed that user has root privileges.

apt-get remove --purge openjdk*

Add repository.

add-apt-repository -y ppa:webupd8team/java

Run the following command to pull the packages information from the newly added repository.

apt-get update

Issue the following command to install Java jdk 1.8.

apt-get -y install oracle-java8-installer

java -version

Output:

Java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

Elasticsearch

Let’s install Elasticsearch, it can be downloaded from the official website.

Download and install GPG signing key.

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Note that the -qO argument is lowercase Quebec followed by capital Oscar.

Save the repository definition to /etc/apt/sources.list.d/elasticsearch.list

echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch.list

Update repository cache and install Elasticsearch

apt-get update && apt-get install elasticsearch

Configure Elasticsearch to start during system startup.

update-rc.d elasticsearch defaults

The only important thing is to set a cluster name as “graylog2“, that is being used by graylog. Now edit the configuration file of Elasticsearch.

nano /etc/elasticsearch/elasticsearch.yml

cluster.name: graylog2

In the same file disable dynamic scripts to avoid remote execution. That can be done by adding the following line:

FOLLOWING LINE GOES INTO GRAYLOG CONF!

script.disable_dynamic: true

Once it is done, we are good to go. Before that, restart the Elasticsearch services to load the modified configuration.

service elasticsearch restart

Wait at least a minute to let the Elasticsearch get fully restarted, otherwise testing will fail. Elastisearch should be now listen on 9200 for processing HTTP request, we can use CURL to get the response. Ensure that it returns with cluster name as “graylog2”

curl -X GET http://localhost:9200

{
  "status" : 200,
  "name" : "Pistol",
  "cluster_name" : "graylog2",
  "version" : {
   "number" : "1.7.1",
   "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
   "build_timestamp" : "2015-07-29T09:54:16Z",
   "build_snapshot" : false,
   "lucene_version" : "4.10.4"
 },
 "tagline" : "You Know, for Search"
}

Optional: Use the following command to check the Elasticsearch cluster health, you must get a cluster status as “green” for graylog to work.

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

{
 "cluster_name" : "graylog2",
 "status" : "green",
 "timed_out" : false,
 "number_of_nodes" : 1,
 "number_of_data_nodes" : 1,
 "active_primary_shards" : 0,
 "active_shards" : 0,
 "relocating_shards" : 0,
 "initializing_shards" : 0,
 "unassigned_shards" : 0,
 "delayed_unassigned_shards" : 0,
 "number_of_pending_tasks" : 0,
 "number_of_in_flight_fetch" : 0
}

MongoDB

MongoDB is available in deb format and same can be downloaded from the official website. Add the following repository information on the system to install MongoDB. Before that we must import public key.

apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10

Add repository by creating the /etc/apt/sources.list.d/mongodb-org-3.0.list list file using the command.

echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list

Update repository cache.

apt-get update

Install MongoDB using the following command.

apt-get install mongodb-org

Start the MongoDB service and enable it to start automatically during the system start-up.

service mongod start

ln -s /usr/bin/mongod /etc/init.d/mongod

Create a symbolic link

update-rc.d mongod defaults

Graylog2

Graylog-server accepts and process the log messages, also spawns the RESTAPI for the requests that comes from graylog-web-interface. Download the latest version of graylog from graylog.org,

Use the following command to install graylog2 repository.

wget https://packages.graylog2.org/repo/packages/graylog-1.2-repository-ubuntu14.04_latest.deb

dpkg -i graylog-1.2-repository-ubuntu14.04_latest.deb

Install https suppport and update the repository cache.

apt-get install apt-transport-https

apt-get update

Install Graylog server using following command.

apt-get install graylog-server 

Edit the server.conf file.

nano /etc/graylog/server/server.conf

Configure the following variables in the above file.

Set a secret to secure the user passwords, use the following command to generate a secret, use at least 64 character’s.

pwgen -N 1 -s 96

OH9wXpsNZVBA8R5vJQSnkhTB1qDOjCxAh3aE3LvXddtfDlZlKYEyGS24BJAiIxI0sbSTSPovTTnhLkkrUvhSSxodTlzDi5gP

If you get a “pwgen: command not found“, use the following command to install pwgen.

apt-get install pwgen

Place the secret.

password_secret = OH9wXpsNZVBA8R5vJQSnkhTB1qDOjCxAh3aE3LvXddtfDlZlKYEyGS24BJAiIxI0sbSTSPovTTnhLkkrUvhSSxodTlzDi5gP

Next is to set a hash password for the root user (not to be confused with system user, root user of graylog is admin). You will use this password for login into the web interface, admin’s password can not be changed using web interface, must edit this variable to set.

Replace “yourpassword” with the choice of your’s.

echo -n yourpassword | sha256sum

e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951

Place the hash password.

root_password_sha2 = e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951

You can setup email address root (admin) user.

root_email = "cyber.web@gmail.com"

Set timezone of root (admin) user.

root_timezone = UTC

Graylog will try to find the Elasticsearch nodes automatically, it uses multicast mode for the same. But when it comes to larger network, it is recommended to use unicast mode which is best suited one for production setups. So add the following two entries to graylog server.conf file, replace ipaddress with live hostname or ipaddress. Multiple hosts can be added with comma separated.

elasticsearch_http_enabled = false
elasticsearch_discovery_zen_ping_unicast_hosts = ipaddress:9300

Set only one master node by defining the below variable, default setting is true, you must set it as a false to make the particular node as a slave. Master node performs some periodic tasks that slave won’t perform.

is_master = true

The following variable sets the number of log messages to keep per index, it is recommended to have several smaller indices instead of larger ones.

elasticsearch_max_docs_per_index = 20000000

The following parameter defines to have total number of indices, if the this number is reached old index will be deleted.

elasticsearch_max_number_of_indices = 20

Shards setting is really depends on the number of nodes in the Elasticsearch cluster, if you have only one node, set it as 1.

elasticsearch_shards = 1

The number of replicas for your indices, if you have only one node in Elasticsearch cluster; set it as 0.

elasticsearch_replicas = 0

Restart Graylog service.

service graylog-server restart

Enable auto start of graylog server service during system startup.

update-rc.d graylog-server defaults

You can check out the server startup logs, it will be useful for you to troubleshoot graylog in case of any issue.

tailf /var/log/graylog-server/server.log

On successful start of graylog-server, you should get the following message in the log file.

2015-09-17T09:35:22.895+02:00 INFO  [ServerBootstrap] Graylog server up and running.

Install Graylog web interface:

To configure graylog-web-interface, you must have at least one graylog-server node. Install Graylog web interface using “apt-get”.

apt-get install graylog-web

Edit the configuration file and set the following parameters.

nano /etc/graylog/web/web.conf

This is the list of graylog-server nodes, you can add multiple nodes, separate by commas.

graylog2-server.uris="http://127.0.0.1:12900/"

Set the application scret and can be generated using pwgen -N 1 -s 96.

application.secret="sNXyFf6B4Au3GqSlZwq7En86xp10JimdxxYiLtpptOejX6tIUpUE4DGRJOrcMj07wcK0wugPaapvzEzCYinEWj7BOtHXVl5Z"

Set Web interface timezone.

Timezone="Europe/Tallinn"

Restart the gralog-web-interface using following command,

service graylog-web restart

Enable auto start of web interface service during system startup.

update-rc.d graylog-web defaults

Add log forwarding to ryslog.

nano /etc/rsyslog.d/50-default.conf
*.*@192.168.56.200:5555

Monitoring Solutions

• Netdata

• Nagios

Netdata Monitoring

Linus Distribution: Debian Linux and its derivatives (including Ubuntu, Mint)

Install nessesary packages:

Install the packages for having a basic netdata installation (system monitoring and many applications, without mysql / mariadb, postgres, named, hardware sensors and SNMP):

 curl -Ss 'https://raw.githubusercontent.com/firehol/netdata-demo-site/master/install-required-packages.sh' >/tmp/kickstart.sh && bash /tmp/kickstart.sh netdata

Install all the required packages for monitoring everything netdata can monitor:

 curl -Ss 'https://raw.githubusercontent.com/firehol/netdata-demo-site/master/install-required-packages.sh' >/tmp/kickstart.sh && bash /tmp/kickstart.sh netdata-all

 apt-get install zlib1g-dev uuid-dev libmnl-dev gcc make git autoconf autoconf-archive autogen automake pkg-config curl

Install Netdata

Download it - the directory 'netdata' will be created

 git clone https://github.com/firehol/netdata.git --depth=1
 cd netdata

Build it, install it, start it

 ./netdata-installer.sh

starting netdata at boot

Copy the netdata startup file to /etc/init.d

 cp system/netdata-lsb /etc/init.d/netdata

Make sure it is executable

 chmod +x /etc/init.d/netdata

Enable it

 update-rc.d netdata defaults

Access to web interface

http://<ipaddress>:19999

Referenced : https://github.com/firehol/netdata/wiki/Installation

Nagios Monitoring

Choosing Nagios implementation

Nagios core and Nagios XI. Depending on what version of Linux you are running you might only be able to support Nagios core. On Ubuntu 15 and ahead PHP5 is unavailable which means that Installing Nagios XI is not possible since it depends on PHP5. In this case you have to install Nagios core using PHP7.:

For Nagios XI

The following steps are for Nagios XI. For Nagios Core install please go below. Install PHP (version 5):

 apt-get install php5-common libapache2-mod-php5 php5-cli

Downloading the Latest Release

Make sure to download the Nagios XI installation package to the /tmp directory of the server on which you wish to install it, as shown in the following commands:

 cd /tmp

To download the latest stable release, use the following command:

 wget http://assets.nagios.com/downloads/nagiosxi/xi-latest.tar.gz

 tar xzf xi-latest.tar.gz

 cd /tmp/nagiosxi

 ./fullinstall

Note: It will take around 5 minutes to finish the installation.

You are access the Nagios XI interface by pointing your web browser to:

 http://<ipaddress>/nagiosxi OR http://<ipaddress>/nagiosql/install

Note: An empty file named ENABLE_INSTALLER is necessary to create in install/ directory to continue with installation.

Troubleshooting

I got the problem with the given IP after finishing installation, it gave me wrong IP to access. But just ignore it and go to your correct IP of your web server:

 http://<ipaddress>

 without "/nagiosxi"

It will show the installation GUI and you just need to follow and login after all.

Referenced: https://assets.nagios.com/downloads/nagiosxi/docs/XI_Manual_Installation_Instructions.pdf

Install Plugins

Locate Your Plugin

Visit this website to download plugins: http://exchange.nagios.org.

Install Plugin

Click the Admin menu --> Manage Plugins 

Click the Browse button --> click the Upload Plugin button.

Test Your Plugin From The Command Line

 cd /usr/local/nagios/libexec
 
 ./plugin_name arg1 arg2 ...

• See the document of plugin to get more detail of its arguments

Define A Command

browse to Configure → Core

Configuration Manager → Commands and click Add New.

Enter a Command Name, usually the same as the plugin filename. Next in the Command Line field, enter $USER1$/plugin_name $ARG1$. The $USER1$ macro is replaced with the path to the plugin directory, while the $ARG1$ macro is a placeholder for an argument you want to specify later on a per-service basis. If you want, you can omit $ARG1$ or add more ($ARG2$, $ARG3$, etc.)

Click Save --> Apply Configuration.

Add A Service

In the left-hand menu

Click Services --> click Add New.

• Config Name field, enter the file name where the

configuration will be stored. Next enter a description of the service in the Description field. Associate the service with a host by clicking the Manage Host button.

• Check command field, select the command you defined

above from the dropdown list. If you had any argument placeholders when you defined the command earlier, enter replacement text in the appropriate argument fields.

• Manage Templates button allows you to apply a template

to this service which will save you from having to fill in all the other configuration options, or you can enter the required Check and Alert settings manually.

Save --> Apply Configuration again

Verify Your Service Is Working

Navigate to

 Home → Service Detail  

Look for your new service in the list. It will probably take a few minutes before it runs its first check.

Referenced: https://assets.nagios.com/downloads/nagiosxi/docs/Managing-Plugins-in-Nagios-XI.pdf

For Nagios Core

Install Nagios on Ubuntu 16.04

Step 1. First make sure that all your system packages are up-to-date by running these following apt-get commands in the terminal.

sudo apt-get update
sudo apt-get upgrade

Step 2. Install LAMP (Linux, Apache, MariaDB, PHP) server.

A Ubuntu 16.04 LAMP server is required. If you do not have LAMP installed, you can follow our guide here. Also install all required PHP modules:

apt-get install php7.0 openssl perl make php7.0-gd libgd2-xpm-dev libapache2-mod-php7.0 libperl-dev libssl-dev daemon wget apache2-utils unzip

Step 3. Create users and groups for Nagios.

Now create a new nagios user account and setup a password to this account:

useradd nagios
groupadd nagcmd
usermod -a -G nagcmd nagios
usermod -a -G nagcmd www-data

Step 4. Installing Nagios and plugins.

First thing to do is to go to Nagios’s download page and download the latest stable version of Nagios, At the moment of writing this article it is version 4.1.1:

wget  https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.1.1.tar.gz
tar -zxvf /tmp/nagios-4.1.1.tar.gz
cd /tmp/nagios-4.1.1/

Perform below steps to compile the Nagios from the source code:

./configure --with-nagios-group=nagios --with-command-group=nagcmd --with-   httpd_conf=/etc/apache2/sites-enabled/
make all
make install
make install-init
make install-config
make install-commandmode
make install-webconf

Next steps, Download latest nagios-plugins source and install using following commands:

wget http://www.nagios-plugins.org/download/nagios-plugins-2.1.1.tar.gz
tar xzf nagios-plugins-2.1.1.tar.gz
cd nagios-plugins-2.1.1
./configure --with-nagios-user=nagios --with-nagios-group=nagios
make
make install

Step 5. Configure Nagios.

Edit the /usr/local/nagios/etc/objects/contacts.cfg config file with your favorite editor and change the email address associated with the nagiosadmin contact definition to the address you’d like to use for receiving alerts.

nano /usr/local/nagios/etc/objects/contacts.cfg

Change the email address field to receive the notification:

[...] define contact{ contact_name nagiosadmin ; Short name of userus generic-contact ; Inherit default values from generic-contact template (defined above) alias Nagios Admin ; Full name of useremail admin@idroot.net ; <<***** CHANGE THIS TO YOUR EMAIL ADDRESS ****** [...]

Step 6. Configure Apache web server for Nagios.

Now create nagios apache2 configuration file:

nano /etc/apache2/sites-enabled/nagios.conf

Edit the following lines if you want to access nagios administrative console from a particular IP series, Here, I want to allow nagios administrative access from 192.168.1.0/24 series only:

[...]

1. 1. Comment the following lines ##
2. Order allow,deny
3. Allow from all

1. 1. Uncomment and Change lines as shown below ##

Order deny,allow Deny from all Allow from 127.0.0.1 192.168.1.0/24 [...]

Enable Apache’s rewrite and cgi modules:

sudo a2enmod rewrite
sudo a2enmod cgi

Configure Apache authentication:

We need to setup the password for the user nagiosadmin. This username will be used to access the web interface so it is important to remember the password that you will input here. Set the password running the following command and enter the password twice:

# sudo htpasswd -s -c /usr/local/nagios/etc/htpasswd.users nagiosadmin
New password:
Re-type new password:
Adding password for user nagiosadmin

Restart Apache for the changes to take effect:

systemctl restart apache2

Step 7. Verify and Start Nagios service.

Next we have to make Nagios start at boot time, so first verify that the configuration file has no errors running the following command:

sudo /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

And you should get the output:

[...]
Checking objects...
   Checked 8 services.
   Checked 1 hosts.
   Checked 1 host groups.
   Checked 0 service groups.
   Checked 1 contacts.
   Checked 1 contact groups.
   Checked 24 commands.
   Checked 5 time periods.
   Checked 0 host escalations.
   Checked 0 service escalations.
Checking for circular paths...
   Checked 1 hosts
   Checked 0 service dependencies
   Checked 0 host dependencies
   Checked 5 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
[...]

Ubuntu 16.04 uses systemd for starting / stopping all the services, so, we need to create nagios.service file:

nano /etc/systemd/system/nagios.service

Add the following lines:

[Unit]
Description=Nagios
BindTo=network.target

[Install]
WantedBy=multi-user.target

[Service]
User=nagios
Group=nagios
Type=simple
ExecStart=/usr/local/nagios/bin/nagios /usr/local/nagios/etc/nagios.cfg

Enable Nagios to start automatically at system startup:

systemctl enable /etc/systemd/system/nagios.service

Now, start Nagios service:

systemctl start nagios

Step 8. Accessing Nagios.

Nagios will be available on HTTP port 80 by default. Open your favorite browser and navigate to http://yourdomain.com/install.php or http://server-ip/install.php and complete the required the steps to finish the installation. When prompted for username and password you will introduce the username “nagiosadmin” and the password that you entered in step 6.

Experiences

Graylog

In during installation time, we have found some problems which will impact on our your installation as well.

- The version of Ubuntu and Graylog might conflict each other as well as Graylog's packages ( e.g : java )

Nagios

I have faced some of certain problem with installing Plugin for Nagios. First of all, you need to check if the plugins you want to use is supported for the OS of the server you are running, for example the plugin is written for Windows Server but you are running Linux server, thus it will give you an error about its format or bunch of error which is difficult to determine exactly its trouble.

Furthermore, once you wish to add the plugin as a service. Please pay attention on the "Service Name" and "Manage Host" which might effect to your installation as well.

Summary

Graylog

During the installation Graylogs in both versions of Ubuntu ( 14.04 and 16.0 ). We realized that it had many differences between these versions. Listed above our objectives what we expect to achieve after installing and using Graylog, along with its useful information and its interaction between Administrator (users) with Graylog.

its advantages:

+ Free.
+ Easy to interact with web interface.
+ Easy to install with the good support from its sources.
+ Help Administrator (user) to collect information in during logging and monitoring straightforwardly.
+ Many useful tools ( plugins ) which supports to work on.

Nagios

The reason I choose Nagios as my monitoring solution is because it is quite fully functions which help people can handle the services on the server for example it points out detail notification of crucial services such as manage users, HTTP, SSH..so forth. I am using a trial version, but it's seem pretty enough for beginner to "hangout" with Monitoring tool at first.

This is my opinion of its advantages:

+ Easy to install with many ways with official guide from Nagios Supports.
+ Good trial version to barely use for beginner.
+ There are some useful extensions, plugins which are free and easy to install.
+ Easy to handle with web interface.
+ Does not take many space of hard disk.

Another hand, it also has disadvantages:

+ Too expensive for license version.
+ Lack of plugins development, some of them is quite old as well.

References

http://www.itzgeek.com/how-tos/linux/ubuntu-how-tos/how-to-install-graylog2-on-ubuntu-14-04.html

https://assets.nagios.com/downloads/nagiosxi/docs/Managing-Plugins-in-Nagios-XI.pdf

https://assets.nagios.com/downloads/nagiosxi/docs/XI_Manual_Installation_Instructions.pdf

https://github.com/firehol/netdata/wiki/Installation

http://docs.graylog.org/en/2.1/

https://assets.nagios.com/downloads/nagioscore/docs/Installing_Nagios_Core_From_Source.pdf

http://idroot.net/linux/install-nagios-ubuntu-16-04/