Logging - Monitoring C21

Logging and Monitoring with Graylog

Course: Logging and Monitoring - Lecturer: Margus Ernits

Group : Cyber Security Engineering (C21)

Team members: Ender Phan, Kustas Kurval, Sheela Gowry Sumathi Raju, Artur Vincent Kerge

Page created by : October 05, 2016

Abstract

In order to understand how to set up the Graylog service as well as understand its crucial roles. We decided to choose Graylog as our application for Logging and Monitoring. Below are our objectives which would be expected to achieve later on:

- How to install Graylog on Ubuntu 14.04/16.0.

- How to use Graylog to protect servers.

(..more)

Installation Guide

Ubuntu 14.04

Install MongoDB The MongoDB installation is simple and quick. Run the following command to import the MongoDB public GPG key into apt:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10 Create the MongoDB source list:

echo 'deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list Update your apt package database:

sudo apt-get update Install the latest stable version of MongoDB with this command:

sudo apt-get install mongodb-org MongoDB should be up and running now. Let's move on to installing Java 7.

Install Java 7 Elasticsearch requires Java 7, so we will install that now. We will install Oracle Java 7 because that is what is recommended on elasticsearch.org. It should, however, work fine with OpenJDK, if you decide to go that route.

Add the Oracle Java PPA to apt:

sudo add-apt-repository ppa:webupd8team/java Update your apt package database:

sudo apt-get update Install the latest stable version of Oracle Java 7 with this command (and accept the license agreement that pops up):

sudo apt-get install oracle-java7-installer Now that Java 7 is installed, let's install Elasticsearch.

Install Elasticsearch Graylog2 v0.20.2 requires Elasticsearch v.0.90.10. Download and install it with these commands:

cd ~; wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.10.deb sudo dpkg -i elasticsearch-0.90.10.deb We need to change the Elasticsearch cluster.name setting. Open the Elasticsearch configuration file:

sudo vi /etc/elasticsearch/elasticsearch.yml Find the section that specifies cluster.name. Uncomment it, and replace the default value with "graylog2", so it looks like the following:

cluster.name: graylog2 You will also want to restrict outside access to your Elasticsearch instance (port 9200), so outsiders can't read your data or shutdown your Elasticseach cluster through the HTTP API. Find the line that specifies network.bind_host and uncomment it so it looks like this:

network.bind_host: localhost Then add the following line somewhere in the file, to disable dynamic scripts:

script.disable_dynamic: true Save and quit. Next, restart Elasticsearch to put our changes into effect:

sudo service elasticsearch restart After a few seconds, run the following to test that Elasticsearch is running properly:

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' Now that Elasticsearch is up and running, let's install the Graylog2 server.

Install Graylog2 server Now that we have installed the other required software, let's install the Graylog2 server. We will install Graylog2 Server v0.20.2 in /opt. First, download the Graylog2 archive to /opt with this command:

cd /opt; sudo wget https://github.com/Graylog2/graylog2-server/releases/download/0.20.2/graylog2-server-0.20.2.tgz Then extract the archive:

sudo tar xvf graylog2-server-0.20.2.tgz Let's create a symbolic link to the newly created directory, to simplify the directory name:

sudo ln -s graylog2-server-0.20.2 graylog2-server Copy the example configuration file to the proper location, in /etc:

sudo cp /opt/graylog2-server/graylog2.conf.example /etc/graylog2.conf Install pwgen, which we will use to generate password secret keys:

sudo apt-get install pwgen Now we must configure the admin password and secret key. The password secret key is configured in graylog2.conf, by the password_secret parameter. We can generate a random key and insert it into the Graylog2 configuration with the following two commands:

SECRET=$(pwgen -s 96 1) sudo -E sed -i -e 's/password_secret =.*/password_secret = '$SECRET'/' /etc/graylog2.conf The admin password is assigned by creating an shasum of the desired password, and assigning it to the root_password_sha2 parameter in the Graylog2 configuration file. Create shasum of your desired password with the following command, substituting the highlighted "password" with your own. The sed command inserts it into the Graylog2 configuration for you:

PASSWORD=$(echo -n password | shasum -a 256 | awk '{print $1}') sudo -E sed -i -e 's/root_password_sha2 =.*/root_password_sha2 = '$PASSWORD'/' /etc/graylog2.conf Now that the admin password is setup, let's open the Graylog2 configuration to make a few changes:

sudo vi /etc/graylog2.conf You should see that password_secret and root_password_sha2 have random strings to them, because of the commands that you ran in the steps above. Now we will configure the rest_transport_uri, which is how the Graylog2 web interface will communicate with the server. Because we are installing all of the components on a single server, let's set the value to 127.0.0.1, or localhost. Find and uncomment rest_transport_uri, and change it's value so it looks like the following:

rest_transport_uri = http://127.0.0.1:12900/ Next, because we only have one Elasticsearch shard (which is running on this server), we will change the value of elasticsearch_shards to 1:

elasticsearch_shards = 1 Save and quit. Now our Graylog2 server is configured and ready to be started.

Optional: If you want to test it out, run the following command:

sudo java -jar /opt/graylog2-server/graylog2-server.jar --debug You should see a lot of output. Once you see output similar to the following lines, you will know that your Graylog2 server was configured correctly:

2014-06-06 14:16:13,420 INFO : org.graylog2.Core - Started REST API at <http://127.0.0.1:12900/> 2014-06-06 14:16:13,421 INFO : org.graylog2.Main - Graylog2 up and running. Press CTRL-C to kill the test and return to the shell.

Now let's install the Graylog2 init script. Copy graylog2ctl to /etc/init.d:

sudo cp /opt/graylog2-server/bin/graylog2ctl /etc/init.d/graylog2 Update the startup script to put the Graylog2 logs in /var/log and to look for the Graylog2 server JAR file in /opt/graylog2-server by running the two following sed commands:

sudo sed -i -e 's/GRAYLOG2_SERVER_JAR=\${GRAYLOG2_SERVER_JAR:=graylog2-server.jar}/GRAYLOG2_SERVER_JAR=\${GRAYLOG2_SERVER_JAR:=\/opt\/graylog2-server\/graylog2-server.jar}/' /etc/init.d/graylog2 sudo sed -i -e 's/LOG_FILE=\${LOG_FILE:=log\/graylog2-server.log}/LOG_FILE=\${LOG_FILE:=\/var\/log\/graylog2-server.log}/' /etc/init.d/graylog2 Next, install the startup script:

sudo update-rc.d graylog2 defaults Now we can start the Graylog2 server with the service command:

sudo service graylog2 start The next step is to install the Graylog2 web interface. Let's do that now!

Install Graylog2 Web Interface We will download and install the Graylog2 v.0.20.2 web interface in /opt with the following commands:

cd /opt; sudo wget https://github.com/Graylog2/graylog2-web-interface/releases/download/0.20.2/graylog2-web-interface-0.20.2.tgz sudo tar xvf graylog2-web-interface-0.20.2.tgz Let's create a symbolic link to the newly created directory, to simplify the directory name:

sudo ln -s graylog2-web-interface-0.20.2 graylog2-web-interface Next, we want to configure the web interface's secret key, the application.secret parameter in graylog2-web-interface.conf. We will generate another key, as we did with the Graylog2 server configuration, and insert it with sed, like so:

SECRET=$(pwgen -s 96 1) sudo -E sed -i -e 's/application\.secret=""/application\.secret="'$SECRET'"/' /opt/graylog2-web-interface/conf/graylog2-web-interface.conf Now open the web interface configuration file, with this command:

sudo vi /opt/graylog2-web-interface/conf/graylog2-web-interface.conf Now we need to update the web interface's configuration to specify the graylog2-server.uris parameter. This is a comma delimited list of the server REST URIs. Since we only have one Graylog2 server node, the value should match that of rest_listen_uri in the Graylog2 server configuration (i.e. "http://127.0.0.1:12900/").

graylog2-server.uris="http://127.0.0.1:12900/" The Graylog2 web interface is now configured. Let's start it up to test it out:

sudo /opt/graylog2-web-interface-0.20.2/bin/graylog2-web-interface You will know it started properly when you see the following two lines:

[info] play - Application started (Prod) [info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000 Hit CTRL-C to kill the web interface. Now let's install a startup script. You can either create your own, or download one that I created for this tutorial. To download the script to your home directory, use this command:

cd ~; wget https://assets.digitalocean.com/articles/graylog2/graylog2-web Next, you will want to copy it to /etc/init.d, and change its ownership to root and its permissions to 755:

sudo cp ~/graylog2-web /etc/init.d/ sudo chown root:root /etc/init.d/graylog2-web sudo chmod 755 /etc/init.d/graylog2-web Now you can install the web interface init script with this command:

sudo update-rc.d graylog2-web defaults Start the Graylog2 web interface:

sudo service graylog2-web start Now we can use the Graylog2 web interface. Let's do that now.

Configure Graylog2 to Receive syslog messages Log into Graylog2 Web Interface

In your favorite browser, go to the port 9000 of your VPS's public IP address:

http://gl2_public_IP:9000/ You should see a login screen. Enter "admin" as your username and the password the admin password that you set earlier.