ICO wiki - User contributions [en]

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T22:47:27Z

Mdhasan:

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04 and the alternatives of ELK stack Graylogs.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepage</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal

- log in as root user

- Run following command for Openssh

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22
</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

[[File:Ping-elk-master.PNG]]

- Connect to elk-master via ssh by this command here naz is the username for elk-master

<code> ssh naz@elk-master </code>

[[File:Ssh-to-elk-master.PNG]]

If you have follow the above steps for virtual box, now you are ready to install the ELK stack.

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

== Alternatives of ELK stack (Graylog2) ==

Graylog is a powerful tool for logs management that gives you lots of options on analyzing incoming logs from different servers. The way Graylog works is pretty much similar to ELK. In addition to the very Graylog server, which consists of the application and the web interface server, you will also need to have MongoDB and Elasticsearch in order to make the whole stack fully operable. <ref>[https://www.graylog.org/features garylog]</ref>

'''ELK stack vs Graylog2'''

ELK stack and Graylog both almost same in terms of features. Graylog can’t read from syslog files, so you need to send your messages to Graylog directly.<ref>[https://medium.com/jetruby/log-management-graylog-vs-elk-fc93428e0f66 medium.co]</ref>. Logstash’s weak spot has always been performance and resource consumption. Graylog also has built is user permissions management, this feature is not available in Kibana. In Graylog you can also configure it to receive alerts via emails. Graylog uses good-ol [https://en.wikipedia.org/wiki/Representational_state_transfer REST API].

== Summary ==

Right now ELK stack is the most popular log management platform.<ref>[https://logz.io/learn/complete-guide-elk-stack/ number of downloads]</ref>. ELK Stack also very useful tools for database log analysis like Redis and MySQL. <ref>[https://logz.io/blog/redis-performance-monitoring-elk-stack/ redis performance]</ref> Another popular uses of ELK is social media data analysis like Slack or Twitter.<ref>[https://logz.io/blog/interpreting-your-slack-data-with-the-elk-stack/ Slack data]</ref> ELK stack through have some disadvantageous like it requires lot of resources. But the versatile uses of ELK stack makes it one of most flexible data and log management tools. It's competitor Graylog also has quite a number advantages over ELK stack.

== References ==

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T22:35:49Z

Mdhasan: /* Introduction */

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T22:02:05Z

Mdhasan: /* Alternatives of ELK stack (Graylog2) */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04 andwhat is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepage</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal

- log in as root user

- Run following command for Openssh

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22
</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

[[File:Ping-elk-master.PNG]]

- Connect to elk-master via ssh by this command here naz is the username for elk-master

<code> ssh naz@elk-master </code>

[[File:Ssh-to-elk-master.PNG]]

If you have follow the above steps for virtual box, now you are ready to install the ELK stack.

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

== Alternatives of ELK stack (Graylog2) ==

Graylog is a powerful tool for logs management that gives you lots of options on analyzing incoming logs from different servers. The way Graylog works is pretty much similar to ELK. In addition to the very Graylog server, which consists of the application and the web interface server, you will also need to have MongoDB and Elasticsearch in order to make the whole stack fully operable. <ref>[https://www.graylog.org/features garylog]</ref>

'''ELK stack vs Graylog2'''

ELK stack and Graylog both almost same in terms of features. Graylog can’t read from syslog files, so you need to send your messages to Graylog directly.<ref>[https://medium.com/jetruby/log-management-graylog-vs-elk-fc93428e0f66 medium.co]</ref>. Logstash’s weak spot has always been performance and resource consumption. Graylog also has built is user permissions management, this feature is not available in Kibana. In Graylog you can also configure it to receive alerts via emails. Graylog uses good-ol [https://en.wikipedia.org/wiki/Representational_state_transfer REST API].
== References ==

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T21:43:28Z

Mdhasan: /* Alternatives of ELK stack (Graylog2) */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04 andwhat is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepage</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal

- log in as root user

- Run following command for Openssh

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22
</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

[[File:Ping-elk-master.PNG]]

- Connect to elk-master via ssh by this command here naz is the username for elk-master

<code> ssh naz@elk-master </code>

[[File:Ssh-to-elk-master.PNG]]

If you have follow the above steps for virtual box, now you are ready to install the ELK stack.

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

== Alternatives of ELK stack (Graylog2) ==

Parse and enrich logs, wire data, and event data from any data source. Graylog also provides centralized configuration management for 3rd party collectors such as beats, fluentd and nxlog. The processing pipelines allow for greater flexibility in routing, blacklisting, modifying and enriching messages in real-time as they enter Graylog. <ref>[https://www.graylog.org/features garylog]</ref>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T21:38:53Z

Mdhasan:

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04 andwhat is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepage</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal

- log in as root user

- Run following command for Openssh

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22
</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

[[File:Ping-elk-master.PNG]]

- Connect to elk-master via ssh by this command here naz is the username for elk-master

<code> ssh naz@elk-master </code>

[[File:Ssh-to-elk-master.PNG]]

If you have follow the above steps for virtual box, now you are ready to install the ELK stack.

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

== Alternatives of ELK stack (Graylog2) ==

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T21:36:59Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04 andwhat is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepage</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal

- log in as root user

- Run following command for Openssh

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22
</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

[[File:Ping-elk-master.PNG]]

- Connect to elk-master via ssh by this command here naz is the username for elk-master

<code> ssh naz@elk-master </code>

[[File:Ssh-to-elk-master.PNG]]

If you have follow the above steps for virtual box, now you are ready to install the ELK stack.

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T21:35:35Z

Mdhasan: /* Introduction */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04 andwhat is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepage</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal

- log in as root user

- Run following command for Openssh

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22
</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

[[File:Ping-elk-master.PNG]]

- Connect to elk-master via ssh by this command here naz is the username for elk-master

<code> ssh naz@elk-master </code>

[[File:Ssh-to-elk-master.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T21:34:34Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal

- log in as root user

- Run following command for Openssh

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22
</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

[[File:Ping-elk-master.PNG]]

- Connect to elk-master via ssh by this command here naz is the username for elk-master

<code> ssh naz@elk-master </code>

[[File:Ssh-to-elk-master.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T21:33:48Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal
- log in as root user
- Run following command for Openssh

<code>

sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

[[File:Ping-elk-master.PNG]]

- Connect to elk-master via ssh by this command here naz is the username for elk-master

<code> ssh naz@elk-master </code>

[[File:Ssh-to-elk-master.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

File:Ssh-to-elk-master.PNG

2017-06-21T21:33:11Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:39:34Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal
- log in as root user
- Run following command for Openssh

<code>

sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

[[File:Ping-elk-master.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

File:Ping-elk-master.PNG

2017-06-21T20:38:13Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:20:25Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal
- log in as root user
- Run following command for Openssh

<code>

sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:19:39Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

'''Step 2 - Setup static IP for Ubuntu Desktop(elk-client)'''

- Open terminal
- log in as root user
- Run following command for Openssh
<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Go to <code> nano /etc/network/interfaces </code>

- Set static IP (example: 192.168.2.106) like ubuntu server

- restart the Ubuntu desktop

- Now ping ELK-MASTER by running this command

<code> ping elk-master </code>

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:11:27Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client

- Select the virtual Machine

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:10:28Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- For both Server and Client follow this settings

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:08:44Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- Select the Virtual machine Ubuntu server 16.04

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server could have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:08:28Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- Select the Virtual machine Ubuntu server 16.04

- go to settings > network

- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)

- Also Add another network NAT (So the server will have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:08:12Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- Select the Virtual machine Ubuntu server 16.04
- go to settings > network
- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)
- Also Add another network NAT (So the server will have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:07:21Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- Select the Virtual machine Ubuntu server 16.04
- go to settings > network
- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)
- Also Add another network NAT (So the server will have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>
192
<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>[[File:Elk-master-hostd.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

File:Elk-master-hostd.PNG

2017-06-21T20:07:07Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:06:31Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- Select the Virtual machine Ubuntu server 16.04
- go to settings > network
- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)
- Also Add another network NAT (So the server will have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>
192
<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

- Now go to <code>nano /etc/hosts </code>

Add IP addresses of elk-master and elk-client as follows:
<pre>
192.168.2.105 elk-master
192.168.2.106 elk-client
</pre>

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T20:01:49Z

Mdhasan: /* Virtual Box setup */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- Select the Virtual machine Ubuntu server 16.04
- go to settings > network
- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)
- Also Add another network NAT (So the server will have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Become a root user <code> sudo -i </code>

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update

sudo apt-get install openssh-server

sudo ufw allow 22

</code>

- Set up static IP

run this command <code> nano /etc/network/interfaces </code>

Now you set a static IP for primary network as screenshot:

[[File:Static-ip-elk-master.PNG]]

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

File:Static-ip-elk-master.PNG

2017-06-21T20:01:21Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T19:55:21Z

Mdhasan:

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Virtual Box setup ==

As we are going to use Ubuntu server 16.04, which has no Graphical user interface (GUI). It is better to add a GUI client to Ubuntu server and ssh to virtual box setup. Here I will show to how to connect to Ubuntu server from a Ubuntu desktop client via SSH in Virtual box.

'''Step 1 - Virtual box Networking'''

- Select the Virtual machine Ubuntu server 16.04
- go to settings > network
- set Adapter 1 as Internal network (It is better to setup static IP for adapter 1)
- Also Add another network NAT (So the server will have internet access )

'''Step 2 - Setup static IP in Ubuntu server (elk-master)'''

- Enable SSH by running following commands<ref>[https://askubuntu.com/questions/218344/why-am-i-getting-a-port-22-connection-refused-error open port 22 ]</ref>

<code>
sudo apt-get update
sudo apt-get install openssh-server
sudo ufw allow 22
</code>

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T16:57:26Z

Mdhasan: /* Step 2 - Install and Configure Elasticsearch */

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ </code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T16:54:46Z

Mdhasan:

==Introduction==

Logging and monitoring of IT systems is very important part of information security. Proper system logs and monitoring can prevent cyber attacks or service failure. In this wiki we will learn how configure logging and monitoring with ELK stack in Ubuntu server 16.04, what is alternatives (Graylog) of ELK stack and disadvantages of ELK stack.

'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

Following installation steps from 1 to 4 is collected from

[http://www.howtoforge.com HowToFroge.com]<ref>[http://thehackernews.com/2013/10/importance-of-logs-and-log-management.html]</ref>

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T16:36:50Z

Mdhasan:

==Introduction==
'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

[[File:https://www.stratalux.com/wp-content/uploads/2016/06/Will-Migrating-to-the-Cloud-Save-Money-5.png]]

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04.

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T16:35:20Z

Mdhasan:

==Introduction==
'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

[[File:https://www.stratalux.com/wp-content/uploads/2016/06/Will-Migrating-to-the-Cloud-Save-Money-5.png]]

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04.

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T16:33:30Z

Mdhasan: /* Introduction */

==Introduction==
'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

[[File:https://www.stratalux.com/wp-content/uploads/2016/06/Will-Migrating-to-the-Cloud-Save-Money-5.png]]

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04.

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T16:33:14Z

Mdhasan: /* Introduction */

==Introduction==
'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>[[File:https://www.stratalux.com/wp-content/uploads/2016/06/Will-Migrating-to-the-Cloud-Save-Money-5.png]]

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04.

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T16:30:37Z

Mdhasan: /* Introduction */

==Introduction==
'''Elasticsearch''' is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

'''Logstash''' is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

'''Kibana''' is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04.

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T16:29:58Z

Mdhasan: /* Introduction */

==Introduction==
Elasticsearch is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch.<ref>[https://www.elastic.co/products/logstash] Logstash homepahe</ref>

Kibana is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. <ref>[https://www.elastic.co/products/kibana] kibana homepage</ref>

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04 and CentOS 7 operating system.

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T16:25:28Z

Mdhasan:

==Introduction==
Elasticsearch is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant [https://en.wikipedia.org/wiki/Full-text_search full-text search engine] with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

Logstash is an open source tool for managing system events and logs. It provides real-time pipelining to collect data. Logstash will collect the log or data, convert all data into JSON documents, and store them in Elasticsearch.

Kibana is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. It's not just beautiful, but also powerful.

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04 and CentOS 7 operating system.

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-21T16:24:24Z

Mdhasan: /* Introduction */

==Introduction==
Elasticsearch is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant full-text search engine with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.<ref>[https://www.elastic.co/products/elasticsearch] Elastic Search Homepage</ref>

Logstash is an open source tool for managing system events and logs. It provides real-time pipelining to collect data. Logstash will collect the log or data, convert all data into JSON documents, and store them in Elasticsearch.

Kibana is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. It's not just beautiful, but also powerful.

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04 and CentOS 7 operating system.

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T22:25:21Z

Mdhasan: /* Prerequisite */

==Introduction==
Elasticsearch is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant full-text search engine with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.

Logstash is an open source tool for managing system events and logs. It provides real-time pipelining to collect data. Logstash will collect the log or data, convert all data into JSON documents, and store them in Elasticsearch.

Kibana is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. It's not just beautiful, but also powerful.

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04 and CentOS 7 operating system.

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T22:25:04Z

Mdhasan:

==Introduction==
Elasticsearch is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant full-text search engine with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.

Logstash is an open source tool for managing system events and logs. It provides real-time pipelining to collect data. Logstash will collect the log or data, convert all data into JSON documents, and store them in Elasticsearch.

Kibana is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. It's not just beautiful, but also powerful.

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04 and CentOS 7 operating system.

== Prerequisite ==

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master

Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1

CentOS 7 64 bit client with 1GB of RAM, hostname - elk-client2

== Step 1 - Install Java ==

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.

Install the new package ''''python-software-properties'''' so we can add a new repository easily with an apt command.

<code>sudo apt-get update </code>

<code>sudo apt-get install -y python-software-properties software-properties-common apt-transport-https</code>

Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.

<code>sudo add-apt-repository ppa:webupd8team/java -y </code>

<code>sudo apt-get update </code>

Install Java 8 from the PPA webpub8 repository.

<code> sudo apt-get install -y oracle-java8-installer</code>

== Step 2 - Install and Configure Elasticsearch ==

In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.

Before installing Elasticsearch, add the elastic repository key to the server.

<code> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - </code>

Add elastic 5.x repository to the 'sources.list.d' directory.

<code> echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list </code>

Update the repository and install Elasticsearch 5.1 with the apt command below.

<code> sudo apt-get update </code>

<code> sudo apt-get install -y elasticsearch </code>

Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.

<code> cd /etc/elasticsearch/ <code>

<code> nano elasticsearch.yml </code>

Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.

<code> bootstrap.memory_lock: true </code>

In the 'Network' block, uncomment the network.host and http.port lines.

''network.host: localhost''

''http.port: 9200''

Save the file and exit nano.

Now edit the elasticsearch service file for the memory lock mlockall configuration.

<code> nano /usr/lib/systemd/system/elasticsearch.service</code>

Uncomment LimitMEMLOCK line.

''LimitMEMLOCK=infinity''

Save the file and exit.

Edit the default configuration for Elasticsearch in the /etc/default directory.

<code>nano /etc/default/elasticsearch </code>

Uncomment line 60 and make sure the value is 'unlimited'.

''MAX_LOCKED_MEMORY=unlimited''

Save and exit.

The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.

Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.

<code>
sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

</code>

Wait a sec for Elasticsearch to run, then check the open port on the server, make sure the 'state' for port 9200 is 'LISTEN'.

<code>netstat -plntu</code>

Then check the memory lock to ensure that mlockall is enabled. Also check that Elasticsearch is running with the commands below.

<code>
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

curl -XGET 'localhost:9200/?pretty'
</code>

You will see the results below.

== Step 3 - Install and Configure Kibana with Nginx ==

In this step, we will install and configure Kibana behind a Nginx web server. Kibana will listen on the localhost IP address only and Nginx acts as the reverse proxy for the Kibana application.

Install Kibana with this apt command:

<code>sudo apt-get install -y kibana</code>

Now edit the kibana.yml configuration file.

<code> nano /etc/kibana/kibana.yml</code>

Uncomment the server.port, server.hos and elasticsearch.url lines.

<code>
server.port: 5601

server.host: "localhost"

elasticsearch.url: "http://localhost:9200"

</code>

Save the file and exit nano.

Add Kibana to run at boot and start it.

<code>
sudo systemctl enable kibana

sudo systemctl start kibana

</code>

Kibana will run on port 5601 as node application.

<code>
netstat -plntu
</code>

Kibana installation is done, now we need to install Nginx and configure it as a reverse proxy to be able to access Kibana from the public IP address.

Next, install the Nginx and apache2-utils packages.

<code>
sudo apt-get install -y nginx apache2-utils
</code>

Apache2-utils is a package that contains tools for the webserver that work with Nginx as well, we will use htpasswd basic authentication for Kibana.

Nginx has been installed. Now we need to create a new virtual host configuration file in the Nginx sites-available directory. Create a new file 'kibana' with nano.

<code>
cd /etc/nginx/
vim sites-available/kibana
</code>

Paste configuration below.

<pre>

server {
listen 80;

server_name elk-stack.co;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
</pre>

Save the file and exit nano

Create a new basic authentication file with the htpasswd command.

<code>sudo htpasswd -c /etc/nginx/.kibana-user admin</code>

TYPE YOUR PASSWORD

Activate the kibana virtual host by creating a symbolic link from the kibana file in 'sites-available' to the 'sites-enabled' directory.

<code>ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ </code>

Test the nginx configuration and make sure there is no error, then add nginx to run at boot time and restart nginx.

<code>
nginx -t

systemctl enable nginx

systemctl restart nginx

</code>

== Step 4 - Install and Configure Logstash ==

In this step, we will install and configure Logsatash to centralize server logs from client sources with filebeat, then filter and transform all data (Syslog) and transport it to the stash (Elasticsearch).

Install Logstash 5 with the apt command below.

<code>sudo apt-get install -y logstash</code>

Edit the hosts file with nano.

nano /etc/hosts

Add the server IP address and hostname.

''10.0.2.15 elk-master''

Save the hosts file and exit the editor.

Now generate a new SSL certificate file with OpenSSL so the client sources can identify the elastic server.
<code>
cd /etc/logstash/

openssl req -subj /CN=elk-master -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout logstash.key -out logstash.crt

</code>

Change the '/CN' value to the elastic server hostname.

Certificate files will be created in the '/etc/logstash/' directory.

Next, we will create the configuration files for logstash. We will create a configuration file 'filebeat-input.conf' as input file from filebeat, 'syslog-filter.conf' for syslog processing, and then a 'output-elasticsearch.conf' file to define the Elasticsearch output.

Go to the logstash configuration directory and create the new configuration files in the 'conf.d' directory.

<code>
cd /etc/logstash/

nano conf.d/filebeat-input.conf
</code>

Input configuration, paste configuration below.

<pre>
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
</pre>

Save and exit.

Create the syslog-filter.conf file.

<code>nano conf.d/syslog-filter.conf </code>

Paste the configuration below.

<pre>
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
</pre>

We use a filter plugin named 'grok' to parse the syslog files.

Save and exit.

Create the output configuration file 'output-elasticsearch.conf'.

<code> nano conf.d/output-elasticsearch.conf </code>

Paste the configuration below.

<pre>
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
</pre>

Save and exit.

When this is done, add logstash to start at boot time and start the service.
<code>

sudo systemctl enable logstash

sudo systemctl start logstash
</code>

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T22:18:21Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T22:11:44Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T22:10:20Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T22:07:17Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T22:03:02Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T22:02:13Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T20:24:04Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T20:23:37Z

Mdhasan: /* Prerequisite */

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T20:23:23Z

Mdhasan:

Logging and monitoring with Elastic stack on Ubuntu 16.04

2017-06-19T20:23:01Z

Mdhasan: Created page with "=Introduction= Elasticsearch is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant full-text search engine with an HTTP..."

=Introduction=
Elasticsearch is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant full-text search engine with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.

Logstash is an open source tool for managing system events and logs. It provides real-time pipelining to collect data. Logstash will collect the log or data, convert all data into JSON documents, and store them in Elasticsearch.

Kibana is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. It's not just beautiful, but also powerful.

In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04 and CentOS 7 operating system.

= Prerequisite =

Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master
Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1
CentOS 7 64 bit client with 1GB of RAM, hostname - elk-client2

How to Integrate iRedMail Roundcube with Samba4 AD DC

2017-06-14T13:43:25Z

Mdhasan: /* Sources */

How to Integrate iRedMail Roundcube with Samba4 AD DC

Page created by : Md Nazmul Hasan

This is part of Authentication and Authorization course.

= Introduction =

The idea of this how to is to create a fresh SAMBA Active Directory Domain Controller in virtual Box and integrate iRedmail (mail server) installed on CENT OS 7.

= Installing Ubuntu server 16.04 on Virtual Box =

- Create a virtual machine and install Ubuntu server 16.04
[https://www.tecmint.com/installation-of-ubuntu-16-04-server-edition/ click here for installtion guide]

= Virtual box networking =
For installing Samba we need to set static IP Address on ubuntu server 16.04. So we need to add two network adapter on virtual box. In one adapter we need set the static IP and in another on will be in DHCP to get the internet access.

- '''Set the first adapter as host only'''

[[File:Host-only-adapter_in_vm.PNG]]

- '''Set the second adapter as NAT'''

[[File:Nat-adapter-in-vm.PNG]]

- Login to Ubuntu server as root

run <code> ifconfig </code>

Make sure that both of the adapter is visible

[[File:Ifconfig-both-adapter.PNG]]

Don't panic if you cannot see one of the adapter

Now run this command

<code> nano /etc/network/interfaces </code>

Set static IP on primary network interface and second network interface as DHCP

[[File:Network-interface_in_vm.PNG]]

= Installing Samba4 on Ubuntu server 16.04 =

Now install SAMBA as follows:

[https://www.tecmint.com/install-samba4-active-directory-ubuntu/ Follow the instruction here]

= Installing Windows 10 on Virtual box =

Create another virtual machine with two network adapter.

- Host only and NAT

- Install windows 10

- Set static IP on first adapter and same DNS server as Ubuntu server 16.04

[[File:Same-dns-server.PNG]]

- Now join the with windows 10 to Samba Domain as follows

[https://www.tecmint.com/manage-samba4-ad-from-windows-via-rsat/ Follow the instructions]

- Restart and login as a domain user

- Now download and install Remote Server Administration Tools for Windows 10

[https://www.microsoft.com/en-us/download/details.aspx?id=45520 Download RSAT]

- Now open active directory users and computers

- Now you will able to control the Active directory Domain controller from Windows 10 using GUI

= Install CENT OS 7 on virtual box =

Create another virtual machine with two network adapter as like Windows and Ubuntu server

[https://www.tecmint.com/join-centos-7-to-samba4-active-directory/ Follow these instructions to install cent OS 7 and join to Samba domain]

= Install iRedmail on Cent OS=

[https://www.tecmint.com/install-iredmail-on-centos-7-for-samba4-ad-integration/ Follow these instruction to install iRedmail on Cent OS 7]

= Integrate iRedMail Services to Samba4 AD DC =

[https://www.tecmint.com/integrate-iredmail-to-samba4-ad-dc-on-centos-7/ Follow these instructions to integrate iRedmail to Samba4 AD DC]

[https://www.tecmint.com/integrate-iredmail-roundcube-with-samba4-ad-dc/ follow these instructions to Integrate iRedmail Roundcube]

= Conclusion =

Samba AD DC is open source solution to Microsoft Active Directory. iRedmail mail server is great solution for email service.

= Sources =
[https://www.samba.org/ samba]

[https://www.microsoft.com/en-us/download/details.aspx?id=45520 RSAT]

[https://www.ubuntu.com/download/server Ubuntu server]

[http://www.iredmail.org/ iRedmail]

How to Integrate iRedMail Roundcube with Samba4 AD DC

2017-06-14T13:41:23Z

Mdhasan:

How to Integrate iRedMail Roundcube with Samba4 AD DC

2017-06-14T13:39:06Z

Mdhasan: