Logging and monitoring with Elastic stack on Ubuntu 16.04
Introduction
Elasticsearch is an open source search engine based on Lucene, developed in java. It provides a distributed and multitenant full-text search engine with an HTTP Dashboard web-interface (Kibana) and JSON documents scheme. Elasticsearch is a scalable search engine that can be used to search for all types of documents, including log file. Elasticsearch is the heart of the 'Elastic Stack' or ELK Stack.
Logstash is an open source tool for managing system events and logs. It provides real-time pipelining to collect data. Logstash will collect the log or data, convert all data into JSON documents, and store them in Elasticsearch.
Kibana is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. It's not just beautiful, but also powerful.
In this HOW TO, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to install 'Elastic beats' on client PCs with Ubuntu 16.04 and CentOS 7 operating system.
Prerequisite
Ubuntu 16.04 64 bit server with 4GB of RAM, hostname - elk-master
Ubuntu 16.04 64 bit client with 1 GB of RAM, hostname - elk-client1
CentOS 7 64 bit client with 1GB of RAM, hostname - elk-client2
Step 1 - Install Java
Java is required for the Elastic stack deployment. Elasticsearch requires Java 8. It is recommended to use the Oracle JDK 1.8. We will install Java 8 from a PPA repository.
Install the new package 'python-software-properties' so we can add a new repository easily with an apt command.
sudo apt-get update
sudo apt-get install -y python-software-properties software-properties-common apt-transport-https
Add the new Java 8 PPA repository with the 'add-apt-repository' command, then update the repository.
sudo add-apt-repository ppa:webupd8team/java -y
sudo apt-get update
Install Java 8 from the PPA webpub8 repository.
sudo apt-get install -y oracle-java8-installer
Step 2 - Install and Configure Elasticsearch
In this step, we will install and configure Elasticsearch. Install Elasticsearch from the elastic repository and configure it to run on the localhost IP.
Before installing Elasticsearch, add the elastic repository key to the server.
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Add elastic 5.x repository to the 'sources.list.d' directory.
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
Update the repository and install Elasticsearch 5.1 with the apt command below.
sudo apt-get update
sudo apt-get install -y elasticsearch
Elasticsearch is installed. Now go to the configuration directory and edit the elasticsaerch.yml configuration file.
cd /etc/elasticsearch/
nano elasticsearch.yml
Enable memory lock for Elasticsearch by removing the comment on line 43. We do this to disable swapping memory for Elasticsearchto avoid overloading the server.
bootstrap.memory_lock: true
In the 'Network' block, uncomment the network.host and http.port lines.
network.host: localhost
http.port: 9200
Save the file and exit nano.
Now edit the elasticsearch service file for the memory lock mlockall configuration.
nano /usr/lib/systemd/system/elasticsearch.service
Uncomment LimitMEMLOCK line.
LimitMEMLOCK=infinity
Save the file and exit.
Edit the default configuration for Elasticsearch in the /etc/default directory.
nano /etc/default/elasticsearch
Uncomment line 60 and make sure the value is 'unlimited'.
MAX_LOCKED_MEMORY=unlimited
Save and exit.
The Elasticsearch configuration is finished. Elasticsearch will run under localhost IP address with port 9200 and we disabled swap memory by enabling mlockall on the Ubuntu server.
Reload the Elasticsearch service file and enable it to run on the boot time, then start the service.
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
