Logging&monitoring - Revision history

Ebarrier: /* Setup of Graylog */

2016-10-13T07:04:01Z

Setup of Graylog

Ebarrier: /* Setup of Graylog */

2016-10-10T14:01:47Z

Setup of Graylog

← Older revision		Revision as of 17:01, 10 October 2016
Line 108:		Line 108:
	==== Prerequisites ====		==== Prerequisites ====
	Ubuntu Linux machine, sudo access and some Linux beginner skills are needed.		Ubuntu Linux machine, sudo access and some Linux beginner skills are needed.

	In this tutorial the Ubuntu 16.04 64-bit VPS server will be used since it is the latest LTS.		In this tutorial the Ubuntu 16.04 64-bit VPS server will be used since it is the latest LTS.

	Unfortunately, Graylog cannot be installed simply by using one command apt-get install graylog, because there is some prerequisite applications needed for it to work.		Unfortunately, Graylog cannot be installed simply by using one command apt-get install graylog, because there is some prerequisite applications needed for it to work.

	This tutorial describes the commands and configuration to make the services work together Graylog.		This tutorial describes the commands and configuration to make the services work together Graylog.

	Attention! All links and packages/versions are present to the time of writing this guide it might need to be updated later on.		Attention! All links and packages/versions are present to the time of writing this guide it might need to be updated later on.

Ebarrier: /* Initial Setup for graylog */

2016-10-10T13:53:26Z

Initial Setup for graylog

Ebarrier: /* The best solution: Graylog */

2016-10-10T13:41:58Z

The best solution: Graylog

← Older revision		Revision as of 16:41, 10 October 2016
Line 33:		Line 33:

	Graylog is made of three components:		Graylog is made of three components:
	*Elasticsearch as the documents indexing engine		*[https://www.elastic.co/products/elasticsearch Elasticsearch] as the documents indexing engine
	*Mongodb for database		*[https://www.mongodb.com Mongodb] for database
	*Graylog itself (server and web interface combined) to collect and view logs		*Graylog itself (server and web interface combined) to collect and view logs

Line 44:		Line 44:

	Its disadvantages are:		Its disadvantages are:
	*Designed to work with Linux distributions		*Designed to work with Linux distributions (no good support for others operating systems)

	===Why monitor our servers===		===Why monitor our servers===

Ebarrier: /* Initial Setup for graylog */

2016-10-06T11:52:19Z

Initial Setup for graylog

← Older revision		Revision as of 14:52, 6 October 2016
Line 157:		Line 157:
	<code>sudo /bin/systemctl restart elasticsearch.service</code>		<code>sudo /bin/systemctl restart elasticsearch.service</code>

	It is possible to check if the elasticsearch configured correctly by using the following command:		It is possible to check if the elasticsearch's name is configured correctly by using the following command:

	<code>curl -XGET http://localhost:9200</code>		<code>curl -XGET http://localhost:9200</code>

			Alternatively it is possible to use the following command which will also show the status of Elasticsearch (it must be "green"):
			<code>curl 'localhost:9200/_cat/health?v'</code>

	'''7)''' '''Graylog''' can now be installed if all the above steps were done successfully		'''7)''' '''Graylog''' can now be installed if all the above steps were done successfully

Aovtsinn: /* Securing during logging */

2016-10-06T06:34:51Z

Securing during logging

← Older revision		Revision as of 09:34, 6 October 2016
Line 59:		Line 59:

	===Securing during logging===		===Securing during logging===
	~~What to put there?~~

	=Topology of the Elab system=		=Topology of the Elab system=

Ebarrier: /* Why monitor our servers */

2016-10-05T22:11:19Z

Why monitor our servers

← Older revision		Revision as of 01:11, 6 October 2016
Line 58:		Line 58:
	In the end, monitoring a system can be seen as an insurance policy. It costs money and time, but the money and time it saves is worth it.		In the end, monitoring a system can be seen as an insurance policy. It costs money and time, but the money and time it saves is worth it.

	*'''Securing during logging~~'''~~		===Securing during logging===
			What to put there?

	=Topology of the Elab system=		=Topology of the Elab system=

Ebarrier: /* Introduction */

2016-10-05T22:10:46Z

Introduction

← Older revision		Revision as of 01:10, 6 October 2016
Line 28:		Line 28:
	*detect whether a system is under attack		*detect whether a system is under attack

	===The best solution ~~for logging~~===		===The best solution: Graylog===
	As of today, [https://www.graylog.org Graylog] is the most popular open-source solution for logging and monitoring computer systems before [https://www.elastic.co/webinars/introduction-elk-stack ELK stack].		As of today, [https://www.graylog.org Graylog] is the most popular open-source solution (developed in Java) for logging and monitoring computer systems before [https://www.elastic.co/webinars/introduction-elk-stack ELK stack].
	In the proprietary world, [https://www.splunk.com Splunk] is the leader.		In the proprietary world, [https://www.splunk.com Splunk] is the leader.

Line 42:		Line 42:
	*Good looking interface and dashboards		*Good looking interface and dashboards
	*Supports user management (LDAP)		*Supports user management (LDAP)

			Its disadvantages are:
			*Designed to work with Linux distributions

	===Why monitor our servers===		===Why monitor our servers===

Ebarrier: /* Introduction */

2016-10-05T22:03:30Z

Introduction

← Older revision		Revision as of 01:03, 6 October 2016
Line 16:		Line 16:
	This article introduces the Logging and Monitoring application called '''Graylog'''.		This article introduces the Logging and Monitoring application called '''Graylog'''.

	*'''Logging and monitoring~~'''~~		===Logging and monitoring===
	Logging is the process of keeping a continuous record of an event.		Logging is the process of keeping a continuous record of an event.
	The general rule of logging is: Log everything which is not predictable.		The general rule of logging is: Log everything which is not predictable.
Line 28:		Line 28:
	*detect whether a system is under attack		*detect whether a system is under attack

	*'''The best solution for logging~~'''~~		===The best solution for logging===
	As of today, [https://www.graylog.org Graylog] is the most popular open-source solution for logging and monitoring computer systems before [https://www.elastic.co/webinars/introduction-elk-stack ELK stack].		As of today, [https://www.graylog.org Graylog] is the most popular open-source solution for logging and monitoring computer systems before [https://www.elastic.co/webinars/introduction-elk-stack ELK stack].
	In the proprietary world, [https://www.splunk.com Splunk] is the leader.		In the proprietary world, [https://www.splunk.com Splunk] is the leader.

	~~This article describes Graylog solution.~~
	Graylog is made of three components:		Graylog is made of three components:
	*Elasticsearch as the documents indexing engine		*Elasticsearch as the documents indexing engine
Line 44:		Line 43:
	*Supports user management (LDAP)		*Supports user management (LDAP)

	*'''Why ~~we should logging~~ our servers~~'''~~		===Why monitor our servers===
			There are many reasons why a system administrator would want to monitor its server(s).
			*Prevent undesired events to happen
			Without monitoring, a system administrator will react to a problem only when it has already occurred. Such issue can in the worst case cause a failure of the [http://whatis.techtarget.com/definition/Confidentiality-integrity-and-availability-CIA CIA triad]. It is of course wiser to anticipate such issues and solve the problem before they arise.
			The monitoring system sends alerts that help to identify potential sources of futures failures to avoid.

			*Understand what happened in case of failure
			In the event of a system failure, the monitoring system will give crucial information to determine where, when and how the problems occurred.
			This information makes the debugging process to be much faster and easier.

			In the end, monitoring a system can be seen as an insurance policy. It costs money and time, but the money and time it saves is worth it.

	*'''Securing during logging'''		*'''Securing during logging'''

Ebarrier: /* Aim of this page */

2016-10-05T21:44:12Z

Aim of this page

← Older revision		Revision as of 00:44, 6 October 2016
Line 13:		Line 13:
	‎Last modified: ‎ 05 October 2016		‎Last modified: ‎ 05 October 2016

	= ~~Aim of this page~~ =		= Introduction =
	~~The Aim of this wiki page is to give an introduction to~~ Logging and Monitoring application called Graylog~~, what are the main benefits of it and how to install it on Ubuntu Machine~~.		This article introduces the Logging and Monitoring application called '''Graylog'''.

	*'''Logging and monitoring'''		*'''Logging and monitoring'''

← Older revision		Revision as of 10:04, 13 October 2016
Line 109:		Line 109:
	Ubuntu Linux machine, sudo access and some Linux beginner skills are needed.		Ubuntu Linux machine, sudo access and some Linux beginner skills are needed.

	In this tutorial ~~the~~ Ubuntu 16.04 64-bit ~~VPS server~~ will be used since it is the latest LTS.		In this tutorial, Ubuntu 16.04 64-bit distribution will be used since it is the latest LTS.

	Unfortunately, Graylog cannot be installed simply by using one command ~~apt-get install graylog~~, because there is some prerequisite applications needed for it to work.		Unfortunately, Graylog cannot be installed simply by using one command, because there are some prerequisite applications needed for it to work.

	This tutorial describes the commands and configuration to make the services work together Graylog.		This tutorial describes the commands and configuration to make the services work together Graylog.

	Attention! All links and packages/versions are present to the time of writing this guide it might need to be updated later on.		Attention! All links and packages/versions are present to the time of writing this guide it might need to be updated later on.

			Remember that to modify a configuration file, you must [http://www.howtogeek.com/118389/how-to-comment-out-and-uncomment-lines-in-a-configuration-file uncomment/comment-out some lines].

	=== Step by Step Installation tutorial on Ubuntu 16.04 Linux host machine ===		=== Step by Step Installation tutorial on Ubuntu 16.04 Linux host machine ===
Line 121:		Line 123:

	''Command:'' <code> sudo apt-get update</code>		''Command:'' <code> sudo apt-get update</code>


	'''2)''' Now we can install the setup base packages:		'''2)''' Now we can install the setup base packages:

	''Command:'' <code> apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen </code>		''Command:'' <code> apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen </code>


	'''3)''' '''MongoDB''' is a Free and open-source cross-platform document-oriented database program written in C++,C and JavaScript. Classified as a NoSQL database program in favor of JSON-like documents with dynamic schemas (format BSON) making the integration of data in certain types of applications like Graylog easier and faster.		'''3)''' '''MongoDB''' is a Free and open-source cross-platform document-oriented database program written in C++,C and JavaScript. Classified as a NoSQL database program in favor of JSON-like documents with dynamic schemas (format BSON) making the integration of data in certain types of applications like Graylog easier and faster.
Line 134:		Line 138:
	''Command:'' <code> apt-get install mongodb-server </code>		''Command:'' <code> apt-get install mongodb-server </code>

	'''4)''' '''Elasticsearch''' is a search engine based on Lucene written in Java. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

	Elasticsearch is required for Graylog 2 and higher to work, so it is possible to see the complete installation instructions on the [https://www.elastic.co/guide/en/elasticsearch/reference/2.3/setup-repositories.html#_apt official page]		'''4)''' '''Elasticsearch''' is a search engine based on Lucene written in Java. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

			Elasticsearch is required for Graylog 2 and higher to work, so it is possible to see the complete installation instructions on the [https://www.elastic.co/guide/en/elasticsearch/reference/2.3/setup-repositories.html#_apt official page].
			It is recommended to have at least 2Gb of RAM to run Elasticsearch.

	''Commands:''		''Commands:''
Line 153:		Line 159:
	''Command:'' <code> nano /etc/elasticsearch/elasticsearch.yml</code>		''Command:'' <code> nano /etc/elasticsearch/elasticsearch.yml</code>
	and change <code> cluster.name:elasticsearch</code> to <code> cluster.name: graylog</code>		and change <code> cluster.name:elasticsearch</code> to <code> cluster.name: graylog</code>


	'''6)''' After config was modified, we can now start Elasticsearch:		'''6)''' After config was modified, we can now start Elasticsearch:
Line 168:		Line 175:
	Alternatively it is possible to use the following command which will also show the status of Elasticsearch (it must be "green"):		Alternatively it is possible to use the following command which will also show the status of Elasticsearch (it must be "green"):
	<code>curl 'localhost:9200/_cat/health?v'</code>		<code>curl 'localhost:9200/_cat/health?v'</code>


	'''7)''' '''Graylog''' can now be installed if all the above steps were done successfully		'''7)''' '''Graylog''' can now be installed if all the above steps were done successfully
Line 187:		Line 195:
	<code>echo -n supersecretpassword123 \| sha256sum</code>		<code>echo -n supersecretpassword123 \| sha256sum</code>
	It will generate the sha256 hash of the password which would be needed to copy-pasted and add it to the <code>password_secret</code> and <code>root_password_sha2</code> in the graylog config (<code>/etc/graylog/server/server.conf</code>). This is mandatory to do! Without it Graylog will not start!		It will generate the sha256 hash of the password which would be needed to copy-pasted and add it to the <code>password_secret</code> and <code>root_password_sha2</code> in the graylog config (<code>/etc/graylog/server/server.conf</code>). This is mandatory to do! Without it Graylog will not start!


	'''9)''' In order to connect to Graylog web-interface it is needed to set <code>rest_listen_uri</code> and <code>web_listen_uri</code> to the public hostname or IP address of the machine.		'''9)''' In order to connect to Graylog web-interface it is needed to set <code>rest_listen_uri</code> and <code>web_listen_uri</code> to the public hostname or IP address of the machine.


	'''10)''' Last step is to enable Graylog during the system startup:		'''10)''' Last step is to enable Graylog during the system startup:


	<code>sudo systemctl daemon-reload</code>		<code>sudo systemctl daemon-reload</code>

← Older revision		Revision as of 16:53, 10 October 2016
Line 104:		Line 104:
	*Now that you have an up-to-date installation of Ubuntu 16.04 LTS, you can use <code> do-release-upgrade</code> to upgrade to the Ubuntu 16.04 LTS release.		*Now that you have an up-to-date installation of Ubuntu 16.04 LTS, you can use <code> do-release-upgrade</code> to upgrade to the Ubuntu 16.04 LTS release.

	= ~~Initial~~ Setup ~~for graylog~~=		= Setup of Graylog=

	==== Prerequisites ====		==== Prerequisites ====
	Ubuntu Linux machine, sudo access and some Linux beginner skills are needed.		Ubuntu Linux machine, sudo access and some Linux beginner skills are needed.
	~~The installation of Graylog is not that hard as described on some webpages.~~
	In this tutorial the Ubuntu 16.04 64-bit VPS server will be used since it is the latest LTS.		In this tutorial the Ubuntu 16.04 64-bit VPS server will be used since it is the latest LTS.
	Unfortunately, Graylog cannot be installed simply by using one command apt-get install graylog, because there is some prerequisite applications needed for it to work.		Unfortunately, Graylog cannot be installed simply by using one command apt-get install graylog, because there is some prerequisite applications needed for it to work.
	~~In this~~ tutorial ~~i will describe~~ the commands~~, what is needed~~ and ~~how~~ to ~~configure~~ the services to work together ~~with the~~ Graylog~~, installation is simple!~~		This tutorial describes the commands and configuration to make the services work together Graylog.
	Attention! All links and packages/versions are present to the time of writing this guide it might need to be updated later on.		Attention! All links and packages/versions are present to the time of writing this guide it might need to be updated later on.

	=== Step by Step Installation tutorial on Ubuntu 16.04 Linux host machine ===		=== Step by Step Installation tutorial on Ubuntu 16.04 Linux host machine ===
	'''1)''' It is important to have the latest package lists to update them to get info on the newst versions of packages and their dependencies. So we need to run the following command to update them:		'''1)''' It is important to have the latest package lists to update them to get info on the newst versions of packages and their dependencies. So we need to run the following command to update them:

	''Command:'' <code> sudo apt-get update</code>		''Command:'' <code> sudo apt-get update</code>
Line 151:		Line 152:
	'''6)''' After config was modified, we can now start Elasticsearch:		'''6)''' After config was modified, we can now start Elasticsearch:

	<code>sudo ~~/bin/~~systemctl daemon-reload</code>		<code>sudo systemctl daemon-reload</code>

	<code>sudo ~~/bin/~~systemctl enable elasticsearch.service</code>		<code>sudo systemctl enable elasticsearch.service</code>

	<code>sudo ~~/bin/~~systemctl restart elasticsearch.service</code>		<code>sudo systemctl restart elasticsearch.service</code>

	It is possible to check if the elasticsearch's name is configured correctly by using the following command:		It is possible to check if the elasticsearch's name is configured correctly by using the following command: