Setting up ElasticSearch and Kibana on EC2

Our Goal:

Our basic goal is to set up ElasticSearch (and Kibana) on an EC2 instance and visualize the data in Grafana. We have another EC2 instance where Grafana is running. So, we won’t discuss that here. We will focus on setting up EC2, ElasticSearch, Kibana and properly configure them so that Grafana can connect to ElasticSearch properly.

Launching EC2:

We will launch a t3a.xlarge type EC2 instance with 8GB of instance storage. We will be using ubuntu-bionic-18.04-amd64-server-20200112 (ami-07ebfd5b3428b6f4d) as our OS. From the instance’s security group, add 2 more inbound rules to open ports 7463 and 7469 alongside port 22 that is used for SSH. We will be using these ports to listen for ElasticSearch and Kibana with Nginx.

Installing ElasticSearch:

Download and install the public signing key:

sudo wget -qO — https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

You may need to install the apt-transport-https package on Debian before proceeding:

sudo apt-get install apt-transport-https

Save the repository definition to /etc/apt/sources.list.d/elastic-7.x.list:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

You can install the Elasticsearch Debian package with:

sudo apt-get update && sudo apt-get install elasticsearch

Elasticsearch is not started automatically after installation. How to start and stop Elasticsearch depends on whether your system uses SysV init or systemd (used by newer distributions). You can tell which is being used by running this command:

ps -p 1

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo systemctl daemon-reload 
sudo systemctl enable elasticsearch.service

Elasticsearch can be started and stopped as follows:

sudo systemctl start elasticsearch.service 
sudo systemctl stop elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/. By default, the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file. When systemd logging is enabled, the logging information is available using the journalctl commands:

To tail the journal:

sudo journalctl -f

Installing Kibana:

As we have already set PGP keys and necessary things previously, we can direct to installing Kibana. You can install the Kibana Debian package with:

sudo apt-get update && sudo apt-get install kibana

To configure Kibana to start automatically when the system boots up, run the following commands:

sudo systemctl daemon-reload 
sudo systemctl enable kibana.service

Kibana can be started and stopped as follows:

sudo systemctl start kibana.service 
sudo systemctl stop kibana.service

Basic Auth using Nginx:

To start the process of adding authentication, we’ll install Nginx:

sudo apt install nginx

We’re also going to install apache2-utils to help us create the accounts used with basic authentication:

sudo apt-get install apache2-utils

Next, we’ll create a user account for the basic authentication (I chose pass-admin, but you can, of course, replace this with any user account you’d like):

sudo htpasswd -c /etc/nginx/htpasswd.users pass-admin

After hitting enter, we’ll be prompted to enter and verify a password for the user.

New password: 
Re-type new password: 
Adding password for user pass-admin

We will use -c flag only for the first time. From next time, we will add user using

sudo htpasswd /etc/nginx/htpasswd.users another-admin

Next, we’re going to create an Nginx configuration file:

sudo nano /etc/nginx/conf.d/kibana.conf

Enter the following configuration:

We are asking Nginx to listen to port 7463 for connections to Elasticsearch and port 7469 for connections to Kibana, using basic authentication with the account we created with htpasswd. That’s all there is to it. Restart Nginx:

sudo systemctl restart nginx

Both Elasticsearch and Kibana are now gated with basic authentication. We can verify this using some cURL commands. For Elasticsearch, use:

curl -verbose http://pass-admin:**************@host:7463

You should see the following output:

* Rebuilt URL to: http://pass-admin:**************@host:7463/
* Trying host…
* TCP_NODELAY set
* Connected to host (host) port 7463 (#0)
* Server auth using Basic with user ‘pass-admin’
> GET / HTTP/1.1
> Host: host:7463
> Authorization: Basic ****************************************
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.14.0 (Ubuntu)
< Date: Fri, 03 Apr 2020 08:13:30 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 541
< Connection: keep-alive
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: Authorization,Accept,Origin,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range
< Access-Control-Allow-Methods: GET,POST,OPTIONS,PUT,DELETE,PATCH
< 
{
 “name” : “ip-172–**********”,
 “cluster_name” : “elasticsearch”,
 “cluster_uuid” : “CIyY-O-BQDiD**********”,
 “version” : {
 “number” : “7.6.2”,
 “build_flavor” : “default”,
 “build_type” : “deb”,
 “build_hash” : “ef48eb35cf30adf4db14086e8aabd0**********”,
 “build_date” : “2020–03–26T06:34:37.794943Z”,
 “build_snapshot” : false,
 “lucene_version” : “8.4.0”,
 “minimum_wire_compatibility_version” : “6.8.0”,
 “minimum_index_compatibility_version” : “6.0.0-beta1”
 },
 “tagline” : “You Know, for Search”
}
* Connection #0 to host host left intact

For Kibana:

curl -verbose http://pass-admin:**************@host:7469

And the output:

* Rebuilt URL to: http://pass-admin:**************@host:7469/
* Trying host…
* TCP_NODELAY set
* Connected to host (host) port 7469 (#0)
* Server auth using Basic with user ‘pass-admin’
> GET / HTTP/1.1
> Host: host:7469
> Authorization: Basic****************************
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Server: nginx/1.14.0 (Ubuntu)
< Date: Fri, 03 Apr 2020 08:16:47 GMT
< Content-Length: 0
< Connection: keep-alive
< location: /spaces/enter
< kbn-name: ip-172–**********
< kbn-license-sig: c508c90ececa9d75c47dd8b645fd26476e5f81c2862b9580606d4b**********
< kbn-xpack-sig: 5bb7bc00a4882183403885**********
< cache-control: no-cache
< 
* Connection #0 to host host left intact

References:
1. https://www.elastic.co/guide/en/elasticsearch/reference/7.x/deb.html
2. https://www.elastic.co/guide/en/kibana/current/deb.html
3. https://logz.io/blog/securing-elk-nginx/