ELK Stack Production Setup: Elasticsearch, Logstash, Kibana in 2026

You have thirty servers and no single place to search their logs. Grepping journal files over SSH does not scale past a handful of hosts, and correlating events across services becomes guesswork. This ELK stack setup walks through installing Elasticsearch, Logstash and Kibana on a single Ubuntu 24.04 LTS node, ingesting syslog and nginx access logs from remote machines, and building a dashboard that answers questions in seconds. All version 8.x, using only free Elastic license features. The scenario: one dedicated log server (4 CPU, 16 GB RAM minimum), a fleet of application and web servers sending logs over TCP, and retention policies that delete old indices automatically. You will configure heap sizes, write grok patterns, fix the three errors that stop most deployments, and have working ingestion by the fourth section. If you need broader context on centralized logging strategies, see the Linux log management guide.

Install Elasticsearch 8.x on Ubuntu 24.04 LTS

Add the Elastic repository with its GPG key and pin the package to prevent accidental upgrades during routine updates:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
sudo apt install elasticsearch

Elasticsearch writes its main configuration to /etc/elasticsearch/elasticsearch.yml. Set the cluster name, node name and network binding:

cluster.name: production-logs
node.name: log-server-01
network.host: 0.0.0.0
http.port: 9200

Binding to 0.0.0.0 allows connections from any interface; use a specific IP if you want to restrict access. Heap size defaults to 1 GB, which is too small for production. Create /etc/elasticsearch/jvm.options.d/heap.options and set it to half your available RAM, capped at 31 GB (compressed pointers break above that):

-Xms8g
-Xmx8g

On a 16 GB node, 8 GB for Elasticsearch leaves room for Logstash, Kibana and the OS page cache. Enable and start the service:

sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

Elasticsearch 8.x generates a random password for the elastic superuser on first start. Retrieve it from /var/log/elasticsearch/production-logs.log or reset it:

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

Verify the cluster responds:

curl -u elastic:<password> http://localhost:9200/_cluster/health

You should see "status":"green" or "status":"yellow" (yellow is normal for a single node with replicas configured). Common errors: if Elasticsearch refuses to start, check journalctl -u elasticsearch for bind failures (port 9200 already in use, usually a stale process) or bootstrap checks complaining about file descriptors. The default systemd unit sets LimitNOFILE=65535, which is usually sufficient. If you see max virtual memory areas vm.max_map_count [65530] is too low, add vm.max_map_count=262144 to /etc/sysctl.conf and run sudo sysctl -p.

Configure Elasticsearch Cluster Settings and Index Templates

For a single-node deployment, set discovery.type: single-node in elasticsearch.yml to skip cluster bootstrapping. For multi-node clusters, define discovery.seed_hosts with the IPs of master-eligible nodes and cluster.initial_master_nodes with their node names on first start. Restart Elasticsearch after changing discovery settings.

Disable security temporarily to simplify initial testing (re-enable it once ingestion works). Add to elasticsearch.yml:

xpack.security.enabled: false

Restart the service. You can now query without authentication. Create an index template that applies settings to all log indices matching a pattern. This example sets one shard and zero replicas (replicas waste disk on a single node) and attaches an ILM policy:

curl -X PUT "http://localhost:9200/_index_template/logs-template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "index.lifecycle.name": "logs-30day-retention"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text" },
        "host": { "type": "keyword" }
      }
    }
  }
}
'

Define the ILM policy to delete indices older than 30 days:

curl -X PUT "http://localhost:9200/_ilm/policy/logs-30day-retention" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
'

Verify with curl http://localhost:9200/_cat/indices?v (empty until you ingest data) and curl http://localhost:9200/_ilm/policy/logs-30day-retention. The template and policy are now ready; any index matching logs-* inherits these settings and deletes automatically after 30 days.

Install and Configure Logstash for Syslog and Nginx Logs

Install Logstash from the same Elastic repository:

sudo apt install logstash

Logstash reads pipeline definitions from /etc/logstash/conf.d/. Create /etc/logstash/conf.d/logs.conf with input, filter and output sections. The input accepts syslog over TCP on port 5140 and Beats on 5044:

input {
  tcp {
    port => 5140
    type => "syslog"
  }
  beats {
    port => 5044
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_host} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      target => "@timestamp"
    }
  }
  if [fields][log_type] == "nginx" {
    grok {
      match => { "message" => "%{IPORHOST:remote_addr} - %{DATA:remote_user} \[%{HTTPDATE:time_local}\] \"%{WORD:request_method} %{DATA:request_uri} HTTP/%{NUMBER:http_version}\" %{NUMBER:status} %{NUMBER:body_bytes_sent} \"%{DATA:http_referer}\" \"%{DATA:http_user_agent}\"" }
    }
    date {
      match => [ "time_local", "dd/MMM/yyyy:HH:mm:ss Z" ]
      target => "@timestamp"
    }
  }
  if "_grokparsefailure" in [tags] {
    mutate {
      add_field => { "parse_error" => "true" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

The syslog grok pattern matches RFC3164 format. The nginx pattern parses the combined log format; adjust if you use a custom format. The date filter overwrites @timestamp with the log's original timestamp instead of ingestion time. The _grokparsefailure tag marks unparseable lines; add a field so you can filter them in Kibana and fix your patterns.

Set Logstash heap in /etc/logstash/jvm.options.d/heap.options. For a 16 GB node already giving 8 GB to Elasticsearch, allocate 2–4 GB:

-Xms2g
-Xmx2g

Enable and start Logstash:

sudo systemctl enable logstash
sudo systemctl start logstash

Check /var/log/logstash/logstash-plain.log for the line Successfully started Logstash API endpoint and Pipeline started. Common errors: grok parse failures show up as events with the _grokparsefailure tag; test patterns at https://grokdebugger.com before deploying. If you see Connection refused [Errno 111] to Elasticsearch, verify Elasticsearch is listening on the host and port in the output block and that security is disabled or credentials are provided.

Send Logs from Remote Servers to Logstash

Configure rsyslog on each remote Ubuntu or Debian server to forward all logs to Logstash. Create /etc/rsyslog.d/50-elk.conf:

*.* @@logstash-host:5140

The @@ prefix sends over TCP; single @ uses UDP. TCP guarantees delivery but requires Logstash to be reachable. Replace logstash-host with the log server's IP or hostname. Restart rsyslog:

sudo systemctl restart rsyslog

Test forwarding with the logger command:

logger "Test message from $(hostname)"

Check Logstash logs for New connection entries and query Elasticsearch:

curl "http://localhost:9200/logs-*/_search?q=message:Test&pretty"

For nginx access logs, configure the server to write in combined format or use a JSON format that Logstash parses directly. In /etc/nginx/nginx.conf, the default combined format works with the grok pattern shown earlier:

access_log /var/log/nginx/access.log combined;

To ship nginx logs, install Filebeat on the remote host. Add the Elastic repository (same steps as Elasticsearch), then:

sudo apt install filebeat

Edit /etc/filebeat/filebeat.yml. Set the log paths and add a custom field to distinguish nginx logs:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields:
    log_type: nginx

output.logstash:
  hosts: ["logstash-host:5044"]

Start Filebeat:

sudo systemctl enable filebeat
sudo systemctl start filebeat

Verify ingestion by searching Kibana (covered in the next section) or querying Elasticsearch for recent documents:

curl "http://localhost:9200/logs-*/_search?size=5&sort=@timestamp:desc&pretty"

You should see documents with fields like remote_addr, status and request_uri for nginx logs, and syslog_host, syslog_program for syslog. If documents appear but fields are missing, check for _grokparsefailure tags and adjust your grok patterns. For broader guidance on log forwarding strategies, see the Linux Log Management eBook.

Install Kibana and Connect to Elasticsearch

Install Kibana from the Elastic repository:

sudo apt install kibana

Edit /etc/kibana/kibana.yml. Set the server host to bind on all interfaces or a specific IP, and point to your Elasticsearch instance:

server.host: "0.0.0.0"
server.port: 5601
elasticsearch.hosts: ["http://localhost:9200"]

If you re-enabled Elasticsearch security, add credentials for the elastic superuser or create a dedicated Kibana user with appropriate roles:

elasticsearch.username: "elastic"
elasticsearch.password: "your-password-here"

For production, use the enrollment token method. Generate a token on the Elasticsearch host:

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token --scope kibana

Start Kibana and paste the token when prompted in the browser setup wizard, or configure it in kibana.yml. Start the service:

sudo systemctl enable kibana
sudo systemctl start kibana

Kibana takes 30–60 seconds to become ready. Check /var/log/kibana/kibana.log for http server running at http://0.0.0.0:5601. Access http://your-server-ip:5601 in a browser. If Kibana shows "Kibana server is not ready yet", wait longer or check the logs for connection errors. Common issue: Unable to retrieve version information from Elasticsearch nodes means Kibana cannot reach Elasticsearch; verify elasticsearch.hosts matches the actual bind address and port, and check firewall rules if Elasticsearch is on a different host.

Once Kibana loads, navigate to Management → Stack Management → Data Views (called Index Patterns in older versions). Click "Create data view" and set the index pattern to logs-* to match all daily indices. Choose @timestamp as the time field. Save the data view, then open Discover from the main menu. You should see log documents with parsed fields in the left sidebar. If no documents appear, verify the time picker in the upper right covers the period when you sent test logs, and confirm Elasticsearch has data with curl http://localhost:9200/_cat/indices?v. Look for indices named logs-YYYY.MM.DD with non-zero document counts.

Build a Kibana Dashboard for Nginx and Syslog

Open Kibana and navigate to Analytics → Visualize Library. Click "Create visualization" and choose Lens for the fastest workflow. For a log count over time chart, select your logs-* data view, drag @timestamp to the horizontal axis, choose "Date histogram" with an interval of 1 hour, then drag "Count of records" to the vertical axis. Switch the chart type to Area for a filled view. Save as "Log Volume Over Time".

Create a pie chart for top hosts: new visualization, drag host.keyword (or syslog_host.keyword for syslog) to the breakdown dimension, set "Top 10 values", add "Count of records" as the metric. Save as "Top Log Sources". For nginx status codes, build a bar chart with status.keyword on the horizontal axis (top 10 values) and count on the vertical. Save as "Nginx Status Codes".

Build a message table: create a new Lens visualization, choose Table, add columns for @timestamp, host.keyword, status (if present), and message. Sort by @timestamp descending. Save as "Recent Logs". Now create a dashboard: Analytics → Dashboard → Create dashboard. Click "Add from library" and select all four saved visualizations. Arrange them in a grid.

Use the search bar at the top to filter: type status:500 to show only 500 errors, or host:web01 for logs from a specific server. Combine filters with status:500 AND host:web01. Set the time picker to "Last 24 hours" in the upper right. Save the dashboard as "Production Logs Overview". Click the refresh icon and set auto-refresh to 1 minute for a live view. Export the dashboard JSON from the management menu (Stack Management → Saved Objects → Export) and store it in version control alongside your infrastructure code.

Index Lifecycle Management and Retention Policy

ILM moves indices through four phases: hot (actively written and searched), warm (read-only, less frequent queries), cold (rarely searched, can move to cheaper storage) and delete (removed after retention expires). For centralized logging, a simple policy keeps data hot for a week then deletes after 30 days. You already created the logs-30day-retention policy; extend it to include a hot phase with rollover:

curl -X PUT "http://localhost:9200/_ilm/policy/logs-30day-retention" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
'

Rollover creates a new index when the current one reaches 50 GB or 7 days old, whichever comes first. To use rollover, create an index alias and a write index. Modify the Logstash output to write to an alias instead of a date-stamped index:

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logs-default"
    ilm_enabled => true
    ilm_rollover_alias => "logs-default"
  }
}

Create the initial write index manually:

curl -X PUT "http://localhost:9200/logs-default-000001" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "logs-default": {
      "is_write_index": true
    }
  }
}
'

The logs-30day-retention policy applies automatically if the index template includes "index.lifecycle.name": "logs-30day-retention" in its settings (already configured earlier). Force a rollover for testing:

curl -X POST "http://localhost:9200/logs-default/_rollover"

Monitor ILM execution with GET /_ilm/status (should return "operation_mode":"RUNNING") and check individual index progress:

curl "http://localhost:9200/logs-default-000001/_ilm/explain?pretty"

Look for "phase": "hot" and "action": "rollover". Adjust retention based on disk space: check current usage with curl http://localhost:9200/_cat/allocation?v. If you see disk usage above 85%, Elasticsearch enters read-only mode for affected indices; either delete old indices manually or shorten retention. Curator was the old way to delete indices on a schedule; ILM is now preferred and built into Elasticsearch, so skip Curator unless you have legacy automation.

Production Checklist and Next Steps

Before calling this stack production-ready, work through these items:

• Re-enable Elasticsearch security (xpack.security.enabled: true) and configure TLS for node-to-node and HTTP traffic. Generate certificates with /usr/share/elasticsearch/bin/elasticsearch-certutil.
• Create dedicated Elasticsearch users for Logstash and Kibana with minimal required roles instead of using the elastic superuser.
• Set up firewall rules to block external access to ports 9200, 5140, 5044 and 5601; use a VPN or SSH tunnel for remote Kibana access.
• Configure replica shards if you add a second Elasticsearch node; change "number_of_replicas": 0 to 1 in the index template.
• Monitor Elasticsearch cluster health with _cluster/health and disk space with _cat/allocation. Set up alerts in Kibana (Stack Management → Rules and Connectors) for disk usage above 80%.
• Test log ingestion failure scenarios: stop Logstash, send logs, verify they queue in rsyslog or Filebeat and replay when Logstash restarts.
• Document your grok patterns and dashboard configurations in a Git repository. Export dashboards, index templates and ILM policies as JSON.

For larger deployments, add more Elasticsearch data nodes and scale Logstash horizontally behind a load balancer. Consider Elasticsearch hot-warm architecture if you ingest more than 500 GB daily. The Linux Log Management eBook covers multi-datacenter log aggregation, log enrichment pipelines and compliance retention strategies. Pair this stack with Prometheus and Grafana for metrics to complement your log search capability.

Frequently Asked Questions

How much RAM does Elasticsearch need for 100 GB of logs per day?

Plan 16–32 GB of heap (half of total RAM) for 100 GB daily ingestion. A 64 GB server with 31 GB heap handles this comfortably. Heap requirements scale with query concurrency and aggregation complexity more than raw ingestion volume. Monitor heap usage with _nodes/stats and increase if you see frequent garbage collection pauses above 1 second in /var/log/elasticsearch. Add more nodes rather than exceeding 31 GB heap on a single instance.

Can I run ELK stack on a single server or do I need a cluster?

A single server works for small deployments ingesting under 50 GB daily and serving fewer than ten concurrent users. Use discovery.type: single-node and set replicas to zero. For production workloads, run at least three Elasticsearch nodes for quorum and one replica. Split Logstash onto separate hosts if CPU becomes a bottleneck during grok parsing. Kibana can share a node with Elasticsearch until you have dozens of active dashboard users.

What is the difference between Filebeat and Logstash for log collection?

Filebeat is a lightweight shipper that reads files and forwards lines to Logstash or Elasticsearch with minimal parsing. It uses less CPU and memory than Logstash (tens of MB versus gigabytes). Logstash parses, transforms and enriches logs with grok, mutate and other filters. Typical pattern: Filebeat on every server tails log files and sends to a central Logstash cluster that does the heavy parsing before writing to Elasticsearch. For simple forwarding without parsing, Filebeat can write directly to Elasticsearch.

How do I delete old Elasticsearch indices to free disk space?

Use ILM policies to delete automatically after a retention period (covered earlier). For manual deletion, list indices with curl http://localhost:9200/_cat/indices?v and delete by name: curl -X DELETE http://localhost:9200/logs-2025.12.15. Delete multiple indices matching a pattern with curl -X DELETE http://localhost:9200/logs-2025.12.*. Check freed space with _cat/allocation?v. Avoid Curator; it is deprecated in favor of built-in ILM.

Why is Logstash not parsing my logs and how do I debug grok patterns?

Check /var/log/logstash/logstash-plain.log for parsing errors. Search Elasticsearch for documents with the _grokparsefailure tag: curl "http://localhost:9200/logs-*/_search?q=tags:_grokparsefailure". Copy a failing message and test your grok pattern at grokdebugger.com or use the Grok Debugger in Kibana (Dev Tools). Common issues: timestamp format mismatches (add more date patterns to the match array), escaped characters in regex, greedy patterns consuming too much. Add break_on_match => false to try multiple patterns.