287 lines
11 KiB
Markdown
287 lines
11 KiB
Markdown
# ELK Stack on Kubernetes (Minikube) with Ansible
|
|
|
|
A fully automated deployment of the ELK (Elasticsearch, Logstash, Kibana) observability stack on Kubernetes using Minikube, managed by Ansible. Includes NGINX reverse proxy for SSL/TLS termination and Authentik for SSO (OpenID Connect) authentication to Kibana.
|
|
|
|
---
|
|
|
|
## Tech Stack & Versions
|
|
|
|
| Component | Version | Purpose |
|
|
|:----------------|:----------|:-------------------------------------------|
|
|
| Ansible | 2.15+ | Deployment automation |
|
|
| Minikube | Latest | Local Kubernetes cluster |
|
|
| Elasticsearch | 8.17.0 | Search & analytics engine |
|
|
| Logstash | 8.17.0 | Data ingestion (JDBC from MySQL) |
|
|
| Kibana | 8.17.0 | Visualization dashboard |
|
|
| NGINX | 1.27 | SSL reverse proxy for Kibana |
|
|
| Authentik | 2024.12 | SSO/OIDC identity provider |
|
|
| MySQL | 8.4 | Data source with demo database |
|
|
| PostgreSQL | 16 | Authentik backend database |
|
|
| Redis | 7 | Authentik cache/message broker |
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
An Ubuntu 22.04+ system (bare-metal or VM) with at minimum 8 GB RAM, 4 CPU cores, and 40 GB disk. Ansible 2.15+, Docker, and internet access are required.
|
|
|
|
Install Ansible if not already present:
|
|
|
|
```bash
|
|
sudo apt update && sudo apt install -y software-properties-common
|
|
sudo add-apt-repository --yes --update ppa:ansible/ansible
|
|
sudo apt install -y ansible
|
|
```
|
|
|
|
---
|
|
|
|
## Project Structure
|
|
|
|
**NOTE:** The ansible folder is available [here](), and the BASH scripts can be found in [this repo]().
|
|
|
|
```
|
|
elk-k8s-deployment/
|
|
├── README.md ← You are here
|
|
├── ansible/
|
|
│ ├── inventory/
|
|
│ │ └── hosts.yml ← Inventory & variables
|
|
│ └── playbooks/
|
|
│ ├── deploy.yml ← Main deployment playbook
|
|
│ └── teardown.yml ← Cleanup playbook
|
|
├── k8s/
|
|
│ ├── namespaces/
|
|
│ │ └── namespaces.yaml ← elk + database namespaces
|
|
│ ├── mysql/
|
|
│ │ ├── configmap.yaml ← Init SQL with dummy data
|
|
│ │ ├── secret.yaml ← MySQL credentials
|
|
│ │ └── deployment.yaml ← Deployment, PVC, Service
|
|
│ ├── elasticsearch/
|
|
│ │ ├── elasticsearch.yaml ← StatefulSet, ConfigMap, Service
|
|
│ │ └── setup-job.yaml ← Post-deploy user/role setup
|
|
│ ├── logstash/
|
|
│ │ └── logstash.yaml ← Deployment, pipelines, Service
|
|
│ ├── kibana/
|
|
│ │ └── kibana.yaml ← Deployment, ConfigMap, Service
|
|
│ ├── nginx/
|
|
│ │ └── nginx.yaml ← Reverse proxy, SSL, NodePort
|
|
│ └── authentik/
|
|
│ └── authentik.yaml ← PG, Redis, Server, Worker
|
|
├── scripts/
|
|
│ ├── generate-certs.sh ← CA + component certificates
|
|
│ └── configure-authentik-oidc.sh ← OIDC provider setup via API
|
|
└── certs/ ← Generated certificates (gitignored)
|
|
```
|
|
|
|
---
|
|
|
|
## Step-by-Step Deployment
|
|
|
|
### Step 1 — Clone and Prepare
|
|
|
|
```bash
|
|
cd elk-k8s-deployment
|
|
chmod +x scripts/*.sh
|
|
```
|
|
|
|
Review and adjust variables in `ansible/inventory/hosts.yml` — especially passwords if deploying beyond a lab.
|
|
|
|
### Step 2 — Run the Full Deployment
|
|
|
|
The single Ansible playbook handles everything from installing prerequisites to starting all services:
|
|
|
|
```bash
|
|
cd ansible
|
|
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --ask-become-pass
|
|
```
|
|
|
|
This executes the following phases in order:
|
|
|
|
1. **Prerequisites** — Installs Docker, kubectl, and Minikube
|
|
2. **Certificates** — Generates a custom CA and TLS certs for all components
|
|
3. **Minikube** — Starts the cluster with 4 CPUs, 8 GB RAM
|
|
4. **Namespaces** — Creates `elk` and `database` namespaces
|
|
5. **Secrets** — Loads TLS certificates into Kubernetes secrets
|
|
6. **MySQL** — Deploys MySQL with the `elkdemo` database and sample data
|
|
7. **Elasticsearch** — Deploys a single-node cluster with X-Pack security
|
|
8. **ES Setup** — Runs a Job to configure users, roles, and index templates
|
|
9. **Authentik** — Deploys PostgreSQL, Redis, Authentik server and worker
|
|
10. **Kibana** — Deploys Kibana configured for OIDC + basic auth
|
|
11. **Logstash** — Deploys Logstash with two JDBC pipelines from MySQL
|
|
12. **NGINX** — Deploys the SSL reverse proxy (NodePort 30443)
|
|
13. **DNS** — Adds `kibana.elk.local` to `/etc/hosts`
|
|
|
|
### Step 3 — Configure Authentik OIDC Provider
|
|
|
|
After all pods are running, configure the OIDC provider in Authentik:
|
|
|
|
```bash
|
|
# Port-forward Authentik
|
|
kubectl port-forward svc/authentik-server 9000:80 -n elk &
|
|
|
|
# Run the configuration script
|
|
bash scripts/configure-authentik-oidc.sh http://localhost:9000
|
|
|
|
# Stop the port-forward
|
|
kill %1
|
|
```
|
|
|
|
This creates an OIDC provider with client ID `kibana` and links it to a Kibana application in Authentik.
|
|
|
|
### Step 4 — Verify the Deployment
|
|
|
|
```bash
|
|
# Check all pods
|
|
kubectl get pods -n elk
|
|
kubectl get pods -n database
|
|
|
|
# Expected output — all pods should be Running:
|
|
# elk namespace: elasticsearch-0, kibana-xxx, logstash-xxx,
|
|
# nginx-xxx, authentik-server-xxx, authentik-worker-xxx,
|
|
# authentik-postgresql-xxx, authentik-redis-xxx
|
|
# database namespace: mysql-xxx
|
|
```
|
|
|
|
### Step 5 — Access the Services
|
|
|
|
| Service | URL | Credentials |
|
|
|:---------------------|:---------------------------------------|:----------------------------------|
|
|
| Kibana (via NGINX) | https://kibana.elk.local:30443 | elastic / ElasticP@ss2024! |
|
|
| Authentik Admin | https://authentik.elk.local:30944 | akadmin / AdminP@ss2024! |
|
|
| Elasticsearch (direct) | `kubectl port-forward svc/elasticsearch 9200:9200 -n elk` | elastic / ElasticP@ss2024! |
|
|
|
|
Since self-signed certificates are used, you will need to accept the browser security warning or import `certs/ca.crt` into your browser's trusted certificate store.
|
|
|
|
---
|
|
|
|
## Running Individual Phases
|
|
|
|
To re-run only specific phases, use Ansible tags:
|
|
|
|
```bash
|
|
# Regenerate certificates only
|
|
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --tags certs,secrets
|
|
|
|
# Redeploy only Logstash
|
|
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --tags logstash
|
|
|
|
# Redeploy Elasticsearch and re-run setup
|
|
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --tags elasticsearch,es-setup
|
|
```
|
|
|
|
Available tags: `prereqs`, `certs`, `minikube`, `namespaces`, `secrets`, `mysql`, `elasticsearch`, `es-setup`, `kibana`, `logstash`, `nginx`, `authentik`, `dns`.
|
|
|
|
---
|
|
|
|
## Data Pipeline Details
|
|
|
|
Logstash runs two JDBC pipelines that pull data from MySQL across namespaces (`mysql.database.svc.cluster.local`):
|
|
|
|
**employees pipeline** — Runs every 2 minutes, tracks the `updated_at` column for incremental syncing. Enriches records with a `salary_band` field (junior / mid / senior). Writes to `employees-YYYY.MM` indices.
|
|
|
|
**access_logs pipeline** — Runs every minute, tracks the `created_at` column. Enriches records with `status_category` (success/redirect/client_error/server_error) and `performance_tier` (fast/normal/slow/critical). Writes to `access-logs-YYYY.MM` indices.
|
|
|
|
Both pipelines use the MySQL Connector/J 8.3.0 JDBC driver, which is downloaded automatically via an init container.
|
|
|
|
---
|
|
|
|
## Authentik SSO Integration
|
|
|
|
The integration uses OpenID Connect between Authentik (Identity Provider) and Elasticsearch/Kibana (Service Provider).
|
|
|
|
**Flow**: User visits Kibana → selects "Log in with Authentik SSO" → redirected to Authentik → authenticates → redirected back to Kibana with OIDC token → Elasticsearch validates the token and maps the user to the `superuser` role.
|
|
|
|
To adjust the role mapping (e.g., map specific groups to specific roles), edit the `authentik_users` role mapping in Elasticsearch:
|
|
|
|
```bash
|
|
curl --cacert certs/ca.crt -u elastic:ElasticP@ss2024! \
|
|
-X PUT "https://localhost:9200/_security/role_mapping/authentik_users" \
|
|
-H "Content-Type: application/json" -d '{
|
|
"roles": ["kibana_admin"],
|
|
"enabled": true,
|
|
"rules": { "field": { "realm.name": "authentik" } }
|
|
}'
|
|
```
|
|
|
|
---
|
|
|
|
## Using Custom CA-Signed Certificates
|
|
|
|
The included `generate-certs.sh` creates a self-signed CA for lab use. To use certificates signed by your organization's CA:
|
|
|
|
1. Place your CA certificate, component certificates, and keys in the `certs/` directory with the same filenames (`ca.crt`, `elasticsearch.crt`, `elasticsearch.key`, etc.)
|
|
2. Generate the PKCS12 keystore for Elasticsearch:
|
|
```bash
|
|
openssl pkcs12 -export -in certs/elasticsearch.crt -inkey certs/elasticsearch.key \
|
|
-CAfile certs/ca.crt -chain -out certs/elasticsearch.p12 -passout pass:changeit
|
|
cp certs/elasticsearch.p12 certs/elastic-http.p12
|
|
```
|
|
3. Re-run the secrets and affected component phases:
|
|
```bash
|
|
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml \
|
|
--tags secrets,elasticsearch,kibana,logstash,nginx
|
|
```
|
|
|
|
Ensure the SANs (Subject Alternative Names) in your certificates match the service DNS names used in the cluster (e.g., `elasticsearch.elk.svc.cluster.local`).
|
|
|
|
---
|
|
|
|
## Teardown
|
|
|
|
```bash
|
|
cd ansible
|
|
|
|
# Remove all K8s resources but keep Minikube running
|
|
ansible-playbook -i inventory/hosts.yml playbooks/teardown.yml --ask-become-pass
|
|
|
|
# Remove everything including Minikube
|
|
ansible-playbook -i inventory/hosts.yml playbooks/teardown.yml --ask-become-pass -e stop_minikube=true
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
**Pod stuck in CrashLoopBackOff**
|
|
```bash
|
|
kubectl logs <pod-name> -n elk --previous
|
|
kubectl describe pod <pod-name> -n elk
|
|
```
|
|
|
|
**Elasticsearch fails to start (vm.max_map_count)**
|
|
```bash
|
|
# On the host (required for Elasticsearch)
|
|
sudo sysctl -w vm.max_map_count=262144
|
|
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
|
|
```
|
|
|
|
**Logstash cannot connect to MySQL**
|
|
```bash
|
|
# Verify cross-namespace DNS resolution
|
|
kubectl run -it --rm debug --image=busybox -n elk -- nslookup mysql.database.svc.cluster.local
|
|
```
|
|
|
|
**Certificate issues**
|
|
```bash
|
|
# Verify certificate SANs
|
|
openssl x509 -in certs/elasticsearch.crt -noout -text | grep -A1 "Subject Alternative Name"
|
|
```
|
|
|
|
**Reset Elasticsearch setup job**
|
|
```bash
|
|
kubectl delete job elasticsearch-setup -n elk
|
|
kubectl apply -f k8s/elasticsearch/setup-job.yaml
|
|
```
|
|
|
|
---
|
|
|
|
## Security Notes
|
|
|
|
This deployment uses demonstration credentials. For any non-lab use:
|
|
|
|
- Replace all passwords in `ansible/inventory/hosts.yml` and corresponding K8s secrets
|
|
- Use Ansible Vault to encrypt sensitive variables: `ansible-vault encrypt_string 'password' --name 'elastic_password'`
|
|
- Use CA-signed certificates from your organization's PKI
|
|
- Restrict the Elasticsearch `superuser` role mapping to specific OIDC groups
|
|
- Enable Kubernetes NetworkPolicies to isolate namespace traffic
|
|
- Set Kubernetes resource quotas appropriate to your cluster
|