put
This commit is contained in:
tsvetkov
319
elk-k8s-deployment/README.md
Normal file
319
elk-k8s-deployment/README.md
Normal file
@@ -0,0 +1,319 @@
|
||||
# ELK Stack on Kubernetes (Minikube) with Ansible
|
||||
|
||||
A fully automated deployment of the ELK (Elasticsearch, Logstash, Kibana) observability stack on Kubernetes using Minikube, managed by Ansible. Includes NGINX reverse proxy for SSL/TLS termination and Authentik for SSO (OpenID Connect) authentication to Kibana.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ Minikube Cluster │
|
||||
│ │
|
||||
│ ┌─── Namespace: elk ─────────────────────────────────────────────────┐ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────┐ SSL ┌──────────┐ ┌────────────────┐ │ │
|
||||
│ │ │ NGINX │◄────────►│ Kibana │◄───────►│ Elasticsearch │ │ │
|
||||
│ │ │ Proxy │ :443 │ :5601 │ :9200 │ (single │ │ │
|
||||
│ │ │ :30443 │ │ │ │ node) │ │ │
|
||||
│ │ └──────────┘ └──────────┘ └────────────────┘ │ │
|
||||
│ │ │ ▲ ▲ │ │
|
||||
│ │ │ :9443 │ OIDC │ │ │
|
||||
│ │ ▼ │ │ │ │
|
||||
│ │ ┌──────────────┐ ┌─────┴──────┐ ┌─────┴──────┐ │ │
|
||||
│ │ │ Authentik │ │ Authentik │ │ Logstash │ │ │
|
||||
│ │ │ PostgreSQL │◄──►│ Server + │ │ (JDBC │ │ │
|
||||
│ │ │ + Redis │ │ Worker │ │ pipelines)│ │ │
|
||||
│ │ └──────────────┘ └────────────┘ └─────┬──────┘ │ │
|
||||
│ │ │ │ │
|
||||
│ └─────────────────────────────────────────────────────┼─────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─── Namespace: database ──────────────────────────────┼─────────────┐ │
|
||||
│ │ ▼ │ │
|
||||
│ │ ┌────────────┐ │ │
|
||||
│ │ │ MySQL │ │ │
|
||||
│ │ │ 8.4 │ │ │
|
||||
│ │ │ (elkdemo) │ │ │
|
||||
│ │ └────────────┘ │ │
|
||||
│ └────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Tech Stack & Versions
|
||||
|
||||
| Component | Version | Purpose |
|
||||
|:----------------|:----------|:-------------------------------------------|
|
||||
| Ansible | 2.15+ | Deployment automation |
|
||||
| Minikube | Latest | Local Kubernetes cluster |
|
||||
| Elasticsearch | 8.17.0 | Search & analytics engine |
|
||||
| Logstash | 8.17.0 | Data ingestion (JDBC from MySQL) |
|
||||
| Kibana | 8.17.0 | Visualization dashboard |
|
||||
| NGINX | 1.27 | SSL reverse proxy for Kibana |
|
||||
| Authentik | 2024.12 | SSO/OIDC identity provider |
|
||||
| MySQL | 8.4 | Data source with demo database |
|
||||
| PostgreSQL | 16 | Authentik backend database |
|
||||
| Redis | 7 | Authentik cache/message broker |
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
An Ubuntu 22.04+ system (bare-metal or VM) with at minimum 8 GB RAM, 4 CPU cores, and 40 GB disk. Ansible 2.15+, Docker, and internet access are required.
|
||||
|
||||
Install Ansible if not already present:
|
||||
|
||||
```bash
|
||||
sudo apt update && sudo apt install -y software-properties-common
|
||||
sudo add-apt-repository --yes --update ppa:ansible/ansible
|
||||
sudo apt install -y ansible
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
elk-k8s-deployment/
|
||||
├── README.md ← You are here
|
||||
├── ansible/
|
||||
│ ├── inventory/
|
||||
│ │ └── hosts.yml ← Inventory & variables
|
||||
│ └── playbooks/
|
||||
│ ├── deploy.yml ← Main deployment playbook
|
||||
│ └── teardown.yml ← Cleanup playbook
|
||||
├── k8s/
|
||||
│ ├── namespaces/
|
||||
│ │ └── namespaces.yaml ← elk + database namespaces
|
||||
│ ├── mysql/
|
||||
│ │ ├── configmap.yaml ← Init SQL with dummy data
|
||||
│ │ ├── secret.yaml ← MySQL credentials
|
||||
│ │ └── deployment.yaml ← Deployment, PVC, Service
|
||||
│ ├── elasticsearch/
|
||||
│ │ ├── elasticsearch.yaml ← StatefulSet, ConfigMap, Service
|
||||
│ │ └── setup-job.yaml ← Post-deploy user/role setup
|
||||
│ ├── logstash/
|
||||
│ │ └── logstash.yaml ← Deployment, pipelines, Service
|
||||
│ ├── kibana/
|
||||
│ │ └── kibana.yaml ← Deployment, ConfigMap, Service
|
||||
│ ├── nginx/
|
||||
│ │ └── nginx.yaml ← Reverse proxy, SSL, NodePort
|
||||
│ └── authentik/
|
||||
│ └── authentik.yaml ← PG, Redis, Server, Worker
|
||||
├── scripts/
|
||||
│ ├── generate-certs.sh ← CA + component certificates
|
||||
│ └── configure-authentik-oidc.sh ← OIDC provider setup via API
|
||||
└── certs/ ← Generated certificates (gitignored)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step-by-Step Deployment
|
||||
|
||||
### Step 1 — Clone and Prepare
|
||||
|
||||
```bash
|
||||
cd elk-k8s-deployment
|
||||
chmod +x scripts/*.sh
|
||||
```
|
||||
|
||||
Review and adjust variables in `ansible/inventory/hosts.yml` — especially passwords if deploying beyond a lab.
|
||||
|
||||
### Step 2 — Run the Full Deployment
|
||||
|
||||
The single Ansible playbook handles everything from installing prerequisites to starting all services:
|
||||
|
||||
```bash
|
||||
cd ansible
|
||||
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --ask-become-pass
|
||||
```
|
||||
|
||||
This executes the following phases in order:
|
||||
|
||||
1. **Prerequisites** — Installs Docker, kubectl, and Minikube
|
||||
2. **Certificates** — Generates a custom CA and TLS certs for all components
|
||||
3. **Minikube** — Starts the cluster with 4 CPUs, 8 GB RAM
|
||||
4. **Namespaces** — Creates `elk` and `database` namespaces
|
||||
5. **Secrets** — Loads TLS certificates into Kubernetes secrets
|
||||
6. **MySQL** — Deploys MySQL with the `elkdemo` database and sample data
|
||||
7. **Elasticsearch** — Deploys a single-node cluster with X-Pack security
|
||||
8. **ES Setup** — Runs a Job to configure users, roles, and index templates
|
||||
9. **Authentik** — Deploys PostgreSQL, Redis, Authentik server and worker
|
||||
10. **Kibana** — Deploys Kibana configured for OIDC + basic auth
|
||||
11. **Logstash** — Deploys Logstash with two JDBC pipelines from MySQL
|
||||
12. **NGINX** — Deploys the SSL reverse proxy (NodePort 30443)
|
||||
13. **DNS** — Adds `kibana.elk.local` to `/etc/hosts`
|
||||
|
||||
### Step 3 — Configure Authentik OIDC Provider
|
||||
|
||||
After all pods are running, configure the OIDC provider in Authentik:
|
||||
|
||||
```bash
|
||||
# Port-forward Authentik
|
||||
kubectl port-forward svc/authentik-server 9000:80 -n elk &
|
||||
|
||||
# Run the configuration script
|
||||
bash scripts/configure-authentik-oidc.sh http://localhost:9000
|
||||
|
||||
# Stop the port-forward
|
||||
kill %1
|
||||
```
|
||||
|
||||
This creates an OIDC provider with client ID `kibana` and links it to a Kibana application in Authentik.
|
||||
|
||||
### Step 4 — Verify the Deployment
|
||||
|
||||
```bash
|
||||
# Check all pods
|
||||
kubectl get pods -n elk
|
||||
kubectl get pods -n database
|
||||
|
||||
# Expected output — all pods should be Running:
|
||||
# elk namespace: elasticsearch-0, kibana-xxx, logstash-xxx,
|
||||
# nginx-xxx, authentik-server-xxx, authentik-worker-xxx,
|
||||
# authentik-postgresql-xxx, authentik-redis-xxx
|
||||
# database namespace: mysql-xxx
|
||||
```
|
||||
|
||||
### Step 5 — Access the Services
|
||||
|
||||
| Service | URL | Credentials |
|
||||
|:---------------------|:---------------------------------------|:----------------------------------|
|
||||
| Kibana (via NGINX) | https://kibana.elk.local:30443 | elastic / ElasticP@ss2024! |
|
||||
| Authentik Admin | https://authentik.elk.local:30944 | akadmin / AdminP@ss2024! |
|
||||
| Elasticsearch (direct) | `kubectl port-forward svc/elasticsearch 9200:9200 -n elk` | elastic / ElasticP@ss2024! |
|
||||
|
||||
Since self-signed certificates are used, you will need to accept the browser security warning or import `certs/ca.crt` into your browser's trusted certificate store.
|
||||
|
||||
---
|
||||
|
||||
## Running Individual Phases
|
||||
|
||||
To re-run only specific phases, use Ansible tags:
|
||||
|
||||
```bash
|
||||
# Regenerate certificates only
|
||||
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --tags certs,secrets
|
||||
|
||||
# Redeploy only Logstash
|
||||
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --tags logstash
|
||||
|
||||
# Redeploy Elasticsearch and re-run setup
|
||||
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --tags elasticsearch,es-setup
|
||||
```
|
||||
|
||||
Available tags: `prereqs`, `certs`, `minikube`, `namespaces`, `secrets`, `mysql`, `elasticsearch`, `es-setup`, `kibana`, `logstash`, `nginx`, `authentik`, `dns`.
|
||||
|
||||
---
|
||||
|
||||
## Data Pipeline Details
|
||||
|
||||
Logstash runs two JDBC pipelines that pull data from MySQL across namespaces (`mysql.database.svc.cluster.local`):
|
||||
|
||||
**employees pipeline** — Runs every 2 minutes, tracks the `updated_at` column for incremental syncing. Enriches records with a `salary_band` field (junior / mid / senior). Writes to `employees-YYYY.MM` indices.
|
||||
|
||||
**access_logs pipeline** — Runs every minute, tracks the `created_at` column. Enriches records with `status_category` (success/redirect/client_error/server_error) and `performance_tier` (fast/normal/slow/critical). Writes to `access-logs-YYYY.MM` indices.
|
||||
|
||||
Both pipelines use the MySQL Connector/J 8.3.0 JDBC driver, which is downloaded automatically via an init container.
|
||||
|
||||
---
|
||||
|
||||
## Authentik SSO Integration
|
||||
|
||||
The integration uses OpenID Connect between Authentik (Identity Provider) and Elasticsearch/Kibana (Service Provider).
|
||||
|
||||
**Flow**: User visits Kibana → selects "Log in with Authentik SSO" → redirected to Authentik → authenticates → redirected back to Kibana with OIDC token → Elasticsearch validates the token and maps the user to the `superuser` role.
|
||||
|
||||
To adjust the role mapping (e.g., map specific groups to specific roles), edit the `authentik_users` role mapping in Elasticsearch:
|
||||
|
||||
```bash
|
||||
curl --cacert certs/ca.crt -u elastic:ElasticP@ss2024! \
|
||||
-X PUT "https://localhost:9200/_security/role_mapping/authentik_users" \
|
||||
-H "Content-Type: application/json" -d '{
|
||||
"roles": ["kibana_admin"],
|
||||
"enabled": true,
|
||||
"rules": { "field": { "realm.name": "authentik" } }
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Using Custom CA-Signed Certificates
|
||||
|
||||
The included `generate-certs.sh` creates a self-signed CA for lab use. To use certificates signed by your organization's CA:
|
||||
|
||||
1. Place your CA certificate, component certificates, and keys in the `certs/` directory with the same filenames (`ca.crt`, `elasticsearch.crt`, `elasticsearch.key`, etc.)
|
||||
2. Generate the PKCS12 keystore for Elasticsearch:
|
||||
```bash
|
||||
openssl pkcs12 -export -in certs/elasticsearch.crt -inkey certs/elasticsearch.key \
|
||||
-CAfile certs/ca.crt -chain -out certs/elasticsearch.p12 -passout pass:changeit
|
||||
cp certs/elasticsearch.p12 certs/elastic-http.p12
|
||||
```
|
||||
3. Re-run the secrets and affected component phases:
|
||||
```bash
|
||||
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml \
|
||||
--tags secrets,elasticsearch,kibana,logstash,nginx
|
||||
```
|
||||
|
||||
Ensure the SANs (Subject Alternative Names) in your certificates match the service DNS names used in the cluster (e.g., `elasticsearch.elk.svc.cluster.local`).
|
||||
|
||||
---
|
||||
|
||||
## Teardown
|
||||
|
||||
```bash
|
||||
cd ansible
|
||||
|
||||
# Remove all K8s resources but keep Minikube running
|
||||
ansible-playbook -i inventory/hosts.yml playbooks/teardown.yml --ask-become-pass
|
||||
|
||||
# Remove everything including Minikube
|
||||
ansible-playbook -i inventory/hosts.yml playbooks/teardown.yml --ask-become-pass -e stop_minikube=true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Pod stuck in CrashLoopBackOff**
|
||||
```bash
|
||||
kubectl logs <pod-name> -n elk --previous
|
||||
kubectl describe pod <pod-name> -n elk
|
||||
```
|
||||
|
||||
**Elasticsearch fails to start (vm.max_map_count)**
|
||||
```bash
|
||||
# On the host (required for Elasticsearch)
|
||||
sudo sysctl -w vm.max_map_count=262144
|
||||
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
|
||||
```
|
||||
|
||||
**Logstash cannot connect to MySQL**
|
||||
```bash
|
||||
# Verify cross-namespace DNS resolution
|
||||
kubectl run -it --rm debug --image=busybox -n elk -- nslookup mysql.database.svc.cluster.local
|
||||
```
|
||||
|
||||
**Certificate issues**
|
||||
```bash
|
||||
# Verify certificate SANs
|
||||
openssl x509 -in certs/elasticsearch.crt -noout -text | grep -A1 "Subject Alternative Name"
|
||||
```
|
||||
|
||||
**Reset Elasticsearch setup job**
|
||||
```bash
|
||||
kubectl delete job elasticsearch-setup -n elk
|
||||
kubectl apply -f k8s/elasticsearch/setup-job.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Notes
|
||||
|
||||
This deployment uses demonstration credentials. For any non-lab use:
|
||||
|
||||
- Replace all passwords in `ansible/inventory/hosts.yml` and corresponding K8s secrets
|
||||
- Use Ansible Vault to encrypt sensitive variables: `ansible-vault encrypt_string 'password' --name 'elastic_password'`
|
||||
- Use CA-signed certificates from your organization's PKI
|
||||
- Restrict the Elasticsearch `superuser` role mapping to specific OIDC groups
|
||||
- Enable Kubernetes NetworkPolicies to isolate namespace traffic
|
||||
- Set Kubernetes resource quotas appropriate to your cluster
|
||||
Reference in New Issue
Block a user