So, DigitalOcean announced an event called DigitalOcean Kubernetes Challenge, which gives me the opportunity to learn Kubernetes skills before the end of the year.

Though with experience in docker and docker-compose, I’m totally new to kubernetes. My impression of k8s is a bunch of yaml files. In this post, I followed this tutorial, but I did a little twist to deploy ELK stack so that I could learn to debug k8s problems instead of simply doing copy pastas.

Kubernetes Cluster

First we need to create a Kubernetes cluster, go to the dashboard and create a k8s cluster with 3 nodes. After the cluster is created, generate an API token for connecting to the cluster using doctl.

# download and extract doctl bin
wget https://github.com/digitalocean/doctl/releases/download/v1.66.0/doctl-1.66.0-linux-amd64.tar.gz
tar xf doctl-1.66.0-linux-amd64.tar.gz
mv doctl /usr/local/bin
# authenticate
doctl auth init    # paste the api token here
doctl account get  # verify the api token
# get k8s config file
doctl kubernetes cluster kubeconfig save k8s-first-proj

Next step, install kubectl to control the Kubernetes cluster manager. To install kubectl on ArchLinux, simply do pacman -S kubectl.

Elasticsearch

This part consist of a Statefulset and a service. K8s Statefulset is for storing persistent volumes. Normally, data are lost after we shut a container down. So using persistent volumes ensure stored contents would not be discarded after container’s lifetime ended. In this YAML file, be sure not to remove the increase-vm-max-map section. This section ensure the mmapfs directory large enough that Elasticsearch requires. Also, the resource.limits section defines this service can use minimum 0.1 vCPU to maximum 1 vCPU.

K8s service is for exporting ports from the Statefulset. Here we export port 9200 for Kibana and Logstash.

# deploy the statefulset
kubectl create -f elasticsearch_statefulset.yaml
# check the progress of deployment
kubectl rollout status sts/es-cluster
kubectl get pods -n kube-logging    # this should show 3 pods running (es-cluster-{0,1,2})
# deploy the service
kubectl create -f elasticsearch_svc.yaml

Kibana

In this part, we define a deployment template with kibana service connected to ES, and export 5601 port for user interface. To interact with the UI, we need to port-forward 5601 to localhost, then open browser with URL: http://localhost:5601

# deploy
kubectl create -f kibana.yaml
# port-forwarding to localhost
kubectl port-forward svc/kibana 5601:5601 --namespace=kube-logging

Logstash

This part consist of a deployment template and a configmap and a service. The configmap here is for modifying input and output config.

In our setup, Logstash get messages from Filebeat and forward them to Elasticsearch. Logs that passed into logstash get into input-filter-output pipeline. In input section, we collect logs from beats with port 5044. In output section, set the Elasticsearch endpoint with http://elasticsearch:9200

# deploy
kubectl create -f logstash.yaml
# check
kubectl rollout status deployment/logstash -n kube-logging

Filebeat

In this part, we define a DaemonSet, which will be deployed on every node for collecting logs. For discovering logs in pods, we also need to define and assign a ServiceAccount with several permissions.

# deploy
kubectl create -f filebeat.yaml
kubectl rollout status ds/filebeat -n kube-logging

This part gave me a headache. I followed every tutorials I could find and nothing really works. I turned on every debug flag and tried to figure out what’s happening. After that, I determined that everything other than Filebeat works. Then, I found that there are no logs in /var/lib/docker/containers. I came across this GitHub Issue and followed the solution and boom! Everything is running! So I guess if we get stuck on a problem, we have to RTFM before doing anything stupid.

There are some commands that I found useful when I debug this problem:

Check logs: kubectl logs ds/filebeat -n kube-logging

Run commands in pod: kubectl exec (POD | TYPE/NAME) -t -- [COMMAND], like kubectl exec -n kube-logging ds/filebeat -t ls /var/log

Show details: kubectl describe ds filebeat -n kube-logging