Previous article used EFK (ElasticSearch/FluentD/Kibana) as standalone executables installed on Windows OS which is suboptimal in production environments. Each layer of EFK stack can be presented as container in K8 cluster.
Walkthrough below will guide through creation of solutions consisting of following
- ElasticSearch standalone service running on Windows/Linux node (I was unable to make ES run reliably as container with persistent storage)
- ElasticSearch service of type
ExternalNamewhich allows both FluentD and Kibana to find ES instance via DNS query - Kibana container which connects to ES and exposed via NodePort service
- FluentD daemonset
Install ElasticSearch
I used Windows 2019 as host for Elasticsearch and installation is simple and straightforward. Once installed. Create elasticsearch service in K8 of type external name which points to the name of your Windows machine hosting ES installation. I put all logging components into kube-logging namespace. YAML for both is below. In my case my Windows VM is called utility-vm and it’s automatically registered to my internal DNS zone on Azure
apiVersion: v1 kind: Namespace metadata: name: kube-logging --- apiVersion: v1 kind: Service metadata: name: elasticsearch-service namespace: kube-logging spec: type: ExternalName externalName: utilityvm.kubernetes.my
Install Kibana container
My cluster consists only of 2 nodes: 1 Linux master and 1 Windows worker. There is no windows image for Kibana so it has to be run on master node. YAML is below taking into account that it has to be run on Linux only and can tolerate master role. YAML also contains service of type NodePort to expose it for consumption on port 30080
apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: kube-logging
labels:
app: kibana
spec:
type: NodePort
ports:
- port: 5601
nodePort: 30080
selector:
app: kibana
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
namespace: kube-logging
labels:
app: kibana
spec:
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
nodeSelector:
beta.kubernetes.io/os: linux
tolerations:
- key: node-role.kubernetes.io/master
operator: "Equal"
effect: "NoSchedule"
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:7.6.2
env:
- name: ELASTICSEARCH_HOSTS
value: http://elasticsearch-service:9200
ports:
- containerPort: 5601
At this point you shall be able to connect to your http://master1:30080 node and see Kibana interface
Configure fluentd daemonset
As of time of writing this article fluentd only provided Windows image for SAC channel 1903. My nodes are running Windows 2019 since Kubernetes 1.18+ inexplicably do not support SAC channels as of right now. So I can not use that image and have to just rebuild fluentd image locally. Download fluentd Repo. Navigate to folder v1.10/windows and replace Dockerfile with file below and build it with docker build . -t fluent/fluentd:local
FROM mcr.microsoft.com/windows/servercore:ltsc2019
RUN powershell -Command "Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"
RUN choco install -y ruby --version 2.6.5.1 --params "'/InstallDir:C:\ruby26'" \
&& choco install -y msys2 --params "'/NoPath /NoUpdate /InstallDir:C:\ruby26\msys64'"
RUN refreshenv \
&& ridk install 2 3 \
RUN echo gem: --no-document >> C:\ProgramData\gemrc \
&& gem install cool.io -v 1.5.4 --platform ruby \
&& gem install oj -v 3.3.10 \
&& gem install json -v 2.2.0 \
&& gem install fluentd -v 1.10.1 \
&& gem install win32-service -v 1.0.1 \
&& gem install win32-ipc -v 0.7.0 \
&& gem install win32-event -v 0.6.3 \
&& gem install windows-pr -v 1.2.6
RUN powershell -Command "Remove-Item -Force C:\ruby26\lib\ruby\gems\2.6.0\cache\*.gem; Remove-Item -Recurse -Force 'C:\ProgramData\chocolatey'"
COPY fluent.conf /fluent/conf/fluent.conf
ENV FLUENTD_CONF="fluent.conf"
EXPOSE 24224 5140
ENTRYPOINT ["cmd", "/k", "fluentd", "-c", "C:\\fluent\\conf\\fluent.conf"]
Fluentd relies on configuration file to inform runtime how to format log entries. Thanks to Mike Kock article following is slight adaptation to my scenario. Configuration file below parsing Windows pods logs, appends some Kubernetes specific information and forwards to ES
apiVersion: v1
data:
fluentd.conf: |
<match fluent.**>
@type null
</match>
#Target Logs (ex:nginx)
<source>
@type tail
@id in_tail_container_logs
path /var/log/containers/*.log
pos_file /var/log/containers/fluentd-containers.log.pos
tag kubernetes.*
read_from_head false
format json
time_format %Y-%m-%dT%H:%M:%S.%N%Z
</source>
<filter kubernetes.**>
@type kubernetes_metadata
@id filter_kube_metadata
</filter>
<filter kubernetes.**>
@type grep
<exclude>
key log
pattern /Reply/
</exclude>
</filter>
<match kubernetes.**>
@type elasticsearch
@id out_es
@log_level info
include_tag_key true
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'false'}"
#user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
#password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'k8log'}"
logstash_format true
type_name fluentd
request_timeout 20s
reload_on_failure true
reconnect_on_error true
with_transporter_log true
<buffer>
flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '1'}"
flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '10s'}"
chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}"
queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"
retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
retry_forever true
</buffer>
</match>
kind: ConfigMap
metadata:
name: fluentd-configmap
namespace: kube-logging
Actual fluentd runtime is deployed as daemonset within kubernetes to pull logs from localhost and send them to ES based on config file above. YAML for deployment is below. Please note that image have to be prebuilt for this to work on node ahead of time. YAML also contains definitions for service account which are used to connect to API server to pull Kubernetes specific information which is inserted into each log entry identifying additional (Kubernetes specific) fields.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: kube-logging
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: fluentd
namespace: kube-logging
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: kube-logging
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-logging
labels:
app: fluentd
spec:
selector:
matchLabels:
app: fluentd
template:
metadata:
labels:
app: fluentd
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
nodeSelector:
beta.kubernetes.io/os: windows
containers:
- name: fluentd
volumeMounts:
- name: config-volume
mountPath: "c:\\fluent\\conf\\K8\\"
- name: varlog
mountPath: /var/log
- name: progdatacontainers
mountPath: /ProgramData/docker/containers
#FluendD only supply image for 1903 and above, so if running anything below that you have to create Dockerfile yourself and build
image: fluent/fluentd:local
command: ["cmd"]
args: ["/c", "gem install fluent-plugin-elasticsearch fluent-plugin-kubernetes_metadata_filter &", "fluentd", "-c", "C:\\fluent\\conf\\K8\\fluentd.conf"]
#args: ["/c", "gem install fluent-plugin-elasticsearch &", "fluentd", "-c", "C:\\fluent\\conf\\K8\\fluentd.conf"]
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elasticsearch-service.kube-logging.svc.cluster.local"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "http"
- name: FLUENTD_SYSTEMD_CONF
value: disable
resources:
limits:
memory: 512Mi
requests:
cpu: 100m
memory: 200Mi
volumes:
- name: config-volume
configMap:
name: fluentd-configmap
- name: varlog
hostPath:
path: /var/log
- name: progdatacontainers
hostPath:
path: /ProgramData/docker/containers
If everything is deployed properly you shall see following in your kube-logging namespace
gregory@master1:~$ k get all -n kube-logging NAME READY STATUS RESTARTS AGE pod/fluentd-zcxj9 1/1 Running 0 31m pod/kibana-699b99d996-vkd27 1/1 Running 3 44h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch-service ExternalName <none> utilityvm.kubernetes.my <none> 44h service/kibana NodePort 10.103.68.166 <none> 5601:30080/TCP 44h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/fluentd 1 1 1 1 1 beta.kubernetes.io/os=windows 25h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kibana 1/1 1 1 44h NAME DESIRED CURRENT READY AGE replicaset.apps/kibana-699b99d996 1 1 1 44h
And following in default namespace
gregory@master1:~$ k get all NAME READY STATUS RESTARTS AGE pod/win-webserver-8d8dcb548-mrnvp 1/1 Running 2 116m pod/win-webserver-8d8dcb548-vnxzq 1/1 Running 2 116m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 45h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/win-webserver 2/2 2 2 44h NAME DESIRED CURRENT READY AGE replicaset.apps/win-webserver-8d8dcb548 2 2 2 44h
Configure Kibana to live stream logs
Go to kibana WebUI (http://master1:30080 in my case) and click gear icon on bottom and then Index Patterns/Create index pattern. Type k8log-* in Define index pattern
Choose @timestamp as you Time field and choose “Create”
You will see all fields available with logs comming from fluentd and specifically ones kubernetes specific (like kubernetes.container_name, kubernetes.labels.app etc).
To stream logs click on Logs on left side and go to Settings tab. Here you will define what indices you want to appear in streaming log and what fields you want to be shown on screen. Example for mine is below
Save settings and switch to Stream tab which will output live logs along with fields you requested
