Using ElasticSearch/FluentD/Kibana containers for Windows containers logging

Previous article used EFK (ElasticSearch/FluentD/Kibana) as standalone executables installed on Windows OS which is suboptimal in production environments. Each layer of EFK stack can be presented as container in K8 cluster.

Walkthrough below will guide through creation of solutions consisting of following

  1. ElasticSearch standalone service running on Windows/Linux node (I was unable to make ES run reliably as container with persistent storage)
  2. ElasticSearch service of type ExternalName which allows both FluentD and Kibana to find ES instance via DNS query
  3. Kibana container which connects to ES and exposed via NodePort service
  4. FluentD daemonset

Install ElasticSearch

I used Windows 2019 as host for Elasticsearch and installation is simple and straightforward. Once installed. Create elasticsearch service in K8 of type external name which points to the name of your Windows machine hosting ES installation. I put all logging components into kube-logging namespace. YAML for both is below. In my case my Windows VM is called utility-vm and it’s automatically registered to my internal DNS zone on Azure

apiVersion: v1
kind: Namespace
metadata:
 name: kube-logging
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-service
  namespace: kube-logging
spec:
  type: ExternalName
  externalName: utilityvm.kubernetes.my

Install Kibana container

My cluster consists only of 2 nodes: 1 Linux master and 1 Windows worker. There is no windows image for Kibana so it has to be run on master node. YAML is below taking into account that it has to be run on Linux only and can tolerate master role. YAML also contains service of type NodePort to expose it for consumption on port 30080

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: kube-logging
  labels:
    app: kibana
spec:
  type: NodePort
  ports:
  - port: 5601
    nodePort: 30080
  selector:
    app: kibana
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: kube-logging
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      nodeSelector:
        beta.kubernetes.io/os: linux
      tolerations:
         - key: node-role.kubernetes.io/master
           operator: "Equal"
           effect: "NoSchedule"
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.6.2
        env:
          - name: ELASTICSEARCH_HOSTS
            value: http://elasticsearch-service:9200
        ports:
        - containerPort: 5601

At this point you shall be able to connect to your http://master1:30080 node and see Kibana interface

Configure fluentd daemonset

As of time of writing this article fluentd only provided Windows image for SAC channel 1903. My nodes are running Windows 2019 since Kubernetes 1.18+ inexplicably do not support SAC channels as of right now. So I can not use that image and have to just rebuild fluentd image locally. Download fluentd Repo. Navigate to folder v1.10/windows and replace Dockerfile with file below and build it with docker build . -t fluent/fluentd:local

FROM mcr.microsoft.com/windows/servercore:ltsc2019
RUN powershell -Command "Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"
RUN choco install -y ruby --version 2.6.5.1 --params "'/InstallDir:C:\ruby26'" \
&& choco install -y msys2 --params "'/NoPath /NoUpdate /InstallDir:C:\ruby26\msys64'"
RUN refreshenv \
&& ridk install 2 3 \
RUN echo gem: --no-document >> C:\ProgramData\gemrc  \
&& gem install cool.io -v 1.5.4 --platform ruby \
&& gem install oj -v 3.3.10 \
&& gem install json -v 2.2.0 \
&& gem install fluentd -v 1.10.1 \
&& gem install win32-service -v 1.0.1 \
&& gem install win32-ipc -v 0.7.0 \
&& gem install win32-event -v 0.6.3 \
&& gem install windows-pr -v 1.2.6 
RUN powershell -Command "Remove-Item -Force C:\ruby26\lib\ruby\gems\2.6.0\cache\*.gem; Remove-Item -Recurse -Force 'C:\ProgramData\chocolatey'"
COPY fluent.conf /fluent/conf/fluent.conf
ENV FLUENTD_CONF="fluent.conf"
EXPOSE 24224 5140
ENTRYPOINT ["cmd", "/k", "fluentd", "-c", "C:\\fluent\\conf\\fluent.conf"]

Fluentd relies on configuration file to inform runtime how to format log entries. Thanks to Mike Kock article following is slight adaptation to my scenario. Configuration file below parsing Windows pods logs, appends some Kubernetes specific information and forwards to ES

apiVersion: v1
data:
  fluentd.conf: |
    <match fluent.**>
      @type null
    </match>
    #Target Logs (ex:nginx)
    <source>
      @type tail
      @id in_tail_container_logs
      path /var/log/containers/*.log
      pos_file /var/log/containers/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head false
      format json
      time_format %Y-%m-%dT%H:%M:%S.%N%Z
    </source>
    <filter kubernetes.**>
      @type kubernetes_metadata
      @id filter_kube_metadata
    </filter>
    <filter kubernetes.**>
      @type grep
      <exclude>
        key log
        pattern /Reply/
      </exclude>
    </filter>
    <match kubernetes.**>
        @type elasticsearch
        @id out_es
        @log_level info
        include_tag_key true
        host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
        port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
        scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
        ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'false'}"
        #user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
        #password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
        reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
        logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'k8log'}"
        logstash_format true
        type_name fluentd
        request_timeout 20s
        reload_on_failure true
        reconnect_on_error true
        with_transporter_log true
        <buffer>
          flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '1'}"
          flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '10s'}"
          chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}"
          queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"
          retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
          retry_forever true
        </buffer>
    </match>
kind: ConfigMap
metadata:
  name: fluentd-configmap
  namespace: kube-logging

Actual fluentd runtime is deployed as daemonset within kubernetes to pull logs from localhost and send them to ES based on config file above. YAML for deployment is below. Please note that image have to be prebuilt for this to work on node ahead of time. YAML also contains definitions for service account which are used to connect to API server to pull Kubernetes specific information which is inserted into each log entry identifying additional (Kubernetes specific) fields.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-logging
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: kube-logging
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-logging
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-logging
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
     app: fluentd
  template:
    metadata:
      labels:
       app: fluentd
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      nodeSelector:
        beta.kubernetes.io/os: windows
      containers:
      - name: fluentd
        volumeMounts:
         - name: config-volume
           mountPath: "c:\\fluent\\conf\\K8\\"
         - name: varlog
           mountPath: /var/log
         - name: progdatacontainers
           mountPath: /ProgramData/docker/containers
#FluendD only supply image for 1903 and above, so if running anything below that you have to create Dockerfile yourself and build
        image: fluent/fluentd:local
        command: ["cmd"]
        args: ["/c", "gem install fluent-plugin-elasticsearch fluent-plugin-kubernetes_metadata_filter &", "fluentd", "-c", "C:\\fluent\\conf\\K8\\fluentd.conf"]
        #args: ["/c", "gem install fluent-plugin-elasticsearch &", "fluentd", "-c", "C:\\fluent\\conf\\K8\\fluentd.conf"]
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch-service.kube-logging.svc.cluster.local"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          - name: FLUENTD_SYSTEMD_CONF
            value: disable
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 200Mi
      volumes:
       - name: config-volume
         configMap:
          name: fluentd-configmap
       - name: varlog
         hostPath:
          path: /var/log
       - name: progdatacontainers
         hostPath:
          path: /ProgramData/docker/containers

If everything is deployed properly you shall see following in your kube-logging namespace

gregory@master1:~$ k get all -n kube-logging
NAME                          READY   STATUS    RESTARTS   AGE
pod/fluentd-zcxj9             1/1     Running   0          31m
pod/kibana-699b99d996-vkd27   1/1     Running   3          44h

NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP               PORT(S)          AGE
service/elasticsearch-service   ExternalName   <none>          utilityvm.kubernetes.my   <none>           44h
service/kibana                  NodePort       10.103.68.166   <none>                    5601:30080/TCP   44h

NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE
daemonset.apps/fluentd   1         1         1       1            1           beta.kubernetes.io/os=windows   25h

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kibana   1/1     1            1           44h

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/kibana-699b99d996   1         1         1       44h

And following in default namespace

gregory@master1:~$ k get all
NAME                                READY   STATUS    RESTARTS   AGE
pod/win-webserver-8d8dcb548-mrnvp   1/1     Running   2          116m
pod/win-webserver-8d8dcb548-vnxzq   1/1     Running   2          116m

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   45h

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/win-webserver   2/2     2            2           44h

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/win-webserver-8d8dcb548   2         2         2       44h

Configure Kibana to live stream logs

Go to kibana WebUI (http://master1:30080 in my case) and click gear icon on bottom and then Index Patterns/Create index pattern. Type k8log-* in Define index pattern

Choose @timestamp as you Time field and choose “Create”

You will see all fields available with logs comming from fluentd and specifically ones kubernetes specific (like kubernetes.container_name, kubernetes.labels.app etc).

To stream logs click on Logs on left side and go to Settings tab. Here you will define what indices you want to appear in streaming log and what fields you want to be shown on screen. Example for mine is below

Save settings and switch to Stream tab which will output live logs along with fields you requested

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s