Setting up logging for windows pods in Kubernetes

There is no currently well supported centralized solution for logging for windows pods in Kubernetes outside of built-in solutions by big cloud providers. Instructions below would allow to setup logging in stand-alone Kubernetes cluster for Windows pods.

Following software will be used

  1. FluentD windows service
  2. ElasticSearch service
  3. Kibana UI front-end to service to display logs

Instructions are built upon Kubernetes deployed locally outlined in earlier post. Cluster config is below

gregory@master1:~$ k get nodes                                 
NAME         STATUS   ROLES    AGE   VERSION
master1      Ready    master   18d   v1.17.4
winworker1   Ready    <none>   18d   v1.17.4

Overall architecture of logging soluion consists of following moving parts:

  1. Docker service would be configured with fluentd logging driver
  2. FluentD service which will parse logs and send it to ElasticSearch
  3. Kibana UI to query logs

Install ElasticSearch

Install ElasticSearch on any Windows nodes to aggregate logs from fluentd service. Accept defaults for installation.

Install FluentD service

Docker daemon (service) running on Windows worker nodes would need to be configured with fluentd logging driver to send data to fluentd service.

To install fluentd download binaries and install it per instructions https://www.fluentd.org/download

Once installed, modify configuration file under C:\opt\td-agent\etc\td-agent\td-agent.conf to contain following entry. Replace host localhost with hostname of host where you installed ElasticSearch in previous step

<source>
  @type forward
</source>
<match *>
  @type elasticsearch
<inject>
   time_key          @timestamp
   time_format       %Y%m%dT%H%M%S%z
</inject>
  host localhost
  port 9200
  index_name fluentd
  logstash_format true
  flush_interval 5s
</match>

Install fluentd as service by starting Td-agent command prompt and executing

> fluentd --reg-winsvc i
> fluentd --reg-winsvc-fluentdopt '-c C:/opt/td-agent/etc/td-agent/td-agent.conf -o C:/opt/td-agent/td-agent.log'

Restart fluentd service restart-service fluentdwinsvc

Configure docker daemon

Configure dockerd to use fluentd logging driver on windows nodes. Edit file C:\programdata\docker\config\daemon.json to have following content. Replace utilityvm with hostname where you installed fluentd in previous step. Restart docker service after change.

{
   "log-driver": "fluentd",
   "log-opts": {
     "fluentd-address": "utilityvm:24224"
   }
 }

Install kibana

Install Kibana on Windows for nice UI for ElasticSearch. Modify file called kibana.config under \config subfolder and add/change following parameters

server.host: 0.0.0.0

elasticsearch.hosts: ["<hostname/port of your elasticsearch host>"]

Once installed launch kubana.bat which will launch service listening on port 5061 by default. So you can access server on port 5061 to get interface in Kibana .

Once in UI add index for logstash as below

Once index is added you can look at logs at Logs tab. And configure to show real time data from logstash* index. Windows containers are configured to output container name and random number every 5s so you will be able to see this information streaming live

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s