History

In our company we use for monitoring Icinga2. And everything was ok until it came to Kubernetes.

As everyone know, best tool for Kubernetes monitoring is Prometheus. But I wanted to keep monitoring and alerting in Icinga2.

I started looking in Internet for existing solution to monitor Kubernetes with Icinga2. And in the end found tools written by guys from Nexinto.

They created two applications:

  • check_kubernetes application to run checks
  • kubernetes_icinga - daemon which scan Kubernetes cluster and automatically change Icinga2 configuration through API

Details about applications

check_kubernetes

Usage of /usr/lib/nagios/plugins/check_kubernetes:
  -alsologtostderr
        log to standard error as well as files
  -kubeconfig string
        kubeconfig location
  -log_backtrace_at value
        when logging hits line file:N, emit a stack trace
  -log_dir string
        If non-empty, write log files in this directory
  -logtostderr
        log to standard error instead of files
  -name string
        name of object to check
  -namespace string
        namespace of object (default "default")
  -stderrthreshold value
        logs at or above this threshold go to stderr
  -type string
        type of object to check
  -v value
        log level for V logs
  -vmodule value
        comma-separated list of pattern=N settings for file-filtered logging

This application perform only simple checks for:

  • Pods
  • ReplicaSets
  • Deployments
  • DaemonSets
  • StatefulSets
  • Nodes
  • ComponentStatuses
  • Services

kubernetes_icinga

This application doesn’t have any help output. It requires define enviroment variables in system.

Here is a list of possible variables:

  • ICINGA_URL
  • ICINGA_USER
  • ICINGA_PASSWORD
  • KUBECONFIG
  • REFRESH_INTERVAL

Variable REFRESH_INTERVAL is optional. Default value is 0. This mean applications would recheck for changes in Kubernetes cluster without delay.

Configuration example

So we need to build two Golang applications. Let’s start with check_kubernetes.

Building check_kubernetes

Let’s clone repository and change our current location to plugin folder.

git clone https://github.com/Nexinto/check_kubernetes/
cd check_kubernetes/

Now we need to get all dependencies for application.

go get -d ./... # collect all dependencies

And finally we can build it.

go build check_kubernetes.go

After we’d finish we can copy binary to nagios plugins folder (in example I use default path for Ubuntu/Debian).

cp check_kubernetes /ust/lib/nagios/plugins/

Adding new command to Icinga2 configuration

Then we neew to modify Icinga2 configuration to add new command to configuration. Open file with commands configuration and add next text

object CheckCommand "check_kubernetes" {
  command = [ PluginDir + "/check_kubernetes" ]

  arguments = {
    "-kubeconfig" = "<path_to_kube.config_file>"
    "-name" = "$host.vars.kubernetes_name$"
    "-namespace" = "$host.vars.kubernetes_namespace$"
    "-type" = "$host.vars.kubernetes_type$"
  }
}

Don’t forget to change <path_to_kube.config_file> variable to your path for kube configuration.

Building kubernetes_icinga

Steps are the same as we have in check_kubernetes section.

So we start with cloning repository and changing current folder to repo folder.

git clone https://github.com/Nexinto/kubernetes_icinga
cd kubernetes_icinga/

Getting all dependencies for application.

go get -d ./... # collect all dependencies

And building it.

go build kubernetes-icinga.go

Starting daemon

IMPORTANT NOTICE !!!!: At the moment I run this daemon application in tmux session and doesn’t have any sort of upstart or systemctl script to automate this. As well as Nexinfo.

To run this application we need to have next variables in out enviroment:

export ICINGA_URL="https://<url_to_icinga2_api>:5665"
export ICINGA_USER=<icinga2-api-user>
export ICINGA_PASSWORD=<icinga2-api-password>
export KUBECONFIG=<path_to_kube.config_file>
export REFRESH_INTERVAL=60

You need to change <url_to_icinga2_api> to your Icinga2 address and <path_to_kube.config_file> to to your path for kube configuration.

Then just run application:

kubernetes_icinga

Once application finish adding hosts and services it’ll recheck Kubernetes cluster every minute for changes.

Screenshot of working example

Kubernetes_Icinga2_Icingaweb2