History
In our company we use for monitoring Icinga2. And everything was ok until it came to Kubernetes.
As everyone know, best tool for Kubernetes monitoring is Prometheus. But I wanted to keep monitoring and alerting in Icinga2.
I started looking in Internet for existing solution to monitor Kubernetes with Icinga2. And in the end found tools written by guys from Nexinto.
They created two applications:
- check_kubernetes application to run checks
- kubernetes_icinga - daemon which scan Kubernetes cluster and automatically change Icinga2 configuration through API
Details about applications
check_kubernetes
Usage of /usr/lib/nagios/plugins/check_kubernetes:
-alsologtostderr
log to standard error as well as files
-kubeconfig string
kubeconfig location
-log_backtrace_at value
when logging hits line file:N, emit a stack trace
-log_dir string
If non-empty, write log files in this directory
-logtostderr
log to standard error instead of files
-name string
name of object to check
-namespace string
namespace of object (default "default")
-stderrthreshold value
logs at or above this threshold go to stderr
-type string
type of object to check
-v value
log level for V logs
-vmodule value
comma-separated list of pattern=N settings for file-filtered logging
This application perform only simple checks for:
- Pods
- ReplicaSets
- Deployments
- DaemonSets
- StatefulSets
- Nodes
- ComponentStatuses
- Services
kubernetes_icinga
This application doesn’t have any help output. It requires define enviroment variables in system.
Here is a list of possible variables:
- ICINGA_URL
- ICINGA_USER
- ICINGA_PASSWORD
- KUBECONFIG
- REFRESH_INTERVAL
Variable REFRESH_INTERVAL is optional. Default value is 0. This mean applications would recheck for changes in Kubernetes cluster without delay.
Configuration example
So we need to build two Golang applications. Let’s start with check_kubernetes.
Building check_kubernetes
Let’s clone repository and change our current location to plugin folder.
git clone https://github.com/Nexinto/check_kubernetes/
cd check_kubernetes/
Now we need to get all dependencies for application.
go get -d ./... # collect all dependencies
And finally we can build it.
go build check_kubernetes.go
After we’d finish we can copy binary to nagios plugins folder (in example I use default path for Ubuntu/Debian).
cp check_kubernetes /ust/lib/nagios/plugins/
Adding new command to Icinga2 configuration
Then we neew to modify Icinga2 configuration to add new command to configuration. Open file with commands configuration and add next text
object CheckCommand "check_kubernetes" {
command = [ PluginDir + "/check_kubernetes" ]
arguments = {
"-kubeconfig" = "<path_to_kube.config_file>"
"-name" = "$host.vars.kubernetes_name$"
"-namespace" = "$host.vars.kubernetes_namespace$"
"-type" = "$host.vars.kubernetes_type$"
}
}
Don’t forget to change <path_to_kube.config_file>
variable to your path for kube configuration.
Building kubernetes_icinga
Steps are the same as we have in check_kubernetes section.
So we start with cloning repository and changing current folder to repo folder.
git clone https://github.com/Nexinto/kubernetes_icinga
cd kubernetes_icinga/
Getting all dependencies for application.
go get -d ./... # collect all dependencies
And building it.
go build kubernetes-icinga.go
Starting daemon
IMPORTANT NOTICE !!!!: At the moment I run this daemon application in tmux session and doesn’t have any sort of upstart or systemctl script to automate this. As well as Nexinfo.
To run this application we need to have next variables in out enviroment:
export ICINGA_URL="https://<url_to_icinga2_api>:5665"
export ICINGA_USER=<icinga2-api-user>
export ICINGA_PASSWORD=<icinga2-api-password>
export KUBECONFIG=<path_to_kube.config_file>
export REFRESH_INTERVAL=60
You need to change <url_to_icinga2_api>
to your Icinga2 address and <path_to_kube.config_file>
to to your path for kube configuration.
Then just run application:
kubernetes_icinga
Once application finish adding hosts and services it’ll recheck Kubernetes cluster every minute for changes.
Screenshot of working example