Introduction

In this article, we will discuss a common issue encountered in Kubernetes deployments involving the expiration of the etcD Prometheus certificate. We will also provide a step-by-step guide on how to renew the certificate using OpenSSL.

Problem

In our Kubernetes deployment, we encountered an error related to the etcD. The error message was as follows:

etcd-manager-main-i-0ee1f712d597448db etcd-manager {"level":"warn","ts":"2024-02-21T12:24:18.774Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.239.31.99:41644","server-name":"","error":"tls: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2024-02-21T12:24:18Z is after 2021-10-04T20:49:28Z"}

This error indicates that the client’s certificate has expired or is not yet valid.

Solution

To resolve this issue, we need to generate a new client certificate. Here are the steps to do this using OpenSSL:

  1. Generate a new RSA private key:

    openssl genrsa -out client.key 2048

  2. Generate a new certificate signing request (CSR) using the private key:

    openssl req -new -key client.key -out client.csr -subj "/CN=kube-apiserver"

  3. Generate a new client certificate using the CSR, the CA certificate, and the CA private key:

    openssl x509 -req -in client.csr -CA /etc/kubernetes/pki/etcd-manager/etcd-clients-ca.crt -CAkey /etc/kubernetes/pki/etcd-manager/etcd-clients-ca.key -CAcreateserial -out client.crt

  4. Encode with base64 contnet of those files:

    • client.key
    • client.crt
    • /etc/kubernetes/pki/etcd-manager/etcd-clients-ca.key

  5. Update etcd-certs secret in the namespace where you run Prometheus with new values

For more information about this issue, you can refer to the advisory on the Kubernetes Operations (kops) website: kops advisory on etcD manager certificate expiration.

By following these steps, you should be able to renew your etcD Prometheus certificate and resolve the error in your Kubernetes deployment.