Prometheus is a popular open source metric monitoring solution and is a part of the Cloud Native Compute Foundation. These tools have recently graduated from the Cloud Native Computing Foundation, which means that they are ready for production use and are first-class citizens among open-source monitoring tools.

With Helm, installing and managing Prometheus and Grafana on your AKS cluster has become much more straightforward. It not only provides you with a hardened, well-tested setup but also provides a lot of preconfigured dashboards to get you started right away.

While Prometheus offers the core monitoring capability of collecting and receiving logs, indexing them, and optimizing storage, Grafana provides a means for visualizing the metrics in graphical dashboards.

In this blog we’ll learn how easy it is to setup Prometheus and Grafana on AKS Cluster.

Prerequisites

1. AKS Cluster: Ensure your AKS cluster is provisioned and in a running state.

2. Azure CLI: Have the Azure CLI installed.

3. AKS Credentials: Use az aks get-credentials -g {resource-group} -n {aks-cluster-name} to get AKS credentials.

4. Helm Installation: Install Helm for your OS from here.

   https://github.com/helm/helm/releases

Steps to configure Prometheus in AKS

1. Define the Helm repo

We need to define the public Kubernetes chart repository in the Helm configuration

helm repo add stable https://charts.helm.sh/stable

2. Install Prometheus Chart

With Helm, you do not need to worry about the internals of writing manifests and wiring things up. Helm provides you with battle-hardened, production-grade setups extensively tested across multiple scenarios and use cases.

We will be using the stable/prometheus-operator chart for this. It's also a good idea to install Prometheus operator in a separate namespaces.

$kubectl create ns monitoring
$helm install prometheus stable/prometheus-operator --namespace monitoring

Let’s valiate if the prometheus operator was installed successfully.

bash-3.2$ kubectl --namespace monitoring get pods -l "release=prometheus"
NAME                                                   READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-9fhrl              1/1     Running   0          21m
prometheus-prometheus-oper-operator-6d9c4bdb9f-hfpbb   2/2     Running   0          21m

This output confirms the pods are up and running.

3. Accessing the Prometheus Dashboard from the AKS Cluster.

We will be using the Kubernetes proxy to access it. The Kube proxy allows us to securely tunnel connections to Prometheus using TLS via the Kube API server.

bash-3.2$ kubectl port-forward -n monitoring prometheus-prometheus-oper-operator-6d9c4bdb9f-hfpbb-0 9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

4. Now access the Prometheus Dashboard from your local laptop/desktop; http://localhost:9090

We can use the Prometheus Query Language to query Kubernetes metrics in the old-fashioned geeky way.

For example, let’s look at the time series for the API Server request total:

4. Accessing the Grafana Dashboard from the AKS Cluster.

Let’s locate the grafana pod and then do a kubectl port-forward

bash-3.2$ kubectl get pod -n monitoring|grep grafana
prometheus-grafana-7c78857f5c-qfhld                      2/2     Running   0          11m
bash-3.2$ kubectl port-forward -n monitoring prometheus-grafana-7c78857f5c-qfhld 3000
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000
Handling connection for 3000
Handling connection for 3000

In order to login to the grafana dashboard we need the username and password. Retrieve these details by executing the following command

kubectl get secret -n monitoring prometheus-grafana -o yaml
Get the username and password.

Access the Grafana dashboard on http://localhost:3000

Congratulations!. You are all set with Prometheus and Grafana on Azure Kubernetes Cluster (AKS)

Conclusion

We can discover a lot of other dashboards that gives us valuable insights about the health of AKS cluster, resource usage patterns of particular application pods, network traffic flow across the cluster, and much more.

Prometheus and Grafana are powerful tools for monitoring your Kubernetes cluster, and with Helm it’s so easy to setup and running in few minutes..