Setup Monitoring of Kubernetes | Prometheus, Grafana & Loki | Part 1

Last Update: 16/05/2024 (Tested with Minikubes and AKS)

We will see how to install/ set-up Loki in Kubernetes, add Promtail, and connect to the Grafana dashboard. Loki in a distributed mode is generally better for scaling compared to running Loki in a single-instance mode.

We have already installed Grafana. So we will focus only on the Loki setup. 

Loki workflow diagram

1. Install prometheus & Grafana.

These are some basic commands, you will understand by looking at them, no need to dive deep into that.

# Check Helm Installation
helm version

# Adding Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Update repo
helm repo update

# helm repo update
helm search repo prometheus-community

# Install Stack
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

# Get all Info
kubectl get all -n monitoring

Output: 

2. Port-forward to open grafana. You can also use Ingress to expose it to outer world.

# Expose Grafana Service
kubectl port-forward service/prometheus-grafana -n monitoring 80:80

# (Optional) Check Prometheus End Point.
kubectl port-forward service/prometheus-operated -n monitoring 9090:9090

Open the serviced URL in browser.

Grafana Login Page

This is default username and password. Checkout secrets if it not works.

Prometheus Dashboard

2. Setup Loki 

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm show values grafana/loki-distributed > loki-distributed-overrides.yaml
helm upgrade --install --values loki-distributed-overrides.yaml loki grafana/loki-distributed -n loki --create-namespace

Update the Endpoint URL of above file. Change Namespace accordingly.

3. Installing Promtail:  

Promtail is a log collector agent, installed on every node and sends data to the Loki operator. It reads log files from various sources and transforms them into a specific format.

helm upgrade --install --values promtail-overrides.yaml promtail grafana/promtail -n loki

Adding Loki data source in Grafana dashboards.

 

Data Source URL: http://loki-loki-distributed-gateway.loki.svc.cluster.local

Conclusion: We have seen how to set up Loki in the existing Grafana environment, setting data source and other configs. In the next post, we will see how to connect it to Azure Storage and maintain a retention policy.