Sign in Subscribe
Cloud-Native

Setting Up Observability for Cloud-Native Applications with Prometheus and Grafana

Justin VanWinkle

• 3 min read

As cloud-native technologies continue to revolutionize software development and deployment, one of the most critical aspects of managing cloud-native applications is observability. Observability encompasses logging, metrics, and tracing, providing a holistic view into the internal state of an application. In this blog post, we will explore how to set up observability for cloud-native applications using Prometheus and Grafana. We will walk you through the steps to monitor a Kubernetes cluster and visualize metrics in Grafana.

Why Observability?

Observability is essential for several reasons:

  • Diagnosing Issues: Quickly identify and diagnose issues in your applications and infrastructure.
  • Performance Monitoring: Monitor application performance and resource usage to ensure efficient operation.
  • Proactive Alerts: Set up alerts to catch potential issues before they impact users.
  • Data-Driven Decisions: Make informed decisions based on real-time data and trends.

Prerequisites

Ensure you have the following prerequisites before starting:

  • A Kubernetes cluster (this tutorial uses a local cluster created with Minikube)
  • kubectl installed and configured
  • Helm installed
  • Basic knowledge of Kubernetes and Prometheus

Step 1: Install Prometheus on Kubernetes

Prometheus is an open-source systems monitoring and alerting toolkit that's ideal for cloud-native environments. We'll use Helm to install Prometheus:

# Add the Prometheus community Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Update your Helm repositories
helm repo update

# Install Prometheus using Helm
helm install prometheus prometheus-community/prometheus --namespace monitoring --create-namespace

Verify the installation:

# Check the pods in the monitoring namespace
kubectl get pods -n monitoring

Step 2: Install Grafana on Kubernetes

Grafana is an open-source analytics and monitoring solution that works seamlessly with Prometheus. We'll use Helm to install Grafana:

# Add the Grafana Helm repository
helm repo add grafana https://grafana.github.io/helm-charts

# Update your Helm repositories
helm repo update

# Install Grafana using Helm
helm install grafana grafana/grafana --namespace monitoring

Verify the installation:

# Check the pods in the monitoring namespace
kubectl get pods -n monitoring

Step 3: Access Grafana

By default, Grafana does not expose an external IP. We'll port-forward the Grafana service to access the dashboard:

# Port-forward Grafana service
kubectl port-forward svc/grafana 3000:80 -n monitoring

Open your browser and navigate to http://localhost:3000. The default login credentials are:

  • Username: admin
  • Password: prom-operator

Step 4: Configure Prometheus as a Data Source in Grafana

Once logged into Grafana, add Prometheus as a data source:

  1. Navigate to Configuration -> Data Sources
  2. Click on "Add data source"
  3. Select "Prometheus"
  4. In the "HTTP" section, set URL to http://prometheus-server.monitoring.svc:80
  5. Click "Save & Test" to verify the connection

Step 5: Create Your First Dashboard

With Prometheus configured as a data source, it's time to create a dashboard to visualize metrics:

  • Navigate to the Dashboard -> Home.
  • Click "New Dashboard" -> "Add new panel".
  • In the query editor, type a PromQL query, e.g., up{job="kubernetes-nodes"}, to get the status of the nodes.
  • Customize the visualization options as needed.
  • Click "Apply" to save the panel.

Advanced Configuration: Alerting

Grafana can also be used to set up alerts based on Prometheus metrics. Here's a simple example to create an alert if a node is down:

  1. Click on the panel you created and select "Edit".
  2. Navigate to the "Alert" tab and click on "Create Alert".
  3. Set a condition, e.g., WHEN last() OF query(A, 5m, now) IS BELOW 1, which checks if the last value of alive nodes is below 1 in the past 5 minutes.
  4. Configure the notification channel (Email, Slack, etc.).
  5. Save the alert configuration.

Lessons Learned and Common Pitfalls

Implementing observability is a continuous process. Here are some common pitfalls and lessons learned:

  • Avoid Alert Fatigue: Be selective with your alerts to avoid overwhelming your team with non-critical notifications.
  • Resource Management: Monitoring tools can consume significant resources. Ensure your cluster has adequate capacity.
  • Regularly Review Dashboards: Continuously review and update dashboards and alerts to reflect the current state of your applications and infrastructure.
  • Security: Secure your monitoring setup by implementing RBAC and network policies to prevent unauthorized access.

Conclusion

Setting up observability using Prometheus and Grafana in Kubernetes provides you with powerful tools to monitor, visualize, and alert on the state of your cloud-native applications. By following the steps outlined in this post, you can ensure your applications are running smoothly and proactively address issues before they become critical.

Have you set up observability in your Kubernetes cluster? Share your experience, challenges, and insights in the comments below!