Linux observability with Grafana

Linux Monitoring with Grafana and Prometheus

Observability has become an essential practice for managing systems, especially in complex and distributed environments. It allows system administrators and developers to monitor, measure, and visualize the performance of their systems. In this article, we will explore how to start using observability for a Linux box with a focus on Grafana, a popular open-source tool…


Observability has become an essential practice for managing systems, especially in complex and distributed environments. It allows system administrators and developers to monitor, measure, and visualize the performance of their systems. In this article, we will explore how to start using observability for a Linux box with a focus on Grafana, a popular open-source tool for monitoring and visualization. We will cover the installation process, configuration, and usage of Grafana, and include code examples, links to images, and other relevant articles.

Table of Contents

  1. What is Observability?
  2. Why Use Grafana for Observability on Linux?
  3. Setting Up Observability with Grafana on a Linux Box
  4. Installing Prometheus as a Data Source
  5. Connecting Prometheus to Grafana
  6. Creating Your First Grafana Dashboard
  7. Additional Resources and Conclusion

1. What is Observability?

Observability is the ability to measure the internal state of a system based on its external outputs. In other words, it enables you to understand what’s happening inside your system without needing to know all the details of the code. This includes monitoring system performance, tracking errors, analyzing logs, and identifying bottlenecks.

Observability involves three primary pillars:

  • Metrics: Numerical data that measure system performance (e.g., CPU usage, memory consumption).
  • Logs: Textual records of events that occur within the system.
  • Traces: Detailed information about individual requests in distributed systems.

Grafana provides a visual interface for working with these pillars, making it easier to monitor and analyze the performance of your Linux box.


2. Why Use Grafana for Observability on Linux?

Grafana is one of the most widely used platforms for observability because of its flexibility and powerful visualization capabilities. It integrates with various data sources such as Prometheus, InfluxDB, Elasticsearch, and more, providing a unified dashboard to monitor different system metrics in real-time.

Key Features of Grafana:

  • Custom Dashboards: You can create highly customizable dashboards that display data in various formats (graphs, heatmaps, tables, etc.).
  • Alerts: Set up alerts to notify you when certain thresholds are reached.
  • Plugins: Extend the functionality with a wide range of plugins.
  • User Authentication: Secure access to dashboards and data.

For Linux users, Grafana can help visualize system health metrics, track CPU, memory, disk usage, and network traffic. It can also integrate with application-level observability tools, such as those for tracking database performance or web server response times.


3. Setting Up Observability with Grafana on a Linux Box

In this section, we’ll go through the steps of installing Grafana and setting up basic observability on a Linux system. We will use Prometheus as our primary data source, a popular open-source monitoring system that scrapes metrics from target systems.

Step 1: Install Grafana

First, we need to install Grafana. On most Linux distributions, Grafana can be installed using either a package manager or manually from its official website.

For Debian/Ubuntu:

sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt-get update
sudo apt-get install grafana

For Red Hat/CentOS:

sudo yum install -y https://packages.grafana.com/oss/rpm/grafana-8.0.3-1.x86_64.rpm
sudo yum install grafana

After installation, start the Grafana server:

sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Grafana will now be running on port 3000 by default. You can access the web interface by navigating to http://localhost:3000 or your server’s IP address in a web browser.


4. Installing Prometheus as a Data Source

Prometheus is a time-series database that is often used to collect and store metrics for monitoring. We will set up Prometheus as the primary data source for Grafana.

Step 1: Install Prometheus

Download the latest release of Prometheus from its official website:

wget https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-amd64.tar.gz
tar -xvzf prometheus-*.tar.gz
cd prometheus-*

Now, start Prometheus by running:

./prometheus --config.file=prometheus.yml

By default, Prometheus will be running on port 9090. You can verify it by navigating to http://localhost:9090 in your browser.

Step 2: Configure Prometheus to Monitor Linux Metrics

To monitor Linux system metrics, we will use a Prometheus exporter called Node Exporter. It collects system-level metrics like CPU usage, memory usage, and disk IO.

Install Node Exporter:

wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar -xvzf node_exporter-*.tar.gz
cd node_exporter-*
./node_exporter

Node Exporter will now be running on port 9100, and Prometheus will scrape metrics from it. You can add the following job to your prometheus.yml configuration file:

scrape_configs:
  - job_name: 'node_exporter'
static_configs:
  - targets: ['localhost:9100']

5. Connecting Prometheus to Grafana

Now that we have Prometheus running and collecting Linux metrics, we can connect it to Grafana.


  1. Open Grafana in your browser (http://localhost:3000).
  2. Log in with the default credentials (admin/admin).
  3. Click on the Configuration icon in the left menu and select Data Sources.
  4. Click Add data source, select Prometheus, and enter http://localhost:9090 as the URL.
  5. Click Save & Test to ensure Grafana can communicate with Prometheus.


6. Creating Your First Grafana Dashboard


Now that Prometheus is connected, we can create a dashboard to visualize system metrics.

  1. In the Grafana interface, click on the + icon on the left and select Dashboard.
  2. Click Add New Panel.
  3. In the Query section, select Prometheus as the data source.
  4. To visualize CPU usage, use the following PromQL query:
rate(node_cpu_seconds_total{mode="idle"}[5m])
  1. This will display a graph showing the CPU idle time over the past five minutes.
  2. Customize the graph using the panel options, and then click Apply to save your panel to the dashboard.
  3. Add more panels for other metrics, such as memory usage, disk IO, or network traffic, using queries like:
    • Memory Usage: node_memory_MemAvailable_bytes
    • Disk IO: rate(node_disk_io_time_seconds_total[5m])
    • Network Traffic: rate(node_network_receive_bytes_total[5m])

You now have a functioning Grafana dashboard that visualizes important metrics for your Linux system.


7. Additional Resources and Conclusion

Grafana and Prometheus provide powerful tools to monitor and visualize system performance. By setting up observability on your Linux box, you gain insights that help in identifying performance bottlenecks and maintaining system stability.

For further reading and tutorials, check out these resources:

By following these steps, you can now start monitoring your Linux system with Grafana and Prometheus. You can continue to expand your dashboards, set up alerts, and integrate other data sources to gain a more comprehensive view of your system’s performance.