Implemented a Modern Observability Stack on RPi with OpenTelemetry

Setting up a professional-grade monitoring system doesn't require a data center. By combining OpenTelemetry (OTel), Prometheus, and Grafana, you can transform a Raspberry Pi into a high-fidelity observability node. This guide walks through the end-to-end configuration I used to monitor system health in real-time.

---

The Architecture

The setup uses a distributed approach: the Raspberry Pi acts as the data producer, while a separate instance (WSL or a server) acts as the data consumer and visualizer.

---

Phase 1: Configuring the Raspberry Pi

The goal is to collect hardware metrics (CPU, RAM, Disk, Network) and expose them in a format Prometheus understands.

1. The OpenTelemetry Configuration

Create a file named otel-config.yaml. This is the "brain" of the collector, defining what to scrape and where to send it.

receivers:
  hostmetrics:
    collection_interval: 5s
    scrapers:
      cpu:
      memory:
      disk:
      filesystem:
      network:
      load:
      paging:
      processes:

processors:
  batch:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [batch]
      exporters: [prometheus]

2. Deploying with Docker

To ensure the collector can see the actual hardware (not just the container’s virtualized environment), we use host networking. Create docker-compose.yaml:

version: "3"
services:
  otel:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-pi
    restart: unless-stopped
    network_mode: host
    pid: host
    volumes:
      - ./otel-config.yaml:/etc/otel-collector-config.yaml
    command: ["--config=/etc/otel-collector-config.yaml"]

Run it: docker compose up -d

---

Phase 2: Setting up the Monitoring Node (WSL/Server)

Now we need to capture that data and visualize it.

1. Prometheus Configuration

Prometheus needs to "scrape" the endpoint we opened on the Pi. Create prometheus.yml:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: 'raspberry-pi'
    static_configs:
      - targets: ['<YOUR_PI_IP_ADDRESS>:8889']

2. The Visualization Stack

Use this docker-compose.yaml on your monitoring machine to link Prometheus and Grafana:

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

---

Phase 3: Visualizing the Data

1. Access Grafana at http://localhost:3000 or http://(your ip address):3000 .

   

2. Add Data Source: Select Prometheus and use the URL http://prometheus:9090 or http://(your ip):9090 .

   

3. Create Dashboard: You can now create panels using PromQL queries like system_cpu_utilization_average_percentage or system_memory_usage.

   

---

Phase 4: The "Stress Test"

To verify the system, I used stress-ng to push the Pi to its limits:

# Install the stress tool
sudo apt install stress-ng -y

# Run a heavy load: 4 CPU cores, 75% RAM, and Disk IO
stress-ng --cpu 4 --vm 2 --vm-bytes 75% --io 2 --timeout 60s

Watching the Grafana gauges spike to 100% in real-time confirms the pipeline is working with minimal latency. This setup provides the same observability patterns used in enterprise environments, scaled down to the palm of your hand.