Monitoring Node.js Microservices with Prometheus and Grafana

Why Default Logging Isn't Enough

Console logs and basic uptime checks tell you a service is alive. They don't tell you it's slow, leaking memory, or that one specific endpoint has a 20% error rate. That gap is where Prometheus and Grafana come in.

• ▸Logs are text hard to query trends, percentiles, or rates across services
• ▸Uptime checks only fire when a service is completely dead, not when it's degraded
• ▸Without histograms you can't see p95/p99 latency averages hide tail latency problems
• ▸Cross-service correlation is impossible without a shared metrics store

Prometheus gives you a time-series metrics store. Grafana gives you dashboards and alerts on top of it. Together they answer: what is my service doing right now, and when did it start going wrong?

Architecture Overview

The full stack is three components alongside your services:

  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐
  │  Service A   │   │  Service B   │   │  Service C   │
  │ :3000/metrics│   │ :3000/metrics│   │ :3000/metrics│
  └──────┬───────┘   └──────┬───────┘   └──────┬───────┘
         │                  │                   │
         └──────────────────┼───────────────────┘
                            ▼
                  ┌──────────────────┐
                  │    Prometheus    │  scrapes /metrics every 15s
                  │    :9090         │  stores time-series data
                  └────────┬─────────┘
                           │
               ┌───────────┴───────────┐
               ▼                       ▼
        ┌────────────┐       ┌──────────────────┐
        │   Grafana  │       │  Alertmanager    │
        │   :3000    │       │  :9093           │
        └────────────┘       └────────┬─────────┘
                                      │
                               Slack / PagerDuty

Each Node.js service exposes a /metrics endpoint in Prometheus text format. Prometheus scrapes all services on a regular interval and stores the data. Grafana queries Prometheus for dashboards. Alertmanager handles routing warnings go to Slack, critical alerts page on-call.

Step 1 Instrument Your Node.js Service

Install the official Prometheus client:

npm install prom-client

Create a metrics module. The default metrics give you Node.js process stats for free CPU usage, heap, event loop lag, GC pauses. Add custom counters and histograms for your HTTP layer on top:

// metrics.ts
import client from "prom-client";

const register = new client.Registry();

// built-in: CPU, memory, event loop lag, GC
client.collectDefaultMetrics({ register, prefix: "node_" });

// custom: HTTP request counter
export const httpRequestsTotal = new client.Counter({
  name: "http_requests_total",
  help: "Total number of HTTP requests",
  labelNames: ["method", "route", "status_code"],
  registers: [register],
});

// custom: request latency histogram
export const httpRequestDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP request latency in seconds",
  labelNames: ["method", "route", "status_code"],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5],
  registers: [register],
});

export { register };

Wire it into your Express app with a middleware and expose the scrape endpoint:

// app.ts
import express from "express";
import { register, httpRequestsTotal, httpRequestDuration } from "./metrics";

const app = express();

app.use((req, res, next) => {
  const end = httpRequestDuration.startTimer();
  res.on("finish", () => {
    const labels = {
      method: req.method,
      route: req.route?.path ?? req.path,
      status_code: String(res.statusCode),
    };
    httpRequestsTotal.inc(labels);
    end(labels);
  });
  next();
});

// Prometheus scrape endpoint
app.get("/metrics", async (_req, res) => {
  res.set("Content-Type", register.contentType);
  res.send(await register.metrics());
});

app.listen(3000);

Step 2 Configure Prometheus

Prometheus is configured with a single YAML file that tells it what to scrape and how often. The simplest setup uses static targets you list each service by hostname or IP:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - "alerts.yml"

scrape_configs:
  - job_name: "service-a"
    static_configs:
      - targets: ["service-a:3000"]
    metrics_path: /metrics

  - job_name: "service-b"
    static_configs:
      - targets: ["service-b:3000"]
    metrics_path: /metrics

Run Prometheus with Docker alongside your services:

# docker-compose.yml (relevant excerpt)
services:
  prometheus:
    image: prom/prometheus:v2.51.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alerts.yml:/etc/prometheus/alerts.yml
      - prometheus-data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=15d"

volumes:
  prometheus-data:

Step 3 Define Alert Rules

Alert rules live in alerts.yml. These are the four I add to every Node.js service from day one:

# alerts.yml
groups:
  - name: nodejs-service
    rules:

      # Service is completely unreachable
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} is down"
          description: "Prometheus cannot scrape {{ $labels.job }}. Check if the process is running."

      # p95 latency above 500ms sustained for 5 minutes
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95,
            rate(http_request_duration_seconds_bucket[5m])
          ) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High p95 latency on {{ $labels.job }}"
          description: "p95 latency is {{ $value | humanizeDuration }} threshold is 500ms"

      # More than 5% of requests returning 5xx
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status_code=~"5.."}[5m])
          /
          rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.job }}"
          description: "{{ $value | humanizePercentage }} of requests are 5xx errors"

      # Heap memory above 80% for 10 minutes heading toward OOM
      - alert: HighHeapUsage
        expr: |
          node_nodejs_heap_size_used_bytes
          /
          node_nodejs_heap_size_total_bytes > 0.8
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High heap on {{ $labels.job }}"
          description: "Heap is at {{ $value | humanizePercentage }} investigate memory leak or scale out"

Step 4 Configure Alertmanager

Alertmanager receives firing alerts from Prometheus and routes them. Warnings go to Slack, critical alerts go to PagerDuty:

# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

route:
  group_by: ["alertname", "job"]
  group_wait: 30s       # wait 30s to batch related alerts
  group_interval: 5m
  repeat_interval: 4h
  receiver: slack-warnings
  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical

receivers:
  - name: slack-warnings
    slack_configs:
      - channel: "#alerts-infra"
        title: "⚠️ {{ .GroupLabels.alertname }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}
{{ end }}"
        send_resolved: true

  - name: pagerduty-critical
    pagerduty_configs:
      - routing_key: "YOUR_PAGERDUTY_INTEGRATION_KEY"
        description: "{{ .GroupLabels.alertname }} {{ .CommonAnnotations.summary }}"
        send_resolved: true

Run Alertmanager alongside Prometheus:

# docker-compose.yml (excerpt)
  alertmanager:
    image: prom/alertmanager:v0.27.0
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

Step 5 Set Up Grafana

Run Grafana and provision Prometheus as a data source via config file so it persists across container restarts:

# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true
    editable: false

# docker-compose.yml (excerpt)
  grafana:
    image: grafana/grafana:10.4.0
    ports:
      - "3001:3000"
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme

The three dashboards I provision for every Node.js deployment:

• ▸Service Health Overview up/down per service, request rate, error rate, p50/p95/p99 latency
• ▸Node.js Runtime heap used vs total, event loop lag, GC pause duration, active handles
• ▸Infrastructure CPU and memory per host or container

Key PromQL Queries to Bookmark

These are the queries I reach for first when investigating an incident:

# Requests per second
rate(http_requests_total[1m])

# Error rate as a percentage
rate(http_requests_total{status_code=~"5.."}[5m])
/ rate(http_requests_total[5m]) * 100

# p95 latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Event loop lag high value means the thread is blocked
node_nodejs_eventloop_lag_seconds

# Heap usage percentage
node_nodejs_heap_size_used_bytes / node_nodejs_heap_size_total_bytes * 100

# Active HTTP connections
node_http_server_connections_total

Production Tips

• ▸Use a named Docker volume for Prometheus data anonymous volumes are wiped when the container is removed
• ▸Set scrape_interval to 15s for most services; drop to 5s only for critical paths (payments, auth) where faster alerting is worth the extra storage
• ▸Add a silence in Alertmanager before planned deployments prevents alert fatigue from expected restarts
• ▸Store Slack webhook URLs and PagerDuty keys in environment variables or a secrets manager, not hardcoded in alertmanager.yml
• ▸Enable Prometheus remote_write to a long-term store (Thanos, VictoriaMetrics) if you need more than 15 days of history for capacity planning

Wrapping Up

The full stack prom-client in each service, Prometheus scraping them, Alertmanager routing to Slack and PagerDuty, Grafana for dashboards runs entirely in Docker and takes half a day to set up the first time. After that, onboarding a new service means adding one job to prometheus.yml and dropping the metrics middleware into the new app.

The payoff: instead of finding out a service is broken when a user reports it, you get a Slack message the moment error rate crosses 5% or heap is climbing toward OOM with a direct link to the Grafana panel showing exactly when and why it started.