April 5, 2026 ·

#observability
#prometheus
#grafana
#devops
#self-hosting

Self-Hosted Observability: Prometheus, Loki, and Grafana Without the SaaS Bill

Running Prometheus, VictoriaMetrics, Loki, and Grafana on your own boxes. What that SaaS invoice really buys, and what you can build yourself.

The room went quiet when I put the projected Datadog bill on the screen. $23 per host per month for infrastructure monitoring, plus $0.10 per ingested GB of logs. Sized against a fleet of about 40 hosts across two environments generating roughly 50 GB of logs per day, the napkin math came out somewhere around $12,000 a month, and that was before APM tracing.

That arithmetic is why I ended up running an observability stack on my own boxes instead. I wrote about multi-tenant platform design in an earlier Kubernetes post. This one is the infrastructure sequel: how to monitor a blue/green Docker fleet, what it actually costs in ops time, and when you should just pay the SaaS bill anyway.

Spoiler: it’s more work than vendors want you to believe. But it’s also not the dark art they imply it is.

The Stack, and Why Each Piece Is There

The stack I settled on has five components. Each one does exactly one job, and that discipline matters more than the specific tools.

Prometheus scrapes metrics endpoints every 15 seconds and evaluates alerting rules. That’s it. It doesn’t try to store data for six months, serve fancy dashboards, or do log aggregation. It’s a scraper with a local TSDB and a rules engine. Simple.

VictoriaMetrics handles long-term metric storage. You might ask why not just use Prometheus for this. The short answer: Prometheus’s local storage wasn’t designed for months of retention on modest hardware. VictoriaMetrics accepts Prometheus remote-write, compresses aggressively (I see roughly 10x compression versus raw Prometheus blocks), and handles a 14-month retention window on a single 2TB NVMe drive. I looked at Thanos and Cortex too. Both are excellent. Both were more operational complexity than this scale needed.

Loki collects logs. Loki’s philosophy clicked for me: don’t index log content, only index labels (service name, environment, host). This makes ingestion cheap and fast, at the cost of slower full-text search on large time ranges. Since roughly 90% of real log searches start with “show me errors from service X in the last hour,” that tradeoff works.

Grafana is the single pane. Metrics from VictoriaMetrics, logs from Loki, alerts from Alertmanager. One URL, one login, one place to look during an incident.

Alertmanager takes alerts from Prometheus and handles deduplication, grouping, silencing, and routing. Page the on-call engineer on Slack for critical alerts. Send a digest email for warnings. Don’t wake anyone for info-level noise.

Getting It Running

The whole stack runs as Docker containers, orchestrated with Compose. Here’s the simplified core:

# docker-compose.observability.yml
services:
  prometheus:
    image: prom/prometheus:v2.52.0
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/rules/:/etc/prometheus/rules/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=7d'    # short local retention
      - '--web.enable-lifecycle'               # hot-reload config via API
    ports:
      - "9090:9090"

  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.101.0
    volumes:
      - vm_data:/victoria-metrics-data
    command:
      - '-retentionPeriod=14'                  # 14 months
      - '-httpListenAddr=:8428'
    ports:
      - "8428:8428"

  loki:
    image: grafana/loki:3.0.0
    volumes:
      - ./loki/loki-config.yml:/etc/loki/config.yml
      - loki_data:/loki
    command: -config.file=/etc/loki/config.yml
    ports:
      - "3100:3100"

  grafana:
    image: grafana/grafana:11.0.0
    volumes:
      - ./grafana/provisioning/:/etc/grafana/provisioning/
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
    ports:
      - "3000:3000"

  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"

volumes:
  prometheus_data:
  vm_data:
  loki_data:
  grafana_data:

Prometheus’s prometheus.yml wires up remote-write to VictoriaMetrics and connects the alerting pipeline:

# prometheus.yml (partial)
global:
  scrape_interval: 15s
  evaluation_interval: 15s

remote_write:
  - url: http://victoriametrics:8428/api/v1/write

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

Nothing here is exotic. The magic is that all five services cooperate through well-defined interfaces: Prometheus remote-writes to VictoriaMetrics, ships alerts to Alertmanager, and Grafana reads from both VictoriaMetrics and Loki as data sources. You can swap any single component without touching the others.

The Blue/Green Wrinkle

Here’s where the setup gets interesting. A blue/green deploy runs two container fleets side by side: blue (current) and green (next). Traffic flips to green only after health probes pass. If the probes fail, traffic stays on blue and the green fleet is torn down. A deploy orchestrator drives all of it, watching health endpoints and toggling the load balancer.

Scraping both fleets cleanly is the tricky part. Prometheus needs to know about containers in both the blue and green environments, and dashboards shouldn’t break just because you’re mid-deploy.

I solved this with relabeling rules. Each container exposes a fleet label (blue or green) and a deploy_generation label (monotonically increasing counter). Prometheus picks up both fleets via Docker service discovery, and relabeling normalizes the job names:

# prometheus.yml - scrape config for the app fleet
scrape_configs:
  - job_name: 'app-fleet'
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 10s
    relabel_configs:
      # Extract fleet color from container label
      - source_labels: [__meta_docker_container_label_fleet]
        target_label: fleet
      # Normalize service name regardless of fleet color
      - source_labels: [__meta_docker_container_label_service]
        target_label: service
      # Tag with deploy generation for post-deploy debugging
      - source_labels: [__meta_docker_container_label_deploy_gen]
        target_label: deploy_generation

The result: Grafana dashboards query by service and don’t care whether they’re showing blue or green metrics. During a deploy, you see both fleets on the same graph. After traffic flips and the old fleet drains, the stale series go quiet and eventually get reaped.

The deploy orchestrator also feeds Prometheus annotations, so you can see exactly when a deploy started, when health probes passed, and when traffic flipped. This makes correlation during incidents trivially easy. When something breaks two minutes after a deploy, you don’t have to ask “did we just deploy?” The annotation is right there on the graph.

Alerts That Don’t Cry Wolf

This is the section I wish I’d read before my first month of self-hosted monitoring, when I was averaging four false-positive pages per night.

Rule 1: Alert on symptoms, not causes. Don’t alert on “CPU is above 80%.” Alert on “p99 response latency has exceeded 500ms for 5 minutes.” CPU at 80% might be perfectly fine during a batch job. A service that’s slow for users is never fine.

Rule 2: Burn-rate beats static thresholds. Instead of “error rate > 1%,” use multi-window burn-rate alerts from the SLO model. A brief spike during a deploy is normal. A sustained burn that’s chewing through your monthly error budget is not. Here’s a simplified version:

# rules/slo-burn-rate.yml
groups:
  - name: slo-burn-rate
    rules:
      # Fast burn: 14x consumption over 1h (pages immediately)
      - alert: HighErrorBurnRate_Fast
        expr: |
          (
            sum(rate(http_requests_total{code=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > (14 * 0.001)
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "SLO burn rate is 14x normal over 1h window"
          runbook: "https://wiki.internal/runbooks/high-error-burn"

      # Slow burn: 3x consumption over 6h (tickets, not pages)
      - alert: HighErrorBurnRate_Slow
        expr: |
          (
            sum(rate(http_requests_total{code=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > (3 * 0.001)
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "SLO burn rate is 3x normal over 6h window"
          runbook: "https://wiki.internal/runbooks/slow-error-burn"

Rule 3: Every alert must have a runbook link. Not “investigate the issue.” An actual link to a page that says: “Check dashboard X. If you see pattern Y, run command Z.” If you can’t write a runbook for an alert, the alert isn’t actionable, and it shouldn’t page anyone.

After implementing these three rules, the false-positive rate dropped from about 30 alerts per week to under 5. The on-call rotation stopped being a punishment.

Retention and Cardinality on a Budget

If you’re running this stack on a couple of dedicated servers instead of a fleet of cloud VMs, you need to think about cardinality and disk. High-cardinality labels are the silent killer of Prometheus-style metric stores.

Label hygiene matters. Never put user IDs, request IDs, or unbounded values into metric labels. Every unique label combination creates a new time series. I learned this the hard way when a well-meaning developer added a request_path label to an HTTP histogram. In a REST API with path parameters, that’s infinite cardinality. The VictoriaMetrics instance ate 300 GB in a weekend before anyone noticed.

Recording rules for dashboard queries. If a dashboard panel runs an expensive query (joins, high-cardinality aggregations), create a recording rule that pre-computes the result:

# rules/recording.yml
groups:
  - name: recording-rules
    rules:
      # Pre-aggregate request rate by service
      - record: service:http_requests:rate5m
        expr: sum by (service) (rate(http_requests_total[5m]))

      # Pre-compute p99 latency by service
      - record: service:http_duration_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (service, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

Disk budgeting. Here’s what this setup actually uses. With ~800 active time series per host, 40 hosts, a 15-second scrape interval, and 14-month retention in VictoriaMetrics: roughly 180 GB. Loki with 50 GB/day of logs and 30-day retention: about 400 GB after compression. Total: one 2TB NVMe drive with plenty of room. Not exotic hardware.

The Honest Cost of Self-Hosting

Here’s where I have to be fair. Self-hosting doesn’t mean free. The SaaS bill goes away, but you take on real costs:

You own the upgrades. Prometheus, VictoriaMetrics, Loki, and Grafana all release regularly. Budget one afternoon per quarter for upgrades. Occasionally a release has breaking changes (Loki’s schema v12 to v13 migration was a memorable afternoon). You need someone on the team who’s comfortable reading changelogs and testing upgrades in staging first.

You own the backups. VictoriaMetrics supports snapshots, so a scheduled snapshot shipped off to object storage is not hard to set up. But if that job quietly breaks and you lose metric history, there’s no support ticket to file.

You own the scaling. When the fleet grew by another 15 hosts, I had to tune Prometheus scrape timeouts and VictoriaMetrics memory limits. Took about two hours. A SaaS vendor handles this automatically.

You own the on-call for your monitoring. Yes, the irony. Who monitors the monitoring? The answer is a dead-simple external healthcheck (a $5/month uptime service pinging Grafana’s health endpoint) that alerts to a phone number if the monitoring stack itself goes down.

When SaaS Is the Right Call

I’m not ideological about this. SaaS observability makes sense when:

Your team is fewer than five engineers and nobody wants to own infrastructure.
Compliance requirements mandate specific certifications that SaaS vendors already have.
You need distributed tracing at scale. Self-hosted Jaeger or Tempo works, but the operational cost is genuinely higher than metrics and logs.
Your time is genuinely more expensive than the bill. Not everyone’s is.

In my case, the math worked out to about $800/month in hardware costs and roughly 4 hours of ops time per month. Compared to $12,000/month for the SaaS equivalent. Your math might look different.

Takeaways

After running this stack for over a year in production, here’s what I’d tell someone starting down the same path:

Start with Prometheus and Grafana. Add VictoriaMetrics when you outgrow the default 15-day retention. Add Loki when you’re tired of SSH-ing into boxes to grep logs.
Get alert design right before you add more data. More metrics won’t help if every alert is noise. Invest in symptom-based, burn-rate alerting from day one.
Label discipline is non-negotiable. Document your labeling conventions. Review metric instrumentation in code review the way you’d review a database schema change.
Automate the boring parts. Config management, backups, upgrades. The stack itself is simple. Letting the maintenance rot is what kills self-hosted setups.
Keep one external healthcheck. Your monitoring can’t tell you it’s down. Spend $5/month on something outside the blast radius.
Know your exit. If the team shrinks or priorities shift, you can migrate to a SaaS vendor. Prometheus-format metrics and LogQL aren’t proprietary. That portability is worth something.

What a SaaS observability invoice really buys you is convenience and staffing, not capability. The capability is all open source, well-documented, and battle-tested. The question is whether your team wants to own the operational burden. For me, the answer was yes, and the monitoring stack turned out to be one of the most reliable parts of the infrastructure.

Sometimes the best investment is the bill you decide not to pay.