Prometheus metrics

Enterprise

Fadebox reads what every runtime and container is costing on a timer, and shows it in the runtimes list, on each instance and on the dashboard. The exporter is the other half: the same numbers as Prometheus series, next to the node-exporter and the alert rules you already run — plus what Micrometer knows about the Fadebox process itself, because the other half of what anyone scrapes a service for is whether the service is well.

Switching the exporter on is the Enterprise act. Everything else about it is not: the scrape answers on every tier once it is on, switching it off is never gated, and collecting and showing what a host costs in the interface is free everywhere. An exporter enabled under a license that has since lapsed keeps answering — dashboards do not go dark on renewal day.

Turning it on

Administration → System → Monitoring, one switch. Below Enterprise the switch is disabled and says so before you meet the refusal.

Off, the scrape endpoint is 404 — not enabled, which is a different fact from not allowed, and the status says which.

The switch writes system.metrics_export_changed to the audit log.

Scraping it

GET /api/metrics, in the Prometheus text format, behind the global admin role. Give the scraper a service account that holds admin and an API key of its own, rather than a person's:

scrape_configs:
  - job_name: fadebox
    metrics_path: /api/metrics
    scheme: https
    static_configs:
      - targets: ['fadebox.example.com']
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/fadebox-api-key

A scrape renders what the last sweep registered. It touches neither a Docker daemon nor the database, so a hung daemon shows up as fadebox_runtime_up 0 and a stale fadebox_runtime_observed_timestamp_seconds, never as a hung scrape. The freshness is the sweep's, not the scrape's: once a minute for CPU and memory, once a quarter hour for disk, both tunable in configuration.

There is exactly one door, and it is this one: the metrics endpoint the underlying framework publishes at /q/metrics is denied outright, since an unauthenticated one would hand every project's figures to anyone who can reach the port.

What it exports

All gauges, all base units, prefixed fadebox_. A series appears when a container does and is gone from the next scrape after the container is.

Series	Labels	What it is
fadebox_runtime_up	runtime	1 when the daemon answered the last sweep, 0 when it did not.
fadebox_runtime_cpus	runtime	Logical processors the daemon reports.
fadebox_runtime_memory_bytes	runtime	The host's physical memory, as the daemon reports it.
fadebox_runtime_disk_bytes	runtime, kind	Daemon-wide bytes stored, by kind: images, volumes, containers.
fadebox_runtime_observed_timestamp_seconds	runtime	When the sweep last read this runtime.
fadebox_container_cpu_cores	runtime, kind, project, environment, instance, service	CPU in cores, averaged over the sweep interval. Absent on a container's first observation.
fadebox_container_memory_bytes	the same	Memory in use, without the page cache.
fadebox_container_memory_limit_bytes	the same	The cgroup limit, or the host's total where the service set none.
fadebox_container_rootfs_rw_bytes	the same	The container's writable layer, from the quarter-hourly disk pass.
fadebox_instance_volumes_bytes	runtime, project, environment, instance	Managed volumes the instance owns, from the same pass.
fadebox_instances	runtime, project, status	Environment instances, counted by lifecycle status. Drafts have no runtime and are not counted.

kind on a container series is instance, test (a template test run, which carries the template as its instance) or system — Fadebox's own ingress stack, which has no project or environment. So the previews and the plumbing can be told apart in a query rather than guessed at.

Runtime totals are deliberately not published. sum by (runtime) (fadebox_container_memory_bytes) is one line of PromQL, and publishing the same number twice is an invitation for the two to disagree.

An elastic runtime publishes container series and no fadebox_runtime_* series at all. Every family in that group describes a daemon — whether it answered, its processors, its memory, what its disk holds, when it was last read — and such a runtime has none: it is one machine per instance. Summing a fleet into a series that used to mean this host would have broken every existing alert quietly, so the series is absent instead. How many machines a runtime is holding is the machines listing's answer, not a gauge. The container series are published, with the same labels as on a host and no machine one — a machine holds exactly one instance, so project, environment and instance already name it. fadebox_container_rootfs_rw_bytes and fadebox_instance_volumes_bytes are the exception: they come from the disk pass, which does not visit machines.

Beside these, the exporter carries the standard process families — JVM heap and threads, HTTP server timings, the database pool — under their usual names.

Queries worth having

# A daemon Fadebox cannot reach right now
fadebox_runtime_up == 0

# Memory a project's previews are using, across the fleet
sum by (project) (fadebox_container_memory_bytes{kind="instance"})

# Containers within 10% of their own memory limit
fadebox_container_memory_bytes / fadebox_container_memory_limit_bytes > 0.9

# Disk a runtime is spending on images
fadebox_runtime_disk_bytes{kind="images"}

# Instances stuck in ERROR
sum by (project, runtime) (fadebox_instances{status="ERROR"})

# The sweep has stopped reading a runtime
time() - fadebox_runtime_observed_timestamp_seconds > 300

Where a series is missing rather than zero, that is the same rule the interface follows: a runtime nobody has swept yet, and one whose daemon was unreachable, report no figure rather than an idle host. Alert on fadebox_runtime_up and on the observation timestamp; do not read a gap as a quiet host.