Setting up an elastic runtime

Elastic runtimes are available on every tier. Their machines are bounded by the per-runtime instance cap — one machine per instance — see how machines count.

An elastic runtime creates one virtual machine per instance instead of sharing a Docker host. Elastic runtimes explains what that means for the people deploying; this page is the administrator's half — the prerequisites, the machine image, the cloud project it runs in, and how a runtime and the machine sizes it offers are defined.

Before you start

A front runtime

A machine has no DNS name, no certificate and no public address: all traffic to an elastic instance enters through the front, and an ingress router on the machine does the last hop. So an elastic runtime has no ingress settings of its own and names a front instead: a static runtime that already has

• an ingress domain with its wildcard DNS record, and
• the managed ingress stack installed on it.

Both are checked when the elastic runtime is saved. A front with a domain but no stack would leave every instance behind it on a URL nothing serves, so it is refused (elastic_front_not_installed, 409, naming the front in args.front) — on a create, and on an update that moves the runtime to another front. An update that sends the front the runtime already has is not asked again, so the retention or the account can still be edited while the front's stack is down. "Installed" is what Fadebox itself last did to the front: an install or update of its stack counts, a removal undoes it. A stack removed on the host behind Fadebox's back still counts, and the front's Ingress dialog is where that shows. The runtime form can set the front up on the spot.

Instance URLs on the elastic runtime are built from the front's domain, scheme and port, and the front forwards each hostname to the machine behind it. The visitor's TLS ends at the front, and the hop to the machine is a separate mutual TLS connection (see the table below). The front can be the seeded local runtime, or a dedicated host; it does not have to run any instances itself.

The front's managed stack asks Fadebox every three seconds where each hostname lives. It uses FADEBOX_PUBLIC_URL when that is set, and otherwise an address Fadebox works out from the front's daemon, so the setting is not required for an elastic runtime. How the stack reaches Fadebox says when that worked-out address is wrong, when Fadebox refuses to install the stack without the setting — a containerized Fadebox with a remote front, for one — and how the front's Ingress dialog shows whether its stack is actually polling: a stale poll means no instance behind the front is reachable. A machine never calls Fadebox, so it needs neither.

An instance that requires a sign-in is checked at the front, like any gated instance on a static runtime, so gating one needs FADEBOX_PUBLIC_URL just as it does there. A gated deploy onto an elastic runtime is checked against the front's stack before any machine is asked for.

Naming a runtime as a front changes nothing on its stack: every managed stack already polls, and a new elastic runtime's hostnames reach it at the next poll. The one exception is a stack installed by a release before 1.4, which reads outdated until you press Update in its Ingress dialog.

On the Free tier

Free has one runtime, and the front is a runtime too — but only an enabled runtime holds the slot, and the front does not have to be enabled to serve. So the front is local: give it an ingress domain and install its managed ingress stack, then disable it, which frees the slot for the one enabled runtime to be the elastic one. The front keeps serving while disabled — a disabled runtime refuses deploys, not traffic. From then on no instance runs on your own host: every one gets a machine at the provider, up to the Free instance cap of five, and local only carries the ingress in front of them. It is the same "disable local" move a remote runtime uses on a single-runtime tier.

Network reachability

No agent runs on a machine, and a machine never dials Fadebox to register itself. Fadebox dials the machine, exactly as it dials a remote daemon — and by default it dials the machine's internal address, because a machine gets no public one. So Fadebox and the front host both have to be on that network or routed onto it: the same VPC, a peering, or a VPN into it. The one alternative is a profile whose externalAddress is dial: then the machine gets a public address and Fadebox and the front dial that, which is how a control plane outside the network — a laptop, another cloud — reaches its machines. Either way there are four routing requirements, and none of them is a machine calling Fadebox:

From	To	Why
The Fadebox server	each machine, tcp:2376	The Docker daemon, over mutual TLS. This is how everything is deployed.
The front runtime's host	each machine, tcp:443	Instance traffic, to the ingress router on the machine, over mutual TLS: the router serves the machine's own certificate and admits only the front, which presents a client certificate from Fadebox's ingress front CA. All of it hairpins through the front.
Each machine	the image registries it pulls from	Directly, or through a NAT gateway. A machine needs no inbound internet access and gets no public address by default; externalAddress: egress gives it one for this alone.
The front runtime's host	Fadebox	The front's stack polls Fadebox for the routes, and checks a sign-in with it for a gated instance. The front sits next to Fadebox, so this is normally a given.

Fadebox itself also needs outbound access to the provider's API to create machines at all; on Google Cloud that is compute.googleapis.com and oauth2.googleapis.com. If your installation routes outbound traffic through a proxy, the provider uses the same egress proxy settings as everything else.

You give Fadebox two CIDRs when you define the runtime, and they become firewall rules on each machine:

• controlCidr — who may reach the Docker daemon on 2376. This is where the Fadebox server sits.
• frontCidr — who may reach the ingress router on a machine, on port 443. This is where the front host sits. The router demands the front's client certificate on every connection, so the range is the second fence rather than the proof; it is also the only range whose forwarded headers that router believes, so a caller inside the network cannot claim to be visiting some other instance's hostname.

Detect fills controlCidr for you, with up to two candidates, because which address the machines see depends on where Fadebox is relative to them:

• On the provider's network — when Fadebox itself runs on a Compute Engine VM in the machines' VPC, the button offers that VM's internal address, read from Google's metadata server. This is the deployment the design assumes, and it is the case an external "what is my IP" service gets wrong: it would report the NAT gateway, which the machine's firewall would then admit while dropping Fadebox.
• Across the internet — when Fadebox reaches the machines through NAT, the button offers the public address an echo service saw it coming from (FADEBOX_EGRESS_ECHO_URL, asked through the egress proxy and only on the click). The echo has to answer with the address itself: a redirect, or an answer longer than an address, is reported as the echo's failure rather than followed or read.

Pick the one that matches your topology; nothing is written until you do, and the field stays yours to edit. Same as control copies the value into frontCidr, which is right when the front runs on the same host as Fadebox.

Both are required, and neither may be /0: a rule that lets everybody in is a rule that does nothing, and on a platform whose only firewall is the machine's own that is the whole protection. Both must be written canonically — 10.0.0.0/16, not 10.0/16, and four dotted decimal octets with no leading zeros.

Two firewalls, and both are needed. The machine's own is the one Fadebox writes, and it is the only one every platform has — it is what closes 2376 and the router's port to everybody outside those two ranges, on every boot, before the daemon starts. The cloud's is yours: Fadebox creates no firewall rules on any platform, so on a network that denies ingress by default nothing reaches a machine until you have opened those two ports to the same ranges on the network tag. The host rule without the cloud rule is a machine nothing can reach; the cloud rule without the host rule would be a daemon open to the whole VPC.

The machine image

A machine boots an image that already carries Docker Engine and cloud-init, because the cloud-init document Fadebox renders assumes them and installs nothing. You do not have to build one.

Fadebox publishes one for Google Compute Engine, the image family projects/fadebox/global/images/family/fadebox-runtime. A machine profile that leaves its boot image on Fadebox runtime image boots it. It is built from the reference Packer template below to the contract on this page, with the ingress router's images baked in, and it is public to every authenticated Google account, the way Google's own debian-cloud images are. The family has no version in its name: the contract only ever gains requirements, so the newest image in it boots every Fadebox release.

Organisations that restrict image projects

An organisation that enforces the policy constraint constraints/compute.trustedImageProjects refuses to boot an image from a project it does not trust. Add projects/fadebox to the trusted projects, or give the profile an image of your own.

Build your own image when the published one does not suit you: a hardened base your security team signs off, extra tooling on every machine, or an organisation that will not boot images from outside its own projects. Such an image must meet the contract below, and the Packer template is the place to start. A profile names it as a custom boot image. To move every profile that does not name one, set FADEBOX_MACHINES_GCP_DEFAULT_IMAGE to your own family instead.

What the image must have

Requirement	Why
Linux with systemd	The daemon's TLS configuration and the firewall are systemd units, enabled and started with systemctl.
cloud-init	It is how Fadebox gets anything onto the machine at all. The document is a #cloud-config and uses two modules, write_files and runcmd — nothing exotic has to be enabled. On Compute Engine it arrives as the instance's user-data metadata key, which is cloud-init's own GCE datasource: Ubuntu's cloud images carry both already, and a Debian image may need cloud-init installed.
Docker Engine, baked in at /usr/bin/dockerd	The drop-in Fadebox writes names that exact path, and installing Docker at boot would add a minute to every cold start. The unit may ship disabled — the document enables docker.service on the first boot — but it must be the packaged unit with its [Install] section, and systemctl must be at /usr/bin/systemctl, which the drop-in runs before every start of the daemon.
openssl 3.0 or newer, with the ca subcommand	The machine issues its own daemon certificate at every boot, for explicit start and end dates. openssl x509 -req could not be told a start date before 3.4, so openssl ca is what is used — a build trimmed of it will not do.
iproute2	ip is what reads the addresses the certificate is issued for.
curl, for externalAddress: dial	A NAT address is on no interface, so the boot script asks the platform's metadata service for it and puts the answer into the certificate — on Google, the attached external IP. Without curl a dialled machine's certificate names its internal address only and is never verified.
iptables and iptables-restore	The host firewall is applied as one transaction. Add ip6tables and ip6tables-restore if you give this runtime an IPv6 CIDR, or if the machine gets a global IPv6 address at all: the firewall never leaves an address family unfiltered, so without them such a machine refuses to come up.
sysctl	On a machine with no IPv6 CIDR and no global IPv6 address, the firewall turns IPv6 off rather than leave it unfiltered — a container could otherwise reach the metadata service over it — and the machine refuses to boot if it cannot.
The kernel's conntrack match for iptables	The rule that protects the ingress router's port matches the port the traffic was published on, which only that match can see. Without it the firewall will not apply — and the daemon is ordered behind the firewall, so the machine fails visibly instead of coming up open.
GNU or BusyBox date, plus awk, cut, sleep, mkdir, chmod, mv, rm and a POSIX /bin/sh	The boot script is plain sh and calls these. date -d @<epoch> is the one a BSD date does not understand.
The platform's time sync left on	Both certificates are backdated a day, so an hour of drift is harmless. A clock more than a day out in either direction makes every dial fail as not yet valid, which reads exactly like an expired certificate.

Either iptables backend works — Fadebox calls whatever iptables and iptables-restore resolve to on the machine — provided Docker is using the same one, since the rules go into the daemon's own chain. Leave the distribution's default alternative alone rather than switching it by hand.

What the image must not carry

Keep out	Why
A second firewall manager — firewalld, ufw	Either of them rewriting the INPUT chain takes Fadebox's rules with it, and nothing puts them back before the next boot: the daemon's port would be open with nothing saying so.
A Docker that listens on TCP	The image's own docker.service starts once before cloud-init has written the TLS drop-in and the firewall. On a Unix socket that first start is harmless. A -H tcp://… in an override the image already carries would make it an open daemon for the length of the boot.
A hosts key in /etc/docker/daemon.json	dockerd refuses the flag and the file together — "the following directives are specified both as a flag and in the configuration file: hosts" in the journal — so the machine never comes up at all.

On Google Compute Engine the machine's address is read from the instance itself, so no guest agent is needed. Platforms that cannot report an address that way read it through a guest agent, and their images will need one; no such provider exists yet. A machine that comes up with no global address does not start Docker at all: the boot script waits a minute for one and then refuses, rather than issuing a certificate Fadebox could never verify, and the provision timeout names the cause.

Bake the ingress router's images in

Not a requirement, but the one that decides how long a cold start takes. Every machine runs a small Traefik plus a read-only Docker socket proxy — its ingress router — and Fadebox starts it during the boot, between the machine's Docker daemon answering and the machine being marked ready.

Image	Why it is there
traefik:v3.7	The router itself: the one hop from the port the front may reach to the container the hostname names.
tecnativa/docker-socket-proxy:v0.5.0	The read-only window Traefik watches the daemon through; Traefik never sees the raw socket.

On an image that does not already carry them, that is two registry pulls on every cold start — and they happen on Fadebox's machine sweep, which works through the fleet one machine at a time, so ten machines booting together wait for each other's pulls. Bake both into the machine image alongside Docker Engine and the router starts in milliseconds and never reaches a registry at all.

Bake them under the names Fadebox will ask for — the two above, unless this installation rewrites its system images through an image registry prefix, which applies to a machine's router as it does to a runtime's ingress stack.

If you cannot bake them in, make sure the two are reachable from the machine network, or mirror them with that prefix. The pull uses the elastic runtime's own registry credentials — the same ones every instance deployed onto its machines pulls with, never the front's — so a router that could not reach a mirror its workloads can is a machine that only half works.

What Fadebox puts on the machine at boot

The cloud-init document is rendered by Fadebox, once, per machine, and the machine is the only place it goes. In plain words, it does five things:

1. Issues the machine's own daemon certificate. Fadebox generates a certificate authority that vouches for exactly one machine, and a client certificate under it. The CA certificate and its key travel in the cloud-init document and live root-only on the machine, in /etc/fadebox/ca, a directory no container is given; a unit that runs before Docker on every boot issues a server certificate for whatever addresses the machine actually has, and then starts dockerd on tcp://0.0.0.0:2376 --tlsverify trusting that CA. A machine that comes back on a different address re-issues and needs nothing from Fadebox.

   A CA key on the machine sounds wrong and is not: it vouches for one machine, and whoever can read a root-only file there already owns that daemon. The ingress router, the one process on the machine a visitor's request reaches, is handed only the daemon's serving pair in /etc/fadebox/docker-tls, never the CA. A machine that gets no address does not start Docker at all — it fails, rather than issuing a certificate nothing can verify, and the provisioning timeout then names the cause.

2. Gives the ingress router the same certificate, and the front's CA. The router Fadebox starts on the machine serves that same server certificate on port 443 and admits only a client certificate issued by Fadebox's ingress front CA — one per installation, minted on the first machine you provision, whose key never leaves Fadebox. The front presents the client certificate issued from it: Fadebox places that certificate and its key in the front's Traefik container when it applies the ingress stack, and the routes the front polls only name them. The daemon keeps trusting the machine's own CA alone, so the front's credential opens a route and never a daemon. The document writes the front CA certificate and the router's TLS file, which names the daemon's pair; nothing is issued twice, and a re-issue after a restart serves both. So the hop from the front is mutual TLS like the daemon's, whether it stays in a VPC or crosses the internet to a dial machine.

3. Sets the host firewall. Port 2376 is open to controlCidr only, and port 443 to frontCidr only. Nothing else is opened, and a container may not publish a port over 2376. The daemon is bound to the firewall unit, so if the firewall does not apply, Docker does not start, on the first boot as on every later one. An IPv6 family the firewall cannot filter is switched off, as the sysctl row above says.

4. Cuts containers off from the instance metadata service. The cloud-init document is readable from the metadata server, so a workload has no business reaching it. The rules cover both address families whether or not the platform has one.

5. Leaves everything else alone. Ports a template publishes itself are not touched.

Then the document is taken back. Once the machine's daemon answers with a certificate its own CA issued — proof that the machine has read the document — Fadebox asks the provider to drop it from the machine's metadata. On Compute Engine that is one setMetadata that removes the user-data item and keeps every other item exactly as it was, block-project-ssh-keys and your own ssh-keys among them; Fadebox waits until a read of the instance no longer shows the item. The machine does not wait for this: it is a host as soon as it is ready. Nothing needs the document again — a stop and start runs the firewall, the certificate script and Docker from the machine's disk, and cloud-init does not re-run those parts on the same instance.

A failed removal never fails the machine and is not shown on the instance or in the machine log; it is a warning in the server log, naming the provider and its reason, and Fadebox asks again about once a minute until one takes. The likely cause is a service account without compute.instances.setMetadata — see the permissions. Machines that were already running when you upgraded to 1.4 are cleaned up on the first such check, and a stopped one on its next start.

The keys are EC P-256 and last ten years — a machine is destroyed long before that, and destroying it, not an expiry, is how access is revoked.

The cloud-init document is readable in your cloud account

Until the machine is ready and the document is taken back, anyone who can read a machine's metadata in the provider account — on Google Cloud, anyone with compute.instances.get — can read its cloud-init document, and that document contains the machine's CA key. Whoever has that key can mint a certificate that this machine's Docker daemon accepts, which is root on it and on everything its instance holds. It reaches nothing else: the CA is single-use, vouches for one machine, and nothing else in the installation trusts it, so another machine, another runtime and Fadebox itself are all out of its reach. The window is the minute or two from create to ready, and longer only for a machine whose removal keeps failing, which the server log names every minute. Reading a project is a far weaker right than logging into a machine, so during boot, read access on the project is access to that machine's daemon. Scope the project, and the roles on it, accordingly.

A reference Packer template

Below is the shortest image that satisfies the contract above, for Google Compute Engine, with Packer's googlecompute builder. It is also a file you can download.

The published Fadebox runtime image is built from this template, so you need it only for an image of your own. Read it as a starting point for that rather than as a product: if it and the contract above ever disagree, the contract is what the code needs.

fadebox-gce.pkr.hcl

packer {
  required_plugins {
    googlecompute = {
      source  = "github.com/hashicorp/googlecompute"
      version = "~> 1"
    }
  }
}

variable "project" {
  type        = string
  description = "The project id the image is built in and published to."
}

variable "zone" {
  type        = string
  default     = "europe-west3-a"
  description = "Where the short-lived build instance runs. The image itself is global."
}

// Ubuntu LTS carries cloud-init and its Compute Engine datasource already, which is why it is
// the default here. Debian works the same way — set source_image_project_id to "debian-cloud"
// and source_image_family to "debian-12" or "debian-13"; the provisioner below installs
// cloud-init where the base image has none.
variable "source_image_project_id" {
  type    = string
  default = "ubuntu-os-cloud"
}

variable "source_image_family" {
  type    = string
  default = "ubuntu-2404-lts-amd64"
}

variable "image_family" {
  type        = string
  default     = "fadebox-machine"
  description = "The family a machine profile's `image` parameter names."
}

// The two images every machine's ingress router runs. Baking them in is what keeps a cold start
// off a registry entirely. The tags are Fadebox's own pins and move with each release — take
// them from the version of Fadebox you run, not from here. An installation that rewrites system
// images through a registry prefix must list the *prefixed* references instead, exactly as
// Fadebox will ask for them, and log the build instance into that registry first.
variable "router_images" {
  type    = list(string)
  default = ["traefik:v3.7", "tecnativa/docker-socket-proxy:v0.5.0"]
}

variable "machine_type" {
  type    = string
  default = "e2-standard-2"
}

variable "disk_size" {
  type        = number
  default     = 20
  description = "The build disk in GB. Instances get their own size from the machine profile."
}

locals {
  image_name = "fadebox-machine-${formatdate("YYYYMMDD-hhmmss", timestamp())}"
}

source "googlecompute" "fadebox" {
  project_id              = var.project
  zone                    = var.zone
  source_image_project_id = [var.source_image_project_id]
  source_image_family     = var.source_image_family

  image_name        = local.image_name
  image_family      = var.image_family
  image_description = "Fadebox elastic-runtime machine: Docker Engine, cloud-init, ingress router images"

  machine_type = var.machine_type
  disk_size    = var.disk_size
  ssh_username = "packer"

  // The build instance needs a route out to Docker's apt repository and to the registry the
  // router images come from. If your policy forbids external addresses, set omit_external_ip
  // and use_internal_ip to true and give the build subnet Cloud NAT.
}

build {
  sources = ["source.googlecompute.fadebox"]

  provisioner "shell" {
    execute_command = "sudo -E bash -c '{{ .Vars }} {{ .Path }}'"

    environment_vars = [
      "DEBIAN_FRONTEND=noninteractive",
      "ROUTER_IMAGES=${join(" ", var.router_images)}",
    ]

    inline_shebang = "/bin/bash -eux -o pipefail"
    inline = [
      // The base image's own first boot has to finish before apt is usable.
      "cloud-init status --wait || true",

      // --- what the bootstrap script calls, and cloud-init to run it --------------------
      "apt-get update",
      "apt-get install -y ca-certificates curl gnupg cloud-init openssl iproute2 iptables",

      // Nothing else may manage the firewall: anything that rewrites INPUT takes Fadebox's
      // jump with it, and nothing puts it back before the next boot.
      "apt-get purge -y ufw firewalld || true",

      // Pin the datasource. Compute Engine hands cloud-init the user-data metadata key, and
      // saying so keeps the first boot from probing for anything else.
      "printf 'datasource_list: [ GCE ]\\n' > /etc/cloud/cloud.cfg.d/90-fadebox-datasource.cfg",

      // --- Docker Engine, from Docker's own repository ---------------------------------
      "install -m 0755 -d /etc/apt/keyrings",
      ". /etc/os-release",
      "curl -fsSL \"https://download.docker.com/linux/${ID}/gpg\" -o /etc/apt/keyrings/docker.asc",
      "chmod a+r /etc/apt/keyrings/docker.asc",
      "echo \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/${ID} ${VERSION_CODENAME} stable\" > /etc/apt/sources.list.d/docker.list",
      "apt-get update",
      "apt-get install -y docker-ce docker-ce-cli containerd.io",
      "systemctl enable docker",

      // --- the two things the image must NOT do ----------------------------------------
      // A `hosts` key in daemon.json and the -H flags Fadebox's drop-in passes are mutually
      // exclusive: dockerd refuses both together and the machine never comes up. A baked-in
      // TCP listener would be an open daemon for the length of every boot, before cloud-init
      // has written the firewall.
      "if [ -f /etc/docker/daemon.json ] && grep -q '\"hosts\"' /etc/docker/daemon.json; then echo 'daemon.json sets hosts; remove it' >&2; exit 1; fi",
      "if grep -rl -- '-H tcp://' /etc/systemd/system/docker.service.d /etc/default/docker 2>/dev/null | grep .; then echo 'the image starts dockerd on TCP; remove that override' >&2; exit 1; fi",

      // --- bake the ingress router's images in ------------------------------------------
      "systemctl start docker",
      "for image in $ROUTER_IMAGES; do docker pull \"$image\"; done",

      // --- prove the contract before the image is published ------------------------------
      "test -x /usr/bin/dockerd",
      // the drop-in runs /usr/bin/systemctl before every start of the daemon, and the boot
      // document enables docker.service itself, which needs the packaged unit's [Install] section
      "test -x /usr/bin/systemctl",
      "systemctl cat docker.service | grep -q '^\\[Install\\]'",
      "for binary in openssl ip iptables iptables-restore sysctl awk cut date; do command -v \"$binary\" >/dev/null || { echo \"missing $binary\" >&2; exit 1; }; done",
      "openssl version",
      "iptables -m conntrack --help >/dev/null",

      // --- leave it as if it had never booted --------------------------------------------
      "systemctl stop docker docker.socket containerd || true",
      "apt-get clean",
      "rm -rf /var/lib/apt/lists/*",
      "rm -rf /root/.docker /home/packer/.docker",
      // Without these two the machine keeps the build's identity and cloud-init decides it has
      // already run, so no instance would ever receive its user-data.
      "cloud-init clean --logs --seed",
      "truncate -s 0 /etc/machine-id",
      "rm -f /var/lib/dbus/machine-id",
    ]
  }
}

Build it, and name what it published in the profile as a custom boot image:

packer init fadebox-gce.pkr.hcl
packer build -var project=acme-dev-7f2c -var zone=europe-west3-a fadebox-gce.pkr.hcl

image = projects/acme-dev-7f2c/global/images/family/fadebox-machine

Four assumptions worth knowing before you run it:

• The identity that builds is not the one Fadebox uses. Packer authenticates the way gcloud does and needs to create instances, disks and images in the build project. That is a person's grant; the service account below never builds anything.
• The build instance needs a route out — to Docker's apt repository and to the registry the two router images come from. The template gives it an external address by default; a policy that forbids one wants Cloud NAT on the build subnet instead.
• The router tags are Fadebox's, and they move. Take them from the release you run rather than from this file, and if your installation rewrites system images through a registry prefix, put the prefixed references in router_images and log the build instance into that registry — the machine has to hold them under the name Fadebox will ask for.
• cloud-init clean and the emptied machine-id are what make the image reusable. Without them every machine built from it would carry the build's identity and cloud-init would decide it had already run, so no instance would ever receive its configuration.

Defining the runtime

Everything below is admin-only, and there are two ways to do it: the form under Administration → Runtimes, and the API. They do the same thing; the API is what a script or a second installation wants, and the numbered steps below are written against it because they are also the shape of the form.

In the form

Administration → Runtimes → Add Runtime. Placement is the first choice — Docker host or Elastic — a machine per instance — and it is the one thing that cannot be changed afterwards; on an existing runtime it is shown as text. The elastic choice is disabled, with a line saying so, when this build carries no provider module at all.

Choosing it opens a Machines block, which leads with the two facts a deployer will meet first — that a deploy waits for a machine to boot, and that host-published ports answer only inside the machine network — and then asks for:

• Provisioner, which is also fixed once the runtime exists.
• The provider account, whose fields are the provisioner's own rather than anything Fadebox knows: they are rendered from its descriptor, marked required or not by it, and shown as password fields where it says the value is secret. On an existing runtime the fields that are not secret show what is stored. A stored secret starts empty with set — leave empty to keep: leave it empty to keep it, or fill it in to rotate it.
• Front runtime, offered from the static runtimes that already have an ingress domain. Choosing one reads its managed ingress stack: an installed stack gets one line (with a note to Update it from that runtime's Ingress dialog if it is older than its saved settings), and one that is not installed gets an Install button that installs it there and then, so the save is not refused for it. On an edit this is asked only when the front changes. If no runtime has an ingress domain yet, a new runtime's form offers to create a front in place instead of an empty list: a slug, its Docker host, its ingress domain (with the wildcard DNS record pointing at that host) and its TLS — HTTP only, or HTTPS on Traefik's own self-signed certificate. It is an ordinary static runtime and can run instances of its own; for Let's Encrypt, which needs a DNS provider and its credentials, edit it afterwards. Once created it is picked as the front, and its stack still needs the Install above.
• Control CIDR and Front CIDR, checked as you type — a /0, a mixed notation or a prefix out of range is refused with the reason.
• Stopped machine retention, in days, empty meaning never.

Saving a new elastic runtime tells you the thing you are about to trip over: it offers no machine size yet, so nothing can be deployed onto it, with a button that opens the Profiles tab of the runtime's machines sheet. Until a size exists, the runtime's row also says no profiles, and clicking that opens the same tab. It is where the sizes are added, edited, deleted and moved between. The first one added becomes the default; the default moves only by making another one the default, and it cannot be deleted while other sizes exist.

A profile's Boot image is a choice of two:

• Fadebox runtime image, selected when you add a profile. It shows the family it boots and the note about organisations that restrict image projects. Leave it here unless you have a reason not to.
• Custom boot image, a text field for an image or family of your own, built to the contract. An empty custom field is refused; choose Fadebox runtime image instead.

Editing a profile that names its own image opens on Custom boot image, filled in. In the list of profiles, one that uses the default reads image=default.

An elastic row in the runtimes table shows its provisioner and its front instead of a daemon URI, has no Ingress button, and says in place of the managed-resources panel that there is no host here to scan or reclaim.

Over the API

The examples use:

HOST=https://fadebox.example.com
AUTH="Authorization: Bearer $FADEBOX_API_KEY"
JSON='Content-Type: application/json'

1. See which providers this build carries

curl -sf -H "$AUTH" "$HOST/api/provisioners"

The answer is the provider modules compiled into this binary, each with the account fields you fill in once and the fields a machine profile is made of. Abridged — Compute Engine's full lists are further down:

[
  {
    "id": "gcp",
    "displayName": "Google Compute Engine",
    "configFields": [
      { "name": "serviceAccountKey", "label": "Service account key",
        "description": "The whole JSON key file of a service account that may create and delete instances in this project. Stored encrypted and never shown again.",
        "required": true, "secret": true },
      { "name": "project", "label": "Project ID",
        "description": "The project id, not its number and not its display name — for example acme-dev-7f2c.",
        "required": true, "secret": false }
    ],
    "profileFields": [
      { "name": "machineType", "label": "Machine type",
        "description": "A machine type in this zone, for example e2-standard-4, or a full machine-type URL.",
        "required": true, "secret": false }
    ]
  }
]

A field marked secret is a credential: write-only on the wire, never logged, encrypted at rest. The provider declares which ones those are — nothing infers it from the name.

From 1.4 the list carries Google Compute Engine, as gcp. A build whose list is empty cannot create an elastic runtime at all.

2. Create the runtime

placement: "ELASTIC" and the elastic block, on the ordinary POST /api/runtimes:

curl -sf -X POST -H "$AUTH" -H "$JSON" "$HOST/api/runtimes" -d '{
  "slug": "gcp-eu",
  "name": "Google Cloud, europe-west3",
  "placement": "ELASTIC",
  "shared": true,
  "elastic": {
    "provisioner": "gcp",
    "config": {
      "serviceAccountKey": "{ … the whole JSON key file … }",
      "project": "acme-dev-7f2c",
      "zone": "europe-west3-a",
      "network": "fadebox-vpc",
      "subnetwork": "fadebox-eu",
      "networkTags": "fadebox-machine"
    },
    "frontRuntime": "local",
    "controlCidr": "10.20.0.0/24",
    "frontCidr": "10.20.1.0/24",
    "stoppedRetentionDays": 14
  }
}'

What the fields mean and what the rules are:

• placement and provisioner are fixed at creation. A static runtime never becomes elastic, and an elastic runtime never changes provider: every machine under it is a handle only the provider that made it can read. A second provider, or a second zone, is a second runtime.
• config is the provider account. It is stored encrypted. Which keys exist, and which of them are secret, is the provisioner's descriptor, from step 1. A read gives an admin the fields that are not secret, as config, and names the secret fields that hold a value in secretsSet; a secret's value never comes back. On a PUT, the fields that are not secret are replaced as sent (one left out or blank is removed), and a secret left out or blank keeps its stored value, so editing the zone does not mean pasting the key again. Leaving config out altogether leaves the stored account untouched.
• The account is checked by key, never by value. A required field left out or a key the provisioner does not take is refused with provisioner_config_invalid, naming what is missing and what is unknown. No value of an account is ever echoed back, logged or shown.
• frontRuntime is the slug of the static runtime that serves this one's URLs. It must have an ingress domain and its managed ingress stack installed — install it first with POST /api/runtimes/{front}/ingress, or the create is refused with elastic_front_not_installed (409). An update is asked only when it changes the front. A runtime that fronts an elastic one cannot be deleted while it does.
• Where the machines are is fixed while there are machines. Changing the account's project or zone, or frontRuntime, on a runtime that still has machines is refused with elastic_placement_fixed (409), naming the fields and the machines: Fadebox would lose track of machines in the old zone, and the old front's URLs would stop resolving. Sending the value the runtime already has is not a change, and rotating the key is always allowed. To move, stop and delete the instances first, or create a second runtime.
• stoppedRetentionDays is how long a switched-off machine keeps its disk. 0 or omitted means never — the disk is kept until somebody deletes the instance. See retention below.
• An elastic runtime takes no ingress settings of its own, and no TLS material. Sending ingressDomain and friends, or caCert, clientCert and clientKey, is refused rather than stored where nothing would read it: each machine's daemon is reached with a certificate Fadebox issues for that machine. An empty value is not refused, so a runtime read and sent back as it is saves. A read shows this runtime's own empty ingress fields rather than the front's.
• Creating it disabled ("enabled": false) is how you stage one before it consumes a licensed runtime slot.
• An elastic runtime that still has machines cannot be deleted, and the refusal names what is holding it. Machine rows are what says a bill is running, so they are given up before the runtime is — stop and delete the instances on it first.

Reading it back:

{
  "slug": "gcp-eu",
  "placement": "ELASTIC",
  "dockerHost": null,
  "elastic": {
    "provisioner": "gcp",
    "configSet": true,
    "config": {
      "project": "acme-dev-7f2c",
      "zone": "europe-west3-a",
      "network": "fadebox-vpc",
      "subnetwork": "fadebox-eu",
      "networkTags": "fadebox-machine"
    },
    "secretsSet": ["serviceAccountKey"],
    "frontRuntime": "local",
    "controlCidr": "10.20.0.0/24",
    "frontCidr": "10.20.1.0/24",
    "stoppedRetentionDays": 14
  }
}

That is an admin's read. Anybody else who may see the runtime, a project member choosing where to deploy, gets provisioner and frontRuntime only: configSet, config, secretsSet, both CIDRs and stoppedRetentionDays are null for them.

3. Add machine profiles

A machine profile is a named set of the provider's own parameters — the sizes this runtime offers. An environment names one; an environment that names none gets the default.

curl -sf -X PUT -H "$AUTH" -H "$JSON" \
  "$HOST/api/runtimes/gcp-eu/profiles/small" -d '{
  "params": {
    "machineType": "e2-standard-2",
    "diskSizeGb": "60",
    "diskType": "pd-balanced"
  },
  "default": true
}'

curl -sf -X PUT -H "$AUTH" -H "$JSON" \
  "$HOST/api/runtimes/gcp-eu/profiles/large" -d '{
  "params": {
    "machineType": "e2-standard-8",
    "image": "projects/acme-dev-7f2c/global/images/family/fadebox-machine",
    "diskSizeGb": "200",
    "diskType": "pd-ssd",
    "spot": "true"
  }
}'

small names no image, so its machines boot the Fadebox runtime image. large names one you built from the reference template, and a profile that names an image always boots that one. A stock distribution family does not do: it does not meet the contract, because it carries no Docker Engine.

• The name is the path segment, so a repeated PUT edits that profile rather than adding a second one.
• params is validated against the provisioner, by key. Every field it marks required must be present and non-blank, and a key it does not name is refused rather than ignored — otherwise a typo would surface as a machine of the wrong size, minutes into a deploy. Both are reported in one machine_profile_invalid. Values are checked only where the provider offers a fixed choice — spot and externalAddress on Compute Engine. A value outside it is refused with machine_profile_value_not_offered, naming the field, the value and what is offered. Anything else is passed through: a machine type that does not exist is the provider's own error on the first deploy that uses the profile.
• A runtime with profiles always has a default. The first profile becomes it whatever the request says. "default": true moves it and clears the previous one; omitting the field leaves the flag where it is, so re-saving a profile's parameters never moves the default by forgetting a field. The current default cannot stop being one while other profiles exist — make another one the default instead.
• The default sizes only a machine being created. An environment that names no profile gets the default of the moment its instance's machine is made; every later deploy or reset wakes that machine at the size it was made with, whatever the default has become since. So moving the default affects new instances, and the ones that exist keep deploying as before.
• Deleting a profile rewrites nothing. Machines created with it keep the name they were sized by, and an instance whose environment names no profile keeps waking its machine even after the profile it was made with is gone. An environment that still asks for the deleted profile by name is refused at its next deploy, listing the profiles the runtime does define.

GET /api/runtimes/gcp-eu/profiles lists them, ordered by name, each profile's parameters in the provisioner's own field order.

4. Test the connection

curl -sf -X POST -H "$AUTH" "$HOST/api/runtimes/gcp-eu/test-connection"

On a static runtime this pings a Docker daemon. On an elastic one there is no daemon to ping until an instance exists, so it is the provider's own verification of the account: one cheap authenticated read that proves the credential was accepted, the project is reachable with it, and the zone is real. A failure answers with the platform's own sentence.

5. Size an environment

Set machineProfile on the environment, by name:

curl -sf -X PUT -H "$AUTH" -H "$JSON" \
  "$HOST/api/projects/shop/environments/staging" -d '{"machineProfile":"large"}'

null leaves the field as it is and an empty string clears it, which means the runtime's default. The name is not checked here — an environment does not know which runtime its next instance will land on — so a name no runtime defines is refused by the deploy that needs it, listing the profiles that runtime does define.

Changing the size does not resize a machine that already exists. An instance that has been deployed once keeps its machine while it is stopped, and that machine has the size it was created with. A deploy that would wake it under a different profile the environment names is refused with machine_profile_changed (409), naming the size the machine has and the size asked for. Either put the environment back on the old size, or get rid of the machine — delete the instance, or move it to another runtime — so that the next deploy creates one of the new size. Stopping the instance does not help, because a stopped machine is kept.

An environment that names no profile is never refused this way. It takes the runtime's default only when its instance's machine is created; after that, deploys and resets wake the machine it has, at its own size, whatever the default is now. The machine log's first line on each deploy — Machine requested on runtime gcp-eu, profile small — names the size of the machine actually woken. To move such an instance to the new default, delete it and deploy again (or move it off the runtime and back).

In the web interface the same thing is the environment's Machine profile picker on Edit Environment. It offers the sizes every elastic runtime the project may deploy onto defines, says which is the default where, warns when a size no runtime offers is typed — and still takes free text, because an environment does not know where its next instance will land. It is hidden on a project with no elastic runtime to deploy onto unless the environment already names a size, which stays visible so it can be read and cleared; and hidden means not sent, so an unrelated edit can never clear a size.

Google Compute Engine

The Compute Engine provider speaks the Compute REST API directly with a service-account key. It is part of every build from 1.4, where GET /api/provisioners lists it as gcp. Besides its tests against a stubbed API, it has been run against a real project: its own test of a machine's lifecycle — create, address, labels, stop, start, destroy — and whole deploys, whose machines booted, became ready and served their instance's URLs through the front.

The account

Field	Required	Notes
serviceAccountKey	yes	The whole JSON key file, pasted in. Secret: stored encrypted and never shown again.
project	yes	The project id — not its number and not its display name, e.g. acme-dev-7f2c.
zone	yes	Where machines are created, e.g. europe-west3-a. One zone per runtime; a second zone is a second runtime.
network	no	The VPC network, by name or as a full URL. The project's default network when empty.
subnetwork	no	The subnetwork in this zone's region, by name or as a full URL. Required by custom-mode networks, left empty for auto-mode ones.
networkTags	no	Comma-separated tags put on every machine. This is what your firewall rule matches on.

A profile

Field	Required	Notes
machineType	yes	A machine type in this zone, e.g. e2-standard-4, or a full machine-type URL.
image	no	The boot image. Left out, the machine boots the installation's default, the Fadebox runtime image unless FADEBOX_MACHINES_GCP_DEFAULT_IMAGE says otherwise. When set, it is an image or image family written so that Compute Engine can resolve it — projects/acme-dev-7f2c/global/images/family/fadebox-machine, not a bare name — passed through exactly as you write it, and built under the image contract; a stock distribution family carries no Docker Engine and will never reach READY. Required on an installation that has set the default to empty.
diskSizeGb	no	The image's own size when empty — usually too small for a stack's images.
diskType	no	pd-balanced, pd-ssd, pd-standard, or a full disk-type URL. The zone's default when empty.
spot	no	true or false — false by default. A choice on the form, and any other value is refused when the profile is saved. See Spot machines.
externalAddress	no	none, egress or dial — none by default. A choice on the form; see below.

A public address, and what for

Whether a machine gets a public address, and who it is for, is the profile's to say, so one runtime can hold sizes for both topologies:

• none — no public address. Fadebox and the front reach the machine on its internal address, which means both are on the VPC or routed onto it. The default, and the deployment the design assumes.
• egress — a public address for the machine's own outbound traffic: pulling images on a network with no Cloud NAT. Fadebox and the front still dial the internal address; nothing inbound changes.
• dial — a public address that Fadebox and the front dial. This is for a control plane outside the network. The machine is then on the internet, and the two CIDRs on the runtime are what stands between its daemon and everybody: the firewall rule admits them to the public address, and the machine's own firewall admits nothing else. Use /32s, and Detect on the runtime form to get them right. The machine learns its public address from the platform at boot and puts it into the daemon's certificate, which needs curl on the image (the reference image has it).

A value outside the three is refused when the profile is saved, not on the first deploy.

One-time prerequisites in the project

Four things are yours to set up, and none of them is automated.

1. The Compute Engine API, enabled

A project that has never used it answers every call with access not configured, and no credential fixes that — it is somebody with rights over the project turning the API on. Fadebox reports it as a refusal to fix rather than as an authentication problem, which is the distinction that matters when you are staring at a rejected account.

2. A service account, and a key

Fadebox authenticates as a service account and pastes its whole JSON key into the runtime's account. Give it roles/compute.instanceAdmin.v1 on the project — the smallest predefined role that covers what the module does:

gcloud iam service-accounts create fadebox-machines
gcloud projects add-iam-policy-binding acme-dev-7f2c \
  --member="serviceAccount:fadebox-machines@acme-dev-7f2c.iam.gserviceaccount.com" \
  --role=roles/compute.instanceAdmin.v1
gcloud iam service-accounts keys create fadebox-machines.json \
  --iam-account=fadebox-machines@acme-dev-7f2c.iam.gserviceaccount.com

The permissions that role is covering, and the call each one is for:

Permission	Where it is needed
compute.zones.get	Reading the zone — which is all the runtime's Test connection does.
compute.instances.create	Creating a machine.
compute.disks.create	Its boot disk, which is created inline with the machine and deleted with it.
compute.images.useReadOnly	The boot image, on that image's project. Public images, the Fadebox runtime image among them, grant it to everyone; your own is a grant on your own project.
compute.subnetworks.use	The machine's network interface. compute.networks.use on a legacy network.
compute.instances.setMetadata	The cloud-init document, which rides along on the create, and removing it again once the machine is ready. Without it the removal fails, the machine still works, its CA key stays readable in the metadata, and the server log warns about it every minute.
compute.instances.setLabels	The three ownership labels, on every machine.
compute.instances.setTags	The network tags, when the account names any.
compute.subnetworks.useExternalIp	Only when a profile sets externalAddress.
compute.instances.get	Reading a machine's state and its address.
compute.instances.list	The reconcile pass, and every reclaim.
compute.instances.start	Starting a stopped machine.
compute.instances.stop	Stopping one.
compute.instances.delete	Destroying one.

Three things that list does not need, and one it cannot carry:

• Not roles/iam.serviceAccountUser, the usual companion for creating VMs. It exists because most callers attach a service account to the machine they create; Fadebox attaches none, on purpose, so nothing has to be acted as.
• No compute.firewalls.*. Fadebox creates no firewall rules on any platform. The rule below is a person's grant, not this account's.
• Nothing outside Compute. The token the key is exchanged for carries one scope, https://www.googleapis.com/auth/compute, so the credential cannot reach another API whatever its roles say.
• Shared VPC is extra. If the network or subnetwork live in a host project, compute.subnetworks.use — and useExternalIp, if a profile asks for one — have to be granted there as well, normally roles/compute.networkUser on the subnet. A project-level role in the service project does not reach it.

To be confirmed

The permission list above is derived from the API calls the module makes. Treat the role as a starting point and narrow it against your own project.

3. A firewall rule on the network tag

There is no Fadebox tag: the tags are whatever you type into the account's networkTags, and that is deliberate, because the rule is yours and has to match something you chose. Put one there, and open on it exactly the two ports the machines listen on:

• tcp:2376 from the same range as controlCidr — the Docker daemon, for Fadebox.
• tcp:443 from the same range as frontCidr — the ingress router, for the front host, over mutual TLS.

gcloud compute firewall-rules create fadebox-control \
  --network=fadebox-vpc --direction=INGRESS --action=allow \
  --target-tags=fadebox-machine \
  --source-ranges=10.20.0.0/24 \
  --rules=tcp:2376

gcloud compute firewall-rules create fadebox-front \
  --network=fadebox-vpc --direction=INGRESS --action=allow \
  --target-tags=fadebox-machine \
  --source-ranges=10.20.1.0/24 \
  --rules=tcp:443

Keep the two ranges the same as the two CIDRs on the runtime. They are not a duplicate of each other: this one is the cloud's, and the machine's own firewall is what Fadebox writes — see two firewalls, and both are needed.

The same rule serves a profile whose externalAddress is dial, with the ranges being the public addresses Fadebox and the front leave from — /32s, as Detect offers them — because the rule then admits those ranges to a machine that is on the internet.

4. A route out, and a route in

• Out: a machine pulls its instance's images and, unless you baked them in, the ingress router's. It gets no external address unless a profile asks for one, so the usual answer is Cloud NAT on the subnetwork; externalAddress: egress per profile is the alternative, and it puts that machine on the internet for its own traffic alone.
• In: Fadebox and the front host both dial the machine's internal address, so both have to be on that VPC or routed onto it by peering or a VPN — unless the profile says dial, in which case they dial the machine's public address and the firewall rule admits them there. Fadebox itself also needs outbound access to compute.googleapis.com and oauth2.googleapis.com.

What Fadebox does not do to your project

• No service account is attached to a machine. The instance is created with an explicitly empty serviceAccounts list, because omitting it would attach the project's default service account — and every container on the machine would then be one metadata read away from a credential for the whole project.
• No SSH keys from the project. Every machine is created with block-project-ssh-keys=true, so the keys in the project's own ssh-keys metadata, where gcloud compute ssh leaves one by default, give nobody a login on it. Instance-level keys and OS Login are left alone: whether an operator gets a way in is your decision, not Fadebox's.
• No external address, unless a profile asks for one. A machine is not on the internet because somebody forgot a checkbox.
• No firewall rules, no DNS records, no reserved addresses. The machine's own host firewall is the one that counts, so that every provider behaves alike.
• No image lookups. A machine type, an image or a disk type is passed through as you wrote it. A wrong one is the provider's own refusal on the first deploy that uses the profile.

The machine's address is the internal IPv4 address of its primary network interface, read from the instance itself. That is what Fadebox and the front both dial.

Spot machines

A profile with spot: true creates a Spot instance, and Google may reclaim one at any time. Fadebox asks for a reclaimed machine to be stopped rather than deleted, so:

• The disk survives — the instance's volumes and its warm image cache are still there.
• The stop is noticed within about a minute, and the instance goes to ERROR saying the machine was stopped. On Compute Engine, a stopped Spot machine that Google gave no reason for is reported as the machine is a Spot instance and was stopped; Google preempts these at any time.
• Nothing restarts it. Redeploy the instance when you want it back; it comes up on the same machine and the same disk.

That makes Spot reasonable for environments somebody is actively using and poor for anything that has to stay up unattended.

Stopped-machine retention

stoppedRetentionDays on the runtime destroys a machine that has been switched off for longer than that many days — and with it the instance's volumes and its image cache. 0, or leaving the field out, means never.

The instance is not touched: it stays STOPPED, and its next deploy comes up on a fresh, empty machine. The give-up is recorded as machine.expired in the audit log, with the retention that expired, how long the machine had been off and which instance's disk it was, and machine.destroyed follows once the provider confirms the machine is gone.

The instance carries the clock, so the setting is not invisible to the people it affects. A stopped machine's card counts down — machine stopped · disk kept 5d 4h more, amber inside the last day — and once retention has taken the machine the card reads disk expired 2d ago in the machine chip's place. Deploying before the moment keeps the disk; deploying after it starts on an empty machine. A deploy whose start the provider refused restarts the keep from that attempt, and the machine's stateSince (the listing's since) moves to it. See stopped machines do not keep their disks for ever.

So it is a spend control with a data consequence, and the default — never — keeps a stopped disk until somebody deletes the instance.

The card is the only warning

Fadebox sends nothing to anybody: there is no mail and no alert, so the countdown reaches whoever opens the instance and nobody else. Shortening a retention on a runtime that already has stopped machines brings every one of those dates forward with no notice of its own, and an instance nobody has looked at for a month is one whose owner finds the empty disk on their next deploy. Tell the people whose environments live there before you set or shorten one.

Keeping track of machines

Every machine is a bill, and the instance that asked for it is not always still there to explain it. Two admin-only reads answer that, and they answer different questions: one is Fadebox's own rows, the other is what the cloud account actually holds. Two writes follow from them — asking again about a machine the provider refused, and destroying one that leaked.

In the web interface all of it is the Machines view — the table, Needs attention, Retry now, the At the provider tab with its Look again and its reclaim. It is a Machines button on an elastic runtime's row under Administration → Runtimes, where a static runtime has its Ingress button, and it opens a sheet over those two tabs and a third, Profiles, for the machine sizes. Everything in it is described below as the calls it makes, which are what the CLI and a script use anyway.

The machines Fadebox has rows for

fadebox runtime machines gcp-eu
fadebox runtime machines gcp-eu --attention
fadebox runtime machines gcp-eu --state STOPPED

or, for the whole row:

curl -sf -H "$AUTH" "$HOST/api/runtimes/gcp-eu/machines?size=20"

Paged like every other listing — page, size, and sort over name, state, stateSince or createdAt, by name when you do not say. Two filters narrow it: state=STOPPED for one machine state, and attention=true for the ones worth looking at. Both narrow inside the query, so total is the size of what you asked for rather than of the page that came back.

{
  "items": [
    {
      "name": "fbx-9f3a1c2d-417",
      "providerRef": "projects/acme-dev-7f2c/zones/europe-west3-a/instances/fbx-9f3a1c2d-417",
      "profile": "small",
      "state": "READY",
      "desired": "RUNNING",
      "stateSince": "2026-09-21T09:14:02Z",
      "address": "10.20.2.14",
      "error": null,
      "createdAt": "2026-09-21T09:11:48Z",
      "instance": {
        "project": "shop",
        "environment": "staging",
        "instance": "pr-812",
        "status": "RUNNING"
      },
      "requestedBy": "ada",
      "attention": null
    },
    {
      "name": "fbx-9f3a1c2d-392",
      "providerRef": "projects/acme-dev-7f2c/zones/europe-west3-a/instances/fbx-9f3a1c2d-392",
      "profile": "large",
      "state": "STOPPED",
      "desired": "STOPPED",
      "stateSince": "2026-09-19T17:02:55Z",
      "address": "10.20.2.9",
      "error": "The machine was stopped outside fadebox: the machine is a Spot instance and was stopped; Google preempts these at any time",
      "createdAt": "2026-09-12T08:40:11Z",
      "instance": {
        "project": "shop",
        "environment": "review",
        "instance": "pr-780",
        "status": "ERROR"
      },
      "requestedBy": "ci-deployer",
      "attention": "KEPT_AFTER_FAILURE"
    }
  ],
  "page": 1,
  "pageSize": 20,
  "total": 2,
  "totalPages": 1
}

What is worth reading twice:

• state against desired. desired is what the machine is wanted to be — RUNNING, STOPPED or GONE — and requests only ever write that; the sweep moves state one step towards it. The two disagreeing is a machine in motion, not a machine in trouble.

• instance: null is the case this listing exists for. The instance was deleted or moved, or its stopped disk aged out; the machine is still there, and still costing something, until it is destroyed.

• profile is the profile the machine was created with, as a name. Editing or deleting that profile afterwards does not rewrite it, so a machine's row still says how it was built.

• error is the last thing that went wrong on the machine's own lifecycle — the same sentence the instance is showing.

• attention is why an admin should look, and it is null for a machine doing exactly what was asked of it. Three values, and attention=true selects exactly the rows that carry one:

  Value	What it means, and what to do
  FAILED	The machine is finished and the sweep will destroy it. The reason is on the row; the instance is in ERROR with the same sentence.
  REFUSED	The provider keeps refusing to stop or destroy it — or accepted the call and has not carried it out within the provision timeout, or would not list the account when Fadebox looked for a machine it never heard the handle of — and Fadebox is asking again every few minutes. It never escalates that into a delete, so this one goes on billing until the provider takes the call or you remove it in the console.
  KEPT_AFTER_FAILURE	It is switched off with its disk kept, after a restart that failed or a machine the cloud took back. The volumes are intact and a redeploy would use them; if nobody comes back for it, retention ends the disk.

• Nothing secret is on it. The machine's TLS material never leaves Fadebox, and the requester is a name rather than an account id.

The CLI prints the same rows as NAME STATE DESIRED PROFILE INSTANCE SINCE ATTENTION ERROR, with the error shortened to one line — -o json has the whole sentence. An empty INSTANCE is the row to look at. It asks for up to 500 machines and says on stderr, past that, Showing 500 of N machines; narrow with --state or --attention., so a redirected stdout is still the listing and nothing else.

Asking again about a refused machine

A machine flagged REFUSED is one the provider will not stop or take away — or one whose stop or destroy the provider accepted and then never carried out within FADEBOX_MACHINE_PROVISION_TIMEOUT — and Fadebox goes on asking every FADEBOX_MACHINE_REFUSED_RETRY_INTERVAL, five minutes by default. The same goes for a machine whose creation Fadebox never heard back from, which it lists the account for before letting the row go: a listing the provider refuses — a service account that may not list instances — flags it the same way, with the provider's sentence and no handle, and it is listed again on the same cadence. Once you have cleared whatever was in the way, this makes it ask on the very next pass instead:

curl -sf -X POST -H "$AUTH" "$HOST/api/runtimes/gcp-eu/machines/fbx-9f3a1c2d-392/retry"

It calls no provider itself and changes nothing about what the machine is wanted to be — all it moves is the backoff, so the reason stays on the machine and the flag stays on the listing until the machine is actually stopped or gone — an accepted repeat does not clear it, and while one is being waited on the reason opens Asked again and accepted, not finished yet. Before that: …. A machine that is not waiting on a refused stop or destroy — including one whose stop was refused and whose instance has since been deployed again — answers 409 rather than quietly doing nothing, and the act is recorded as machine.retry_requested in the audit log.

What the provider is actually holding

Rows are only half the answer. Every ten minutes Fadebox lists each elastic runtime's machines at the provider and compares them with its rows; what it found last is a read of its own:

curl -sf -H "$AUTH" "$HOST/api/runtimes/gcp-eu/machines/findings"

This calls no provider — it reads what the last pass recorded — so you can refresh it as often as you like and it will not tell you anything new.

{
  "checkedAt": "2026-09-21T10:30:04Z",
  "stale": false,
  "error": null,
  "truncated": 0,
  "orphans": [
    {
      "name": "fbx-9f3a1c2d-361",
      "providerRef": "projects/acme-dev-7f2c/zones/europe-west3-a/instances/fbx-9f3a1c2d-361",
      "createdAt": "2026-09-18T22:07:31Z",
      "phase": "RUNNING",
      "labels": {
        "fadebox_installation": "6b21f0c4a9d34e7f",
        "fadebox_runtime": "7",
        "fadebox_machine": "361"
      },
      "kind": "ORPHAN",
      "note": null
    }
  ],
  "foreign": [
    {
      "name": "fbx-31c0aa4e-77",
      "providerRef": "projects/acme-dev-7f2c/zones/europe-west3-a/instances/fbx-31c0aa4e-77",
      "createdAt": "2026-09-02T06:55:10Z",
      "phase": "RUNNING",
      "labels": {
        "fadebox_installation": "31c0aa4e8b6d2f10",
        "fadebox_runtime": "2",
        "fadebox_machine": "77"
      },
      "kind": "FOREIGN",
      "note": null
    }
  ],
  "missingAtProvider": []
}

• orphans are machines carrying this installation's ownership labels that no row is tracking — a leak, billing with nothing watching it. This list is the only handle on one.
• foreign are machines another Fadebox on the same cloud account made, or machines whose ownership labels are too incomplete to say whose they are. They are reported so that whoever reads the invoice can see them, and never touched — not by the pass, and not by anything Fadebox will offer later. Two installations sharing one account is a legitimate arrangement, and a cleanup that could not tell them apart would destroy somebody else's running fleet, so "cannot say whose this is" has to mean "not ours".
• note says why a machine ended up where it did, and is null for the plain case: mismatch (its labels name a row that is tracking something else, so nothing was attached — a partial match is never adopted), deletedRuntime (it names an elastic runtime that no longer exists, and this account is the only thing that can still reach it), incompleteLabels (its ownership record is too partial to place), and ageUnknown (the provider could not say how old it is, so the grace below could not hold it back — look twice before removing it).
• checkedAt: null means no pass has reached this runtime yet, which after a restart is the honest answer and is not the same as having found nothing. The findings live in memory and are rebuilt by the next pass.
• stale: true, with error, means the last attempt to reach the provider failed and the machines listed are as the last good pass saw them. A machine that was billing before the call failed is billing still, so they are kept rather than blanked.
• truncated counts what did not fit. Each list holds at most a hundred machines: an account may carry thousands that are not Fadebox's, and "there are more of these" is the number to act on.
• missingAtProvider names machines Fadebox has a handle for that the provider has not listed twice running. Deciding what that means is the machine sweep's job, not this pass's; the finding is there because the sweep walks a bounded slice of the fleet a minute and may be minutes from reaching it.

Asking for a pass now

For after you have been tidying up in the cloud console — Look again on the At the provider tab, or:

curl -sf -X POST -H "$AUTH" "$HOST/api/runtimes/gcp-eu/machines/reconcile"

It runs one runtime's pass immediately and answers the same body. A provider that will not answer is still a 200, with stale and error on the body: the findings it failed to refresh are every bit as true, and still billing. The pass is recorded in the audit log as machine.reconciled against the runtime, with you as the actor — the periodic pass writes no event, since one every ten minutes for ever would be noise.

The one thing a pass writes is the lost reply: a machine whose ownership labels name a row that never received its handle — the provider created it, and Fadebox never heard back — gets that handle attached, recorded as machine.reattached, and the ordinary lifecycle takes it from there. Nothing else is written, and nothing at all is destroyed.

Neither the pass nor these reads are gated on the licence, or on the runtime being enabled. Of everything that keeps running when a licence lapses, this is the last thing to switch off: it is what finds the machines nobody is watching but the provider's billing.

Reclaiming an orphan

An orphan can be destroyed from Fadebox — machine and disk, at the provider. It is the only call in the product that takes a cloud machine away because somebody asked, so it is fenced rather than convenient:

curl -sf -X POST -H "$AUTH" -H "$JSON" \
  "$HOST/api/runtimes/gcp-eu/machines/reclaim" -d '{
  "machines": [
    { "providerRef": "projects/acme-dev-7f2c/zones/europe-west3-a/instances/fbx-9f3a1c2d-361",
      "name": "fbx-9f3a1c2d-361" }
  ]
}'

• The machines are named one by one. There is no "reclaim everything", no filter and no "the orphans on this page" — a request that could grow to mean the fleet, because a filter matched more than it did when the page was drawn, is the request that empties an account. At most 25 per call, none of them twice; the bound is what a person can be expected to have read.
• Both fields, every time. A handle alone is not an identity on a platform that hands a destroyed machine's id to the next one created, so the name has to agree too.
• The selection is re-checked against a listing taken during the call, never against the findings it was made from — those are up to a reconcile interval old. A machine is destroyed only if that fresh listing still shows it under the same handle and the same name, still carries this installation's and this runtime's labels, still has no row tracking it, and is still older than the grace.
• A findings list that was truncated does not stop a reclaim. What the cap could not show has no bearing on the machines that were named.

The answer is one row per machine you named, in the order you named them — DESTROYED, REFUSED (Fadebox would not ask, and reason says which check failed) or FAILED (the provider was asked and said no, in its own words). One refusal never stops the rest: a selection that mixes a leak with somebody else's machine ends with the leak gone and the other one refused.

reason	What it means
NOT_LISTED	The provider is not holding anything under that handle any more. Passing it on would name whatever the platform has since given the id to.
NAME_MISMATCH	The handle is there, on a machine with a different name. Look at the findings again.
TRACKED	Fadebox has a row for it after all, so it is a host with a lifecycle — stop or delete its instance and the sweep takes it away.
ROW_DISAGREES	Its labels name a machine row that is tracking something else. One of the two records is wrong, and which one has to be settled first.
FOREIGN	Another installation stamped it. Nothing here will ever destroy one.
INCOMPLETE_LABELS	Its ownership record cannot say which machine row it belongs to, so there is nothing to hold it against.
OTHER_RUNTIME	It belongs to another elastic runtime of this installation — reclaim it from that runtime, whose account made it.
DELETED_RUNTIME	It names an elastic runtime that no longer exists, so no row is left to check it against. The cloud console, or define that runtime again and let its own pass find it.
TOO_YOUNG	Newer than the reconcile grace. A create answers before the transaction storing its handle commits, so a machine this new is most likely a deploy in the middle of committing.
AGE_UNKNOWN	The provider cannot say when it was made, so the grace could not be applied. Select it again with "confirmUnknownAge": true on that machine's entry if you know it is not a deploy in flight.

Two whole-request refusals are worth knowing. A listing Fadebox could not take refuses everything — machine_list_unavailable (409), with the provider's own sentence, and nothing is destroyed: the only alternatives were "destroy on a stale reading" and "destroy nothing". And a static runtime has no machines to reclaim at all.

A reclaim is the only record that the machine ever existed — the row that would have said so is precisely what was missing — so machine.reclaimed in the audit log carries its evidence: the runtime, the name and handle the provider knew it by, its three ownership labels and when the platform said it was made. One event per machine actually destroyed, and none for a refusal.

A foreign machine is never offered. Removing one is a trip to the cloud console and a decision somebody else should be part of.

The grace is FADEBOX_MACHINE_RECONCILE_GRACE, ten minutes by default. It exists because a machine exists at the provider for a moment before Fadebox has finished recording that it does, and without it every deploy in progress would read as a leak. It bounds orphans alone: a foreign machine is reported whatever its age, and a lost reply is attached back at once. The same rule decides what a pass hides and what a reclaim refuses, so the two never disagree.

Configuration

Machines are moved along by a periodic pass rather than by a worker parked on a deploy, which is what lets an instance keep waiting across a restart of Fadebox. Seven variables set its cadences, and the defaults are right for almost everyone: FADEBOX_MACHINE_SWEEP_INTERVAL (PT10S), FADEBOX_MACHINE_PROVISION_TIMEOUT (PT15M), FADEBOX_MACHINE_READY_CHECK_INTERVAL (PT1M), FADEBOX_MACHINE_RETENTION_CHECK_INTERVAL (PT1H), FADEBOX_MACHINE_REFUSED_RETRY_INTERVAL (PT5M), FADEBOX_MACHINE_RECONCILE_INTERVAL (PT10M) and FADEBOX_MACHINE_RECONCILE_GRACE (PT10M). What each one does, and which of them you might reasonably change, is in Elastic machines.

The one worth knowing about before your first deploy is the provision timeout: it bounds each step of a machine coming up, not the machine as a whole, so an image that takes a long time to boot is the reason to raise it.

When a machine does not come up

In order of what actually tells you something:

1. The instance's machine log — the machine entry in its status panel's log pane, fadebox instance up --wait's machine: lines, or GET …/instances/{slug}/machine/log. Its last line says where the boot stopped: a log that ends on Waiting for the machine to start never got a running machine from the provider; one that ends on waiting for its Docker daemon booted and never answered, which is the image contract or the firewall below; one that ends on starting the ingress router or not up yet; trying again is the router. It names no address, so for that read on.

2. The instance's own error. A failed machine puts the provider's sentence on the instance, and that is where to start: a quota, an exhausted zone, a rejected credential, or one of Fadebox's own step-specific messages — the machine reported no address, the Docker daemon on 10.0.0.7 never answered over mutual TLS, the ingress router could not be started on 10.0.0.7: …. The second is almost always the image contract or the firewall: a machine that boots but whose daemon never answers is a machine whose cloud-init could not do its job, or one your firewall rule does not cover on 2376.

3. The machines listing, which says the same thing per machine and says it for machines no instance is pointing at any more. attention=true is the short version of "what should I look at".

4. The audit log, on the administration side. A machine's whole life is there: machine.created when the provider accepted it, machine.started and machine.stopped bracketing the hours it was charged for, machine.failed with the reason, machine.expired when retention gave it up, and machine.destroyed — the last thing ever said about a machine, and the only record that its bill has ended for good. The actor is Fadebox's own machine sweep, with whoever asked for the machine in the event's detail.

5. The findings, for the machines Fadebox has no row for — and reclaim for the ones they say are yours. What is left after that is the provider's console: every machine Fadebox creates is labelled fadebox_installation, fadebox_runtime and fadebox_machine, which is what to filter on there.

Five failures are worth recognising by shape:

• The machine's reason says the ingress router could not be started on <address>. The machine booted and its Docker daemon answered; what did not happen is the small Traefik and socket proxy Fadebox starts on it before calling it ready. What follows the colon is what stopped it, and on a cold machine it is nearly always a pull: the image is not baked into the machine image and the machine cannot reach the registry it would come from — see Bake the ingress router's images in. It is retried on every sweep, so a transient failure clears itself; a permanent one ends at the provision timeout, and the deploy then fails naming the router and what it could not do. A frontCidr that is not a range is refused at once rather than retried, because waiting could not fix it.
• A machine shows attention: "REFUSED". The provider will not accept a stop or a destroy, or accepted one and never carried it out, and Fadebox is asking again every five minutes for as long as it takes; it never turns a refused stop into a delete. Read the machine's error. A refusal is the provider's own words — fix it there: a deletion lock, a policy, an exhausted quota on the operation. An accepted call that never finished reads The provider accepted the destroy 15m1s ago and still reports the machine as RUNNING; the destroy will be asked for again — look at the operation in the provider's console, or whether somebody started the machine again. When the repeated call is accepted the flag clears while Fadebox waits for it, and comes back if that one never finishes either. Until the call takes, that machine is billing.
• A machine shows attention: "KEPT_AFTER_FAILURE". It is off with its disk intact, after a restart the provider refused or a machine the cloud took back — a preempted Spot instance is the common one. Nothing restarts it automatically. Redeploying the instance brings the same machine and the same disk back; if the environment is finished with, delete the instance, or leave it and retention will end the disk.
• machine_not_released (409) on a deploy means a previous machine of this instance is still being destroyed — it was moved off the runtime, or it failed and was given up, including one the provider made while Fadebox never heard back and found again by its labels. The machine name is deterministic per instance, so the next one cannot be created until the old one is gone. It clears itself; deploy again in a minute.
• machine_not_ready (409) on logs or a log export means the instance is on an elastic runtime and its machine is not reachable — it is still provisioning, or it is switched off. The status reads normally; there is simply no daemon to ask. The machine log answers regardless.

What is not supported on an elastic runtime

Everything below is deliberate in this first version, not a gap waiting to close. The first four are refused with 409 and one sentence — "Cannot <what> on runtime '<slug>': an elastic runtime has no daemon of its own — it provisions a machine per instance" — because they all act on a runtime's own daemon, and there is none: its hosts are the machines under it.

What	Where you meet it
Template test runs	Asking for one on an elastic runtime, refused before it claims a run slot so the template never sticks in STARTING.
Image reclaim and image retention	Reading or reclaiming a runtime's images. The retention timer never reaches such a runtime either, since it rides on the disk pass, which does not visit machines.
The resource scan, its reclaim and its purge	Reading a runtime's managed resources, reclaiming the orphans among them, or purging them. A machine is destroyed whole, so nothing accumulates on it to tidy.
An ingress stack of its own	Installing, reading or removing one. The front serves this runtime's instances, and each machine runs its own ingress router — which Fadebox starts, and which is not the same thing.

Four more are limits rather than refusals:

• No automatic restart of a machine that went away. A preempted spot machine, or one somebody stopped in the provider's console, puts its instance in ERROR with the disk kept and stays off. A machine that keeps being taken back is something to see, not something to retry: redeploy the instance when you want it back.
• No per-instance machine size. The profile is the environment's, and there is no override on an instance.
• Host-published ports answer inside the machine network only. A service that publishes a port without going through ingress is on the machine's own address, reachable from the front's side of that network and from nowhere else. Everything a visitor reaches goes through the front.
• No usage figures for the runtime itself — no CPU, memory or disk bars, and no fadebox_runtime_* Prometheus series. Every one of those describes a daemon, and an elastic runtime has none; summing a fleet that grows and shrinks with the instances on it would be a different number wearing the same name. What it does report at its own level is how many machines answered the last reading and how many were ready and would not. Per-container figures are read off each machine, and are the same series and the same bars as anywhere else. Disk is the exception: the disk pass does not visit machines, so images and volumes on one show a dash.

And one thing to know about rather than plan around: Fadebox starts a machine's ingress router while the machine is coming up, and does not watch it afterwards. Both containers restart themselves after a crash or a reboot, so nothing ordinary takes them away — but a router removed by hand on a machine that is already ready is not noticed, and not put back, until that instance is next stopped and started.

Providers beyond Google Compute Engine — VMware vSphere and Proxmox VE — are planned, and each begins with a feasibility spike before it is scheduled. Neither exists.