Skip to content

Maintain ​

This page is for the box in day-to-day use. It covers what runs, where the state lives, updates, backups, disk space, logs, and what each error message means.

What runs ​

The stack is three containers.

ServiceRolePublished
orchestratorThe server and the operator app127.0.0.1:8080 (ORCHESTRATOR_PORT)
postgresThe databaseNo
opensandboxStarts and stops machines' containers through the Docker socketNo

Only the orchestrator is reachable from outside Docker, and only on this host. Agents run on their organisation's machines: one sandbox container per machine, shared by every agent placed on it (Machines). Sandboxes can reach the public internet.

The agent image ​

Every machine runs agent-runtime:0.4 (SANDBOX_IMAGE). It has a browser and the coding toolchains. ./setup.sh builds it from runtime/ with two build arguments at their defaults:

ArgumentDefaultOther value
BROWSERchrome: Google Chrome on amd64, Debian's Chromium on arm64chromium: Debian's Chromium on every platform
HARNESSESclaude-code cursor antigravityAny subset of those three. Codex is always installed. Empty leaves Codex as the one harness

To build it another way, run docker build yourself:

text
docker build --build-arg BROWSER=chromium --build-arg HARNESSES= --tag agent-runtime:0.4 runtime/

./setup.sh rebuilds agent-runtime:0.4 with the defaults on every run, the update included. After it, build your own image again, or give it another tag and name that tag in SANDBOX_IMAGE.

A coding run of an agent on a harness the image lacks fails at start with an error that names the missing harness. Harnesses explains which harnesses an image carries.

Where the state lives ​

STATE_ROOT in .env is an absolute path on this host. The stack keeps everything under it. Each directory is mounted into the container that uses it. The stack creates no named Docker volume.

TreeHolds
postgres/The PostgreSQL data directory: every table
opensandbox/The sandbox server's SQLite lifecycle store
workspaces/One directory per machine, mounted at /home in the machine's container, with one home per agent on it: workspaces/<machine id>/<agent id>, /home/<agent id> inside. Scripts, the browser profile, and a coding agent's repositories live there
notes/The notes vault
attachments/Chat images, one file per attachment
connectors/Connector packages installed from a registry

The path must be absolute because the sandbox server mounts each machine's directory by that same path.

Everything the box knows is this tree plus .env.

Update ​

Read the changelog. Each version names its effect (Versions).

EffectDo
no impactNothing
rebuildThe commands below
breakingBack up, then follow the steps in that version's entry

To rebuild:

text
git pull
./setup.sh
docker compose up --detach --wait --build

./setup.sh keeps every value already in .env. It rebuilds the agent image, so a change under runtime/ reaches the next machine container. --build rebuilds the orchestrator image from the checkout.

The server applies new database migrations at boot, before it listens (Orchestrator).

Back up ​

PostgreSQL holds notes, records, memory, scripts, runs, and every credential. Credentials are encrypted with the key in .env. workspaces/ holds what agents wrote to disk. attachments/ holds uploaded images.

Stop the stack, then copy STATE_ROOT:

text
docker compose stop

Or dump the database while the stack runs:

text
docker compose exec -T postgres pg_dump -U agent_platform -Fc agent_platform > guilds.dump

Keep .env with the backup. Without it, the credentials in the backup cannot be decrypted.

Restore ​

Stop the stack. Put the copy of STATE_ROOT back in place, with the .env it was made under. Then start the stack:

text
docker compose up --detach --wait

To load a dump instead, stop the orchestrator, restore into the running database, then start the orchestrator again:

text
docker compose stop orchestrator
docker compose exec -T postgres pg_restore -U agent_platform -d agent_platform --clean --if-exists < guilds.dump
docker compose start orchestrator

Run setup.sh again ​

./setup.sh is safe to run again. It keeps every value .env already has, and it rebuilds the agent image.

It refuses to generate a new secret when STATE_ROOT/postgres already holds a database. That database was created with values only its owner has. Put those values into .env, or point STATE_ROOT at an empty directory.

./setup.sh --env-only writes .env and the state directories. It builds no image.

The stack does not start without .env. Each missing value says run ./setup.sh first, which writes it into .env.

Free disk space ​

A deleted machine or agent can leave its directory under workspaces/. The sweep lists the directories no live machine or agent owns, and that nothing changed in for a number of days:

text
docker compose exec orchestrator pnpm sweep:workspaces --age-days 7

It prints each path it would remove. Add --apply to remove them.

An agent's home is also bounded by its max_workspace_mib, and a machine's directory by the machine's cap (Agent home).

Start over ​

To go back to an empty box, stop the stack, remove the state directories, and start the stack again:

sh
docker compose down
rm -rf .state/{postgres,opensandbox,workspaces,notes,attachments,connectors}
docker compose up --detach --wait

.state is the default STATE_ROOT. Use the path .env names when it differs.

The server applies its migrations to the empty database and creates the first operator and the Home organisation again. .env and the images stay. On a development checkout, make reset does the same (Development).

Logs and metrics ​

The orchestrator writes one JSON line per event to stdout. Docker keeps them as json-file logs, five files of 10 MB each. Follow them with:

text
docker compose logs --follow orchestrator

Each line carries timestamp, level, run_id, guild_id, agent_id, operation, error_type, message, and traceback. It also carries source, request_id, session_id, path, method, and status when they apply. Warnings and errors are also written to stderr as one readable line.

Secrets never reach the log. Every value of an environment variable whose name ends in API_KEY, SECRET, TOKEN, PASSWORD, or PASSWD is replaced by [REDACTED:<name>] before a line is written.

Every API call from the operator app carries that tab's X-Request-Id. It appears in the log as request_id. The app also reports its own uncaught errors and failed calls to the server, as source: frontend. One id lines up what the operator saw with what the server did.

At boot the server logs listening once it accepts requests, and sentry enabled or sentry disabled.

The event loop counts as stalled when it is held 1 second or more. A stall logs event loop stalled for <ms> ms at warn, with operation: runtime.stall. A pooled database connection lost while idle logs idle database connection lost: <reason> at warn, with operation: db.pool. The pool drops it and opens another on the next query.

With SENTRY_DSN set, Sentry receives, under SENTRY_ENVIRONMENT (Configure):

  • errors, as issues
  • traces, sampled at SENTRY_TRACES_SAMPLE_RATE
  • CPU profiles while a sampled trace is open, for a process sampled at SENTRY_PROFILE_SESSION_SAMPLE_RATE
  • every info, warn, and error log line, as a Sentry log whose attributes are the line's fields, already redacted
  • an EventLoopBlocked event with the main thread's stack for each stall, at most 10 an hour
  • runtime metrics every 10 seconds, under Explore, Metrics:
MetricUnitMeasures
node.event_loop.delay_max, node.event_loop.delay_p99msEvent loop delay
node.cpupercent of one coreServer CPU. Two busy cores read 200
node.memory.heap_used, node.memory.rssbytesServer memory
host.load_1mHost load average
host.memory.usedpercentHost memory
host.disk.free, host.disk.usedbytes, percentThe filesystem under / (mount: root) and under WORKSPACE_ROOT (mount: workspace)
db.pool.total, db.pool.idle, db.pool.waitingThe connection pool
db.server.connections, db.server.max_connectionsPostgreSQL client connections
node.event_loop.stallcountEach 10 seconds with a stall
db.pool.errorcountEach idle connection lost

Sentry never receives user information, cookies, request bodies, model inputs and outputs, or query data.

GET /health answers {"status":"ok"} after a database ping. A monitor outside the box can poll it.

When it fails ​

MessageCause
run ./setup.sh first, which writes it into .envdocker compose found no .env, or one without STATE_ROOT, SECRETS_MASTER_KEY, POSTGRES_PASSWORD, or OPENSANDBOX_API_KEY
STATE_ROOT must be an absolute path./setup.sh found a relative path in .env. The sandbox server mounts machine directories by the absolute one
… already holds a database, and .env has no …./setup.sh would generate a secret over a database created with another one. Put that database's values into .env, or point STATE_ROOT at an empty directory
SECRETS_MASTER_KEY is the all-zero development key: set 32 random bytesThe server refuses the development key outside the development stack (Security register, SEC-14)
SECRETS_MASTER_KEY must be 32 bytes (base64 or hex)The key is not 32 bytes
AUTH_MODE must be compat-cookie, trusted-header.env names a mode core does not have
TRUSTED_PROXY_SECRET is required when AUTH_MODE=trusted-header, and the same for PUBLIC_BASE_URL and OPERATOR_EMAILThe proxy mode is missing one of its three settings (Remote access)
OPERATOR_EMAIL is another user's emailA user other than the box's first already holds that address
Create agent, Studio, the Builder, or recipe apply shows Connect a model provider firstThe organisation has no provider connected. Connect one under Settings, Model keys (Model keys)
A run ends with connect a … key on Model keysThe organisation has no key for that agent's model provider, or removed it. Connect one under Settings, Model keys (Model keys)
Model keys answers <Provider> rejected the key or did not accept the keyThe provider refused GET /models with that key. Nothing was saved (Model keys)
The picker lists a model under Not offeredThe agent's stored model is not among the organisation's chosen models. It still runs while the provider's key is connected (Model keys)
Model keys cannot browse a provider's catalog, or lists stale modelsThe box could not fetch models.dev. It serves its last catalog or the snapshot it ships with (Configure)
A run fails at start with workspace.fullThe agent's home is over its max_workspace_mib (or WORKSPACE_MAX_MIB), or its machine's directory is over the machine's cap. The message names which. Free space from the CLI tab, or raise the cap in the agent's Settings or under Settings, Machines
machine_id is required: the organisation has more than one machineAn agent create named no machine while the organisation has several
machine … is fullThe machine already holds 32 agents, or the policy's agents-per-machine cap
Stored credentials fail to decrypt after a restoreThe restore runs under a different SECRETS_MASTER_KEY than the one the data was written with

The orchestrator container restarts on its own (restart: unless-stopped). A setting it refuses shows in its log on every attempt.

Free software under the GNU Affero General Public License, version 3 only.