Local Golem Development Server (golem server)

The golem server command runs a self-contained Golem server on the local machine for development and testing. It bundles all Golem services (worker executor, component compilation, shard manager, registry, and router) into a single process.

Note: Only the golem binary supports this command. golem-cli does not include golem server.

Subcommands

golem server run

Starts the local Golem server.

golem server run

Parameters

Parameter	Description	Default
--router-addr <ADDR>	Address to serve the main API on	0.0.0.0
--router-port <PORT>	Port to serve the main API on	9881
--custom-request-port <PORT>	Port to serve custom HTTP requests on (HTTP API endpoints)	9006
--mcp-port <PORT>	Port to serve the MCP server on	9007
--ports-file <PATH>	Write discovered startup ports to this JSON file	(none)
--data-dir <PATH>	Directory to store data in	Platform-specific (see below)
--system-memory-override <SIZE>	Override detected system memory, e.g. 2GiB; not a hard RSS limit	Environment variable, manifest, then host/cgroup detection
--clean	Clean the data directory before starting	false
--agent-filesystem-root <PATH>	Use deterministic agent filesystem directories rooted at the given path instead of random temp directories. Layout: <root>/<environment_id>/<component_id>/<agent_name>/	(none)

Examples

Start with defaults:

golem server run

Start on a custom port:

golem server run --router-port 8080

Start fresh, deleting all previous state:

golem server run --clean

Start with a custom data directory and deterministic agent filesystems:

golem server run --data-dir ./my-data --agent-filesystem-root ./agent-fs

Write port information to a file (useful for scripting and CI):

golem server run --ports-file ./ports.json

Default Data Directory

The default data directory is platform-specific:

Platform	Default Path
macOS	~/Library/Application Support/golem
Linux	~/.local/share/golem
Windows	C:\Users\<USER>\AppData\Local\golem

Ports File Format

When --ports-file is specified, the server writes a JSON file with the actual ports it bound to. This is useful when using port 0 (OS-assigned) or for scripting and CI automation. The file is written atomically (via a .tmp rename) once all services are ready.

{
  "routerPort": 9881,
  "customRequestPort": 9006,
  "mcpPort": 9007
}

The application manifest can configure the built-in local preset in its localServer section. routerAddr and routerPort define the main API endpoint used both by golem server run and by other golem commands that target the built-in local preset. customRequestPort and mcpPort control how deployment subdomain values expand: HTTP API domains resolve to <label>.localhost:<customRequestPort> and MCP domains resolve to <label>.localhost:<mcpPort>. Use nonzero ports for deployments that register persistent subdomains. Load golem-configure-api-domain or golem-configure-mcp-server for the full subdomain versus domain rules.

Memory Budget

Override detected system memory to leave room for other applications and control how much memory a local server uses for agent admission and eviction. This is useful for everyday development, smaller machines, and running multiple servers. Without an override, Golem detects host or cgroup memory; on macOS, a local server sees the full host RAM.

golem server run --system-memory-override 2GiB
GOLEM_LOCAL_SERVER_SYSTEM_MEMORY_OVERRIDE=2GiB golem server run

All three inputs use the same positive byte-size format. Units are case-insensitive: 1mb and 1MB mean 1,000,000 bytes; 1MiB means 1,048,576 bytes. Whole-number quantities and ASCII spaces are accepted; use 1536 MiB instead of 1.5 GiB. A value without a unit means bytes. Quote unitless values in YAML. Precedence is the --system-memory-override flag, then GOLEM_LOCAL_SERVER_SYSTEM_MEMORY_OVERRIDE, then localServer.systemMemoryOverride in the manifest, then automatic detection. These map to the executor's memory.system_memory_override. The executor uses 80% for agent admission and eviction, leaving 20% for host overhead. This is not an OS-enforced process RSS limit; compilation and other host allocations can exceed it. Check the startup log's measured memory limit and monitor RSS under load.

localServer:
  systemMemoryOverride: 2GiB

Manual Testing with Free Ports

For manual testing in temporary apps, pass 0 directly as golem server run flags to let the OS assign free ports:

golem server run \
  --router-port 0 \
  --custom-request-port 0 \
  --mcp-port 0 \
  --ports-file .golem/ports.json \
  --data-dir .golem/data

The 0 flag values request OS-assigned free ports. The --ports-file path records the actual bound ports and is also the readiness signal; wait for .golem/ports.json before sending requests.

Do not put 0 in manifest localServer port fields. Manifest localServer.customRequestPort and localServer.mcpPort values are used for deployment subdomain expansion and must be stable nonzero ports.

For persistent local deployment subdomains, use stable manifest ports or omit the port fields to use the defaults:

localServer:
  routerAddr: 0.0.0.0
  routerPort: 9881
  customRequestPort: 9006
  mcpPort: 9007
  portsFile: .golem/ports.json
  dataDir: .golem/data

routerAddr and routerPort define where the built-in local preset is reachable: golem server run binds the server there, and other golem commands connect there. routerAddr must be an IPv4 address literal; host names are not supported. When the server binds to 0.0.0.0, commands connect through 127.0.0.1. Setting only portsFile records the selected ports but does not request free ports. Explicit CLI flags such as --router-port, --custom-request-port, --mcp-port, --ports-file, and --data-dir override the manifest values for server run.

Warning: --agent-filesystem-root

Do not use --agent-filesystem-root unless you have a specific reason. This option replaces the default random temporary directories with deterministic paths. Manually modifying files under this root while agents are running can break durable execution guarantees — Golem relies on controlling the agent filesystem to ensure consistency across restarts and replays. This flag is intended for advanced debugging and inspection scenarios only.

golem server clean

Deletes the local server's data directory without starting the server.

golem server clean

This removes all stored state including deployed components, agent data, and operation logs.

When a manifest is discovered, this removes localServer.dataDir. Without a manifest it removes the platform default data directory. Pass -X to ignore a discovered manifest and clean the platform default. The CLI displays the resolved absolute path and asks for confirmation before deleting it; filesystem roots are always rejected.

Important Notes

• Cleanup deletes all state: Both golem server clean and golem server run --clean delete all existing agents, deployed components, and data from the resolved directory. Never run either command without explicitly asking the user for confirmation first. In a non-interactive workflow, pass --yes only after the user approves the exact target path.
• The server runs in the foreground: It blocks the terminal. Run it in a separate terminal or background process before deploying or invoking agents.
• Deploy after starting: Components must be deployed with golem deploy after the server is running before agents can be invoked.
• File limits: On startup the server automatically attempts to increase the OS file descriptor limit to 1,000,000 for better performance.

Typical Development Workflow

1. Start the server: golem server run
2. In another terminal, deploy: golem deploy --yes
3. Invoke agents or use the REPL: golem repl
4. After code changes, redeploy: golem deploy --yes --reset

Debugging a Running Server

The server runs in the foreground and writes structured logs to stderr. Verbosity is controlled by the -v flag — note that golem server defaults to a higher base level than the rest of the CLI, so the mapping is different from other golem subcommands:

Flag	golem server run level	other golem ... commands
(no flag, default)	INFO	ERROR
-v	WARN	WARN
-vv	INFO	INFO
-vvv	DEBUG	DEBUG
-vvvv	TRACE	TRACE

Use -vvv to see the durability and retry decision logs that are usually needed for diagnosing runtime behavior:

golem -vvv server run

RUST_LOG is ignored. The server builds its own tracing filter from the -v flag and does not consult the RUST_LOG environment variable. Use -v levels instead.

Useful Tracing Targets

When diagnosing a specific subsystem you can grep the server's stderr by tracing target. The most useful prefixes are:

Prefix	What it covers
golem_worker_executor::durable_host::http::inline_retry	HTTP status-code retry decisions and eligibility
golem_worker_executor::durable_host::http	All outgoing HTTP host calls and durability events
golem_worker_executor::durable_host::durability	Durable host function replay and retry resolution
golem_worker_executor::durable_host (semantic trap retry)	"Semantic trap retry: …" decision lines
golem_worker_executor::services::events	Internal worker events (invocations, suspends, ...)

Key Log Lines When Diagnosing Common Issues

These are the first debug lines to grep for when a feature "doesn't seem to work":

• Status-code retry not firing for an outgoing HTTP request:

  HTTP status retry skipped              reason=<NotIdempotent|BodyNotFinished|NoRetry|...>  uri=...  status=...
  HTTP status retry skipped: inside atomic region
  

  The reason field is the source of truth — it tells you exactly why a particular request was not retried (most commonly NotIdempotent for opt-out-of-idempotence cases, or NoRetry when no status-code-keyed policy matched). Look for it before assuming the policy or the feature is broken.

• Semantic trap retry policy decisions:

  Semantic trap retry: delaying          retry_policy=...  delay_ms=...  attempt=...  trap=...
  Semantic trap retry: exhausted         retry_policy=...  attempt=...  trap=...
  

  Indicates which user-defined retry policy matched the trap and how it decided. Absence of these lines for a 5xx-throwing handler means no named trap policy matched (the legacy retry config is used instead).

• HTTP retry policy resolution failed (genuine error):

  WARN  Failed resolving semantic trap retry policy, falling back to legacy retry config
  

  This now only fires for genuine evaluation errors (e.g. type-coercion failures inside a predicate). Policies whose predicate references a property that does not exist in the current context (e.g. a status-code-keyed policy in the trap context) are silently skipped instead.

Running the Server in the Background

To keep the server running while inspecting logs from another terminal:

golem -vvv server run > /tmp/golem-server.log 2>&1 &
tail -f /tmp/golem-server.log | grep -E "HTTP status retry|Semantic trap retry"