How can I orchestrate my agents?

, ,

Motivation

The AI space moves at the speed of tokens, which means we often have to try things and throw them away when they do not fit. The latest addition to this pool is Agent Substrate. At a high level, it is an agent runtime purpose-built for agents, with a focus on sandboxing and quick suspension and resumption. The bet is that agents spend most of their time waiting on tool calls, humans or models, so keeping one warm container per agent is wasteful if we can quickly snapshot and restore them.

Substrate keeps a small pool of warm worker pods and a much larger set of actors that are scheduled onto them. When an actor is suspended, its state is snapshotted to an object store and the worker returns to the available pool. The next request, assigned through the Substrate router, resumes on whichever worker is available. It is still fundamentally Kubernetes under the hood, but Kubernetes mostly handles the pods, nodes and autoscaling.

Once the workload orchestrator is set up, we need a way to define the workload. This is where AX comes in. It gives us a task definition that feels similar to a Kubernetes manifest: apply a YAML file and get a sandbox on the other side. Task state lives in a separate Redis instance rather than a custom resource definition (CRD), so short-lived tasks do not need to go through etcd.

The interesting economic point is not simply that a suspended agent is cheap. It is that the number of agents we have stops being a compute cost. Only the agents doing something occupy a worker; the rest take up space in an S3 bucket. The compute bill therefore scales with active agents, while the token bill scales with useful work, or can be reduced by routing appropriate work to cheaper models. Neither scales with the number of agents that merely exist.

Architecture

At a high level, there are three layers:

  • Kubernetes manages the infrastructure. It runs the control-plane services and worker pods, schedules those pods onto nodes and provides the usual infrastructure lifecycle.
  • Substrate manages actors. An actor is the workload; a worker is the pod that hosts it while it is active. Substrate assigns actors to available workers, handles their suspend-and-resume lifecycle and routes traffic to them.
  • AX manages tasks. It provides the task manifest, workspace configuration and commands. Its controller talks to Substrate to create and manage the underlying actors.

The control flow looks roughly like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
Task manifest
     |
     v
AX API → Redis → AX controller
                       |
                       v
                Substrate API
                       |
                       v
               Actor on a worker
                       |
                       v
                Your agent code

Kubernetes supplies the infrastructure, Substrate supplies the actor runtime and AX supplies the task lifecycle. The application, however, decides what work to do and what constitutes progress.

Trying it out

Installing Substrate

Substrate runs on Kubernetes, but we do not need a real cluster to try it. We can use kind, through the scripts shipped in the repository, or another local Kubernetes setup such as Minikube, OrbStack or Docker.

We can tell an agent to run the installer, but at a high level we need to:

  • Create the cluster:

    1
    hack/create-kind-cluster.sh

    This gives us a kind node and a local image registry at localhost:5001. Be aware that the script first deletes any existing kind cluster named kind.

  • Deploy Substrate:

    1
    hack/install-ate-kind.sh --deploy-ate-system

    Substrate installs WorkerPool, SandboxConfig and CSIDriverConfig as CRDs. It then sets up certificates for mutual TLS, several Go binaries in a local registry, PostgreSQL, RustFS for the S3-compatible bucket and an observability stack that includes Prometheus and Jaeger.

  • Install the CLI plugin:

    1
    go install ./cmd/kubectl-ate

    Actors, templates and ATE spaces are not Kubernetes objects. Instead, they live in Substrate’s own API, backed by PostgreSQL. As a result, kubectl get cannot see them; kubectl ate is the client for that API.

Installing AX

AX has its own small control plane that talks to Substrate’s API. It needs ko on our PATH; Substrate’s pinned version is available with:

1
hack/run-tool.sh --print-bin-path ko

It also needs a registry from which the cluster can pull, such as the kind registry created earlier. The rough steps are:

  • Build and push the task runner image:

    1
    make build-task-runner TASK_RUNNER_REPO=localhost:5001/ax-task-runner

    Push that image somewhere accessible to the cluster. The official example points to gcr.io/ax-substrate/..., which is not publicly pullable, so for now we need to build our own. See google/ax#364.

  • Configure snapshot storage in Substrate. Actor templates specify the storage location through snapshotsConfig.storageLocation, while the installer can configure the backing bucket through BUCKET_NAME.

  • Deploy AX:

    1
    make deploy AX_IMAGE_REPO=localhost:5001

    This deploys Redis, ax-server and ax-controller in ax-system. Then install the CLI, with the same PATH caveat as before:

    1
    go install ./cmd/ax

  • Create a WorkerPool.

We should now have a fully set-up local instance ready for tasks. A friendly-neighborhood LLM can help fix any issues encountered during installation.

The counter demo

Before involving AX, let’s see what Substrate can do directly. We will use the built-in counter demo, which runs a small Go HTTP server. Each request increments two counters: one in process memory and another in a file. Deploy it with:

1
hack/install-ate-kind.sh --deploy-demo-counter

We get a throwaway “golden” actor, which is essentially the prime instance. The scripts snapshot this golden state, and each subsequent actor starts from that snapshot instead of performing a cold boot. In Ethereum network testing, this could be a golden state containing a finalized, healthy network as the starting point.

Create an actor from the template and port-forward the router:

1
2
3
4
kubectl ate create actor my-counter-1 \
  -a ate-demo-counter \
  --template counter
kubectl port-forward -n ate-system svc/atenet-router 8000:80

Then send a request with the actor’s identity in the header:

1
2
3
curl -X POST \
  -H "ate-target-actor: ate-demo-counter/my-counter-1" \
  http://localhost:8000/

The response reports both the in-memory and file counters:

1
hello from: 169.254.17.2 | preserved memory count: 1 | preserved file counter: 1

That alone does not tell us much about snapshots, so let’s suspend the actor:

1
kubectl ate suspend actor my-counter-1 -a ate-demo-counter

We can check its state with:

1
kubectl ate get actor my-counter-1 -a ate-demo-counter

Once it is suspended, retry the earlier curl request. The response should look like this:

1
hello from: <actor-ip> | preserved memory count: 2 | preserved file counter: 2

The interesting part is that both the memory counter and the file counter increased by one. This shows the process being restored from a snapshot, while the particular worker node onto which it is scheduled is immaterial.

A task on AX

The counter demo shows that state survives suspension. Now let’s give that state a purpose: a model does some work, writes a proposal and waits for a human to review it. I can imagine disaggregated work queues operating this way: an agent performs a task, a user reviews and updates the task description, and the agent picks the task back up and continues to completion.

I used the existing Chalk workspace and a small Python script inside the AX task. The script calls DeepSeek V4.1 Flash through OpenRouter. It has two phases: propose a documentation change, then draft it using the saved proposal and review feedback.

The first request was simple:

Read this README and suggest one small documentation improvement. Don’t implement it. Save a plan with the target section, rationale, and next action.

At this point, the model had finished its work and needed review, so I suspended the task:

1
ax suspend task blog-marker-task && ax describe task blog-marker-task

I waited for Phase: Suspended. Abstractly, this means that the agent is blocked on user review, so I can provide feedback on the plan with a command such as:

1
ax ssh blog-marker-task -- sh -c '...'

I can imagine a work queue around this pattern: an agent finishes a stage, a human reviews and updates the instructions, and the agent returns to complete the next stage. AX supplies the suspend-and-resume lifecycle; the queue and approval rules sit around it. I performed most of these steps manually while learning.

Rough edges

This is definitely still early-stage software, and a lot of polish is missing when trying to run it ourselves. Full-scope snapshots restore an actor’s RAM and disk state, which is why both counters in the demo survive suspension. Transient or external state is a different matter: in-flight requests and open network connections will not resume at their exact point of interruption.

We also cannot select an arbitrary snapshot as the starting point for any task. I wanted to test whether task forking was possible, but the architecture currently seems better suited to resuming existing tasks than to forking them.

I have not measured density, latency or cost savings, so this is not a performance verdict.

Conclusion

The interesting possibility is separating an agent’s lifetime from the lifetime of the compute it occupies. In a RAM-constrained world where many people run agent swarms with unclear lifecycles, this could be a useful way to save compute.

I can imagine this kind of work queue emerging at companies: agents investigate, humans review, instructions change, and agents return for the next stage. The workspace becomes the handoff between those stages. It is also possible that, in a future software factory, another agent replaces the human reviewer, but the infrastructure gains would still be valid.