Observability

A model records what it does as it does it. Every command, every event it commits, and every policy, view, projection and saga that reacts to it gets a span, named after the building block. The spans connect into traces that follow a flow from a click to everything it caused.

Part 7 of the tutorial sets this up step by step for the bank.

Export the traces

A hosted model records into whatever tracer the runtime provides. Send it to an OpenTelemetry backend, such as Axiom, Honeycomb or your own collector, to see each flow as one connected trace. The bank declares its Axiom datasets and an ingest token next to its Worker:

return Axiom.Telemetry({ token: ingest, traces, logs, serviceName: "bank" });

and provides that Layer to the Worker, with the rest of what it hosts. Alchemy.Telemetry.layerOtlp does the same for any OTLP endpoint.

Cloudflare.Telemetry() shows a Worker’s spans in Cloudflare’s dashboard instead, with no backend to run. Cloudflare starts a new trace for every Worker and Durable Object event, so a flow that crosses objects shows up as separate traces, joined only by domain.correlation_id, and spans from a browser can’t be added.

Follow a command

A withdrawal sent from the app shows up as one trace, crossing the Worker and two Durable Objects:

Domain.send Account.WithdrawMoney           the Worker, where the Api received it
└─ Domain.decide Account.WithdrawMoney      the account's Durable Object: decide and commit
   │   event MoneyWithdrawn                 one span event per committed event
   ├─ Domain.view AccountSummary
   ├─ Domain.view DailyOutflow
   └─ Domain.policy FraudReview
      ├─ Domain.external FraudCheck.score
      └─ Domain.send Account.FreezeAccount
         └─ Domain.decide Account.FreezeAccount

Each event carries the span that committed it, so the policies and views that react to it continue the same trace, however long the delivery took.

A query is a Domain.query <Name> span in the Worker, with domain.outcome and the rows it returned (domain.query.rows). Waiting for the caller’s own write adds a Domain.caughtUp <Projection> span. A view’s or stream’s validate is a Domain.validate <Name> span with domain.mode. Nothing from an input is recorded unless its handler annotates it.

Trace the browser

foldkit traces every Command it runs, and every Api call carries the caller’s trace on, so the browser can start the trace: a click, the commands it sends, and everything they cause on the server. Give the app a tracer that exports to its own origin:

const Tracing = OtlpTracer.layer({
  url: `${location.origin}/api/telemetry/v1/traces`,
  resource: { serviceName: "bank-web" },
}).pipe(Layer.provide(OtlpSerialization.layerJson));

Collect it through your Worker

Telemetry.relay receives the browser’s spans in your own Worker and forwards them to wherever the Worker’s telemetry goes, with the Worker’s credentials. No ingest token reaches a browser, and only signed-in users may send. signedIn is the Worker’s check of the request’s session (see Sessions and middleware):

const telemetry = Telemetry.relay({
  authorize: HttpServerRequest.pipe(Effect.flatMap(signedIn), Effect.map(Option.isSome)),
});

Mount it under the Api’s path, so the app’s Worker forwards it like any other call:

if (path.startsWith("/api/telemetry/")) return yield* telemetry;

route decides where each batch goes: the Worker’s own destinations, a separate dataset for browser traces, or anywhere else. Cloudflare Workers Observability only records a Worker’s own spans, so send browser spans to an OTLP destination such as Axiom.

See every outcome

A decide span records how the command ended in domain.outcome: accepted, or the tag of its rejection. A rejection is an answer, not a failure, so its span ends cleanly; only a defect marks a span as failed.

Follow a flow across traces

Some work starts later, or has many causes. A saga can run for days, a projection batch handles events from many commits, and a scheduled command fires long after it was scheduled. Each starts its own trace and links to the span that caused it.

Every span carries domain.correlation_id, the id of the first command in the flow. One query on it finds a whole business flow, across every trace it took.

Record what matters to you

Nothing from a command, an event or a query’s input is recorded unless a handler asks for it. Every handler runs inside the engine’s span for it, already named (Domain.decide Account.WithdrawMoney, Domain.query AccountLedger, Domain.projection Ledger), so plain Effect annotates that span:

decide: {
  WithdrawMoney: Effect.fn(function* (account, cmd, { now }) {
    // Recorded on this command's decide span; nothing else from it is.
    yield* Effect.annotateCurrentSpan("bank.amount", cmd.amount);
    const o = yield* authorized(account);
    // ...
  }),
},

An effectful decide, a query’s handler, a view’s validate, a policy, a saga step, a projection and an external system’s operation all work the same way. Use a plain Effect.fn: naming it would add a span of its own, nested under the engine’s.

Keep secrets out

Record only what you annotate, and never annotate a secret. Make a secret field Schema.Redacted(...), and unwrap it only where it’s used, so it never reaches a trace or a log by accident.

Logs from a handler carry the span’s trace and its domain.* attributes, so they line up with the trace that caused them.

Watch it keep up

The model counts and times its work as metrics: domain.commands and domain.command.duration by command and outcome, domain.events by event, domain.lag from commit to handling for each policy and projection, domain.runs by outcome, domain.projection.batch_size, and domain.external.calls. Lag is the number that tells you whether reactions keep up with commits.

Trace less

Views apply every event they fold, so they’re the busiest spans. Turn them off on the Layer that hosts the model:

Bank.layer(BankHandlers, { telemetry: { views: false } })