Economic debugger service

Status: implemented self-hostable evidence slice, authenticated SDK ingestion, workflow-version decisions and schedules, plus provider-bill reconciliation, 2026-08-31. This page separates verified capability from intended paid behavior. It is not a claim that a hosted Zeroth service is already available.

One recognizable job

Zeroth explains the economics of an AI workflow before and after a change:

• where spend accumulated over time;
• which workflow version, step, user, customer, or cohort produced it;
• which groups fail or break the pipeline most often;
• what an accepted outcome actually costs; and
• whether a proposed change would reduce cost without violating outcome or governance constraints.

Zeroth does not need to replace LangGraph, an agent framework, an LLM gateway, or an observability backend. Those systems execute, route, or trace work. Zeroth joins economic evidence to outcomes and produces a defensible decision about what to cut, change, or retain.

Product and paywall boundary

The local debugger is the free trust layer. The subscription sells continuous hosted operation and decision history rather than a basic cost dashboard.

Free forever	Approved Solo scope, pending release acceptance
SDK, ingestion, and local verification	Managed ingestion and retained evidence
Single-team cost per successful outcome	Scheduled workflow-version decisions
Run and step economic timeline	Bounded exploratory model experiments
Cohort and breakage queries	Retained comparison and experiment history
Local caps and enforcement mechanics	Metered managed experiment calls
Local model-swap backtests	Observed usage and rate-card projections

The land user may be a solo developer or small AI team that needs a recurring, low-touch way to review observed cost and outcome changes. Organization finance and governance controls are the expansion motion. Project API keys, retained decisions, schedules, enforced quotas, and a vendor-neutral subscription projection are implemented. The approved self-serve offer is Solo at $39/month after a 14-day trial, capped at three hosted backtests and 300 provider-call credits per billing period. WorkOS AuthKit identity and Paddle checkout, portal, and verified webhook adaptation are also implemented behind the optional hosted configuration. Team checkout is disabled until governance limits are enforceable. Production vendor projects, an applied Railway deployment, service terms, and a real transaction are not yet complete.

The billing projection accepts normalized events only after an external adapter has verified their authenticity and resolved the tenant. It keeps exact retries idempotent, rejects changed replays, and retains but does not apply older or equal-time events. This prevents delayed webhook delivery from silently restoring canceled access. Equal timestamps are intentionally first-wins and must be repaired by periodic merchant-state reconciliation when ambiguous.

See the provider bill reconciliation guide for the import contract, closure states, and current limits.

The evidence spine

Every supported ingestion path must converge on one queryable identity chain:

tenant
  → workflow + workflow_version
    → run
      → step + attempt
        → cost + outcome + subject_id + dimensions + occurred_at

Required invariants:

• tenant_id is the security and account boundary; subject_id is the entity being analyzed, such as an end user, customer, account, or job.
• Dimensions are typed and bounded, not an unrestricted high-cardinality bag.
• Workflow version, event time, step identity, and attempt are first-class.
• Measured provider dollars and locally estimated dollars remain separate.
• Missing cost, outcome, or attribution is reported as incomplete evidence, never silently converted to zero.
• Success semantics are immutable per workflow version. A tenant Admin binds one terminal outcome type to an equality or numeric-threshold predicate; changing that rule requires a new workflow version.
• Retries and attempts remain individually visible so repeated work can be distinguished from one expensive first attempt.
• Events may be exported as OpenTelemetry GenAI semantic-convention data; Zeroth does not need to build a second generic tracing dashboard.

Four event contracts currently exist

The first convergence slice extends the working instrumentation and plane contracts with the same debugger identity. It does not introduce a fifth schema.

1. The standalone package under packaging/sdk defines execution, outcome, decision, schedule, and backtest contracts. Execution, outcome, backtest, comparison, schedule, and retained-history routes are served by the plane.
2. zeroth.econ.instrumentation is the absorbed Regulus instrumentation SDK. This is what the runtime actually emits.
3. zeroth.econ.plane owns the server-side ingestion schema. The instrumentation-to-plane path works today.
4. Native Run and NodeAuditRecord persistence owns runtime and signed audit evidence. Instrumentation also writes into this path.

The public SDK is adapted into the working plane evidence store rather than becoming a parallel truth model. The in-repo route gap is closed; the wheel remains blocked because the repository still proves no supported production endpoint, managed provider credentials, checkout, or release operations.

Free debugger queries

The self-hostable service exposes three questions over the converged spine:

1. Timeline: how did cost, successful outcomes, and failure tax change by workflow version and time window?
2. Cohorts: how do cost per successful outcome and failure rate differ by subject_id or typed dimensions?
3. Breakage: which versions and steps contain the most failed-run exposure and explicitly identified repeated-attempt spend?

The implemented routes are GET /v1/debugger/timeline, GET /v1/debugger/cohorts, GET /v1/debugger/breakage, and GET /v1/debugger/report. POST /v1/debugger/outcome-definitions creates the immutable success contract; an exact replay is idempotent, while a changed rule for the same workflow version is rejected. Reads require a tenant-bearing JWT and a read-capable role; definition creation requires Admin. The UI remains an open-source option and is not a prerequisite for the service.

Evidence-gated version decisions

POST /v1/decisions/compare compares two exact workflow versions by measured cost per accepted outcome. It returns an immutable decision artifact with a pass, fail, or abstain verdict and a recommended action. The engine abstains when the run count or outcome coverage is too low, cost is estimated or unmeasured, outcomes are inferred, or the baseline has no accepted outcomes. It also requires an explicit policy.min_success_rate, including in schedules. Omission or null abstains; zero is an explicit customer choice. A policy pass uses review_candidate and means the declared sample constraints were met. The default cost-growth tolerance is 10%, so a pass is not a savings claim. It does not convert missing evidence into zero or market a projected saving as realized value.

Summary values use exact count ratios and decimal cost totals. Economic constraints use two-sided intervals at policy.confidence_level (default 0.95): Newcombe for the success-rate change, Wilson for the candidate rate, and a log-ratio delta interval for cost per accepted outcome. A constraint fails only when its interval is wholly beyond the limit; every interval must clear its limit to pass. Otherwise the comparison abstains and reports the unresolved constraint and estimated additional_runs_required. Small samples can remain undecided even when their point estimates look favorable. Interval calculations assume independent, representative runs; source completeness, causal effects and repeated-look authorization remain unverified. Display rounding does not determine the verdict. A zero baseline cost leaves relative cost change undefined; that comparison abstains.

GET /v1/decisions returns retained history. POST and GET /v1/decision-schedules manage recurring comparisons, and the worker discovers due tenants without a caller-supplied tenant override. Project API keys are high-entropy, hashed at rest, revealed once, role-scoped, and revocable. Hosted-mode quotas cover events, decision scans, active schedules, and schedule frequency. Backtests additionally meter actual provider calls, reserve at most four calls per case, and do not charge an exact immutable retry. Entitlement enforcement is disabled by default for self-hosting.

Backtests and honest attribution

Current model backtesting is real but bounded. POST /v1/backtests accepts 5–25 labeled, tool-free cases, replays the incumbent and one candidate, judges correctness, checks the minimum success-rate constraint, and requires positive projected model savings before it can pass. Raw inputs, expected outputs, and instructions are not retained: history stores a keyed request digest plus the result. Exact retries return that immutable result without another model call. The quality floor must be supplied as constraints.min_success_rate before provider work starts. Input/output usage is priced separately for incumbent, candidate and judge using retained rates. Judge expense is not workload savings. Missing usage abstains; rate-card estimates do not establish invoice charges.

The correctness judge receives the workflow instruction and each case's input alongside its expected answer and replay output. Put shared policy in instruction and case-specific facts in cases[].input. The judge's request groups these as workflow_instruction and case_input; replay still receives the original input. The judge prompt treats expected as supplied reference evidence; assess its validity and record its provenance before using it. For a task that produces both a decision and a customer reply, provide a complete reference covering the required facts, clarifications and permitted promises. Agreement on the decision field alone does not establish correctness of the whole response. These input requirements do not establish judge calibration or replace independent held-out validation.

New hosted backtests use whole-answer verdicts in correctness-replay/4. The judge assesses every substantive claim and field against the workflow, case facts and supplied reference. A matching decision cannot excuse invented facts, unauthorized promises, contradictions or missing required information. Harmless paraphrases are acceptable; conflicting or insufficient evaluation evidence yields unresolved. This rubric requires independent validation; it is not proof that an LLM will apply those rules accurately.

evaluation_evidence retains the catalog-resolved incumbent, candidate and judge model references, evaluator version and rubric hash. The configured judge is openai/gpt-5.6-terra. Comparisons with Terra (including dated and -latest suffixes) abstain before provider calls. No dated Terra snapshot is currently exposed, so this model ID does not freeze future provider revisions. Distinct model identity removes direct self-grading; shared-provider bias and label uncertainty remain. Each role is priced using its own observed usage and retained catalog rates. Catalog quotes can differ from current promotional or cache-adjusted charges; these projections do not establish the provider bill. parameters="judge_max_tokens_8192" records a 8192-token judge output limit, including reasoning tokens; all other judge parameters and replay parameters use provider defaults (Terra reasoning is currently medium). Incomplete or malformed judge output remains an evaluation error. Candidate settings other than model abstain before provider work.

Each evidence entry pairs incumbent and candidate outcomes by case_index, the zero-based position in the case list bound by the request digest. correct maps to passed/1 and incorrect to failed/0. The retained pass_threshold=1 describes this binary encoding; no numeric confidence score or calibrated cutoff determines the verdict. unresolved, replay_error, judge_error and not_run carry null. Divide passed cases by all cases to reproduce each reported success rate. Unresolved judgments and execution errors count against that rate and force abstention; they are not established wrong answers. An unrun candidate has unavailable rates. The existing candidate success-rate floor still applies to the observed fraction of correct replies.

Historical /1, /2 and /3 evidence and exact retries retain their original version, rubric, threshold and results. /3 changed reference-provenance wording; /2 added workflow context to judge requests; /1 relied on case input for that context. Those methods used numeric grading at 0.7 with the incumbent as judge. The legacy OSS CorrectnessScorer keeps that behavior.

Raw case IDs, inputs, references, outputs, rationales and exception text are not retained in this object. Customers retain their original ordered cases for audit.

Exact retries and history return the retained evaluation evidence without another judge call. Historical reports lacking the object return null; their stored data and original verdict remain unchanged. This supports arithmetic reconstruction, not judge calibration or reproduction of a provider's future answer.

Claim and method contract

Output	Claim class / method	Evidence scope and limitations
Debugger report and aggregate rows	Historical accounting: observed_economic_exposure / observed-accounting/1	Recorded events in the requested window, with measured/estimated/missing money and unresolved outcomes separate. Failed-run exposure is not step causality.
Provider-bill reconciliation	Historical accounting: measured_cost_proportional allocation / measured-cost-proportional/1	Named statement digest and period. Allocation follows measured telemetry; unmatched dollars and variance remain visible.
Version comparison and scheduled result	observed_comparison / interval-policy/1	Received evidence for the named versions; outcomes use the latest non-future assertion and resolve only with declared final maturity and compatible definitions. Source completeness and business truth are not established. Policy checks describe that evidence only.
Hosted model backtest	exploratory_model_experiment / observed-replay-policy/1	Supplied 5–25 cases; observed correctness according to the current judge and text-usage projections at retained rates. Judge calibration and population generalization are unvalidated. Cache, tools, downstream charges and unobserved retries are excluded.
Public model-migration result	Experimental scenario diagnostics / nested-paired-monthly-v3-hoeffding99-math1-predictive1 in the retained request	Supplied paired observations, demand periods and policy assumptions. Predictive reliability is unapproved; the public result always abstains. An unqualified risk law returns no action distributions.
Migration PDF and delivery	Experimental presentation / model-migration-v3	Renders the retained decision snapshot. Simulation percentiles and recorded legacy actions do not establish future coverage or authorize rollout. Previously generated artifacts remain unchanged.
Randomized-rollout verification	Legacy causal method / homogeneous-assignment-v1 in the verification digest	Requires a pre-existing recommended decision, retained assignments and post-assignment observations. Creation, assignment and verification require a legacy JWT; paid keys and browser sessions cannot invoke them, including for retained state. Legacy verification is not accepted paid causal authorization.
Historical result without these fields	legacy_unclassified / legacy_unversioned	Original values and action remain readable. Missing metadata cannot establish the current method or completeness.
Legacy counterfactual estimates	Legacy OSS method only	Heuristic confidence, calibration and proxy-dollar outputs are not paid statistical, causal or forecasting authorization.

Fresh accounting reports and debugger rows include method_version; Markdown exports preserve it. Old payloads without that field load as legacy_unversioned. The identifier names the calculation rules, not a frozen source snapshot. Retain the response and request filters to audit a live query; a later query can select new evidence even under the same method version.

New version comparisons and backtests retain claim_class, method_version and limitations. Both use review_candidate for a pass, never automatic rollout. Their no_statistical_causal_or_forecast_authorization limitation applies even when all observed cases pass. Repeated scheduled looks do not create population confidence. Retained normalized calculation inputs make comparison arithmetic reproducible. Raw source snapshots, source closure and independent outcome truth remain acceptance work.

Hosted comparisons bind each version to a source_evidence fingerprint tagged stored-assertions/1 (v2 for capture-window identity, v3 for charge ownership, v4 for declared outcome maturity, v5 for charge-cost revisions), with selected execution, outcome and cost-revision counts. The digest covers immutable assertions and their multiplicity, independent of record order, database row IDs and receipt times. A later selected outcome or zero-cost step creates a new retained revision even when the totals do not change. No raw payload is copied into the report. A fingerprint cannot prove missing streams were delivered, reconstruct erased source records, or serve as a signature. Historical reports keep an empty source binding.

Reconstruct a retained comparison

New comparison results include calculation_inputs, version run-economics/1. Its baseline and candidate lists preserve the normalized economic inputs used for that report. Each row contains cost_usd, cost_measurement, accepted, outcome_measurement and runs. Identical input tuples are grouped; runs is their multiplicity. Zero is an observed amount and null is unknown. Amounts are exact decimal strings and may use exponent notation such as 1E-8; parse them with decimal or rational arithmetic, not binary floating point.

For each side, these rules reconstruct VersionEconomics:

Number	Calculation from input rows
Runs	Sum runs over every row.
Labeled, accepted, rejected runs	Sum multiplicities where accepted is non-null, true, or false, respectively.
Outcome coverage	Labeled runs / all runs; zero when there are no runs.
Success rate	Accepted / labeled runs; unavailable when there are no labels.
Inferred outcome runs	Labeled multiplicities whose outcome_measurement is not measured.
Measured, estimated, unmeasured runs	Sum multiplicities by cost_measurement.
Measured and estimated cost	Sum cost_usd × runs separately for each provenance.
Cost per accepted outcome	Divide eligible total cost by accepted runs, only when all runs are labeled, no cost is unknown, at least one run is accepted, and estimated costs are either absent or explicitly allowed by the retained policy.

The input unit is a whole run. If one required charge is unknown or execution delivery mismatches the inventory, that run's cost is unknown here. A debugger's visible subtotal of known components has a different scope. Unresolved outcomes never become rejected outcomes merely to complete a denominator. Inferred labels and estimated costs keep their provenance even when a policy permits them.

The report also retains the policy and source/definition bindings. Use exact ratios of the retained totals and counts when checking its cost-change or success-change constraints; displayed decimal quotients and floating-point changes are presentation values. A calculation replay does not establish statistical or causal validity, source delivery, invoice truth or the correctness of normalization.

Late evidence produces a new comparison record. Read the original decision ID in GET /v1/decisions to retrieve its original calculation inputs; do not recompute the old report from today's event selection. These inputs remain with retained reports after source erasure, but add no run, event, charge or subject identifiers, prompts, metadata or outcome payloads. Small-population economic figures are not promised to be anonymous. Existing report-retention obligations still apply. Historical reports without calculation_inputs return null; they cannot acquire missing inputs retrospectively. The independent source ledger is still needed to trace assertions to provider charges and verify their meaning.

The application must assign a new workflow version when prompts, models, tools or other execution configuration affecting economics change. An unchanged label does not prove an unchanged configuration. Comparisons permit the same version on both sides, including distinct declared execution windows; identical inputs describe the same observations, not a new improvement. No automatic freshness cutoff is applied. Review window age and population relevance before using old observations for a new change. Identical requests and selected evidence reuse the original retained decision and evaluation time. Policy or selected evidence changes produce another report revision; rerunning unchanged evidence does not refresh it.

Comparisons can also receive caller-owned source inventories for both versions. Executions carry an immutable source_window_id; each inventory declares its inclusive capture interval, terminal runs, and expected event counts/ID digests. The producer must construct this inventory independently of successful delivery. Entirely missing runs remain visible with unknown cost. Partial runs, unexpected records and mismatched IDs cannot pass the comparison. Reports retain the inventory digest and reconciliation counts in source_delivery; a changed inventory or selected record set creates another immutable report revision. They do not retain another list of customer run IDs.

This is bounded reconciliation of received executions with a supplied declaration. It does not freeze business outcomes, prove physical charge ownership, or verify that all source windows were declared. A matched inventory retains the existing source-completeness limitation. Fixed inventories are optional for direct comparisons; recurring schedules retain their observed-history behavior. See the standalone SDK README for the wire format and independent producer-ledger example.

Stored comparisons require an immutable outcome definition for each workflow version, using the same predicate as the debugger and provider allocation. Missing rules and wrong outcome types remain unresolved; different cross-version predicates force abstention. The latest uninterpretable observation cannot restore an older success. Reports retain outcome_semantics definition/rule digests in the existing decision identity, so later definitions create a new revision without changing old history. Definitions specify interpretation, not outcome maturity or independent business truth. The existing outcome-definition endpoint accepts Admin cloud credentials; the lean SDK exposes create_outcome_definition. See its README for the declaration and compatibility transition.

Explicit monetary ownership uses cost_role="charge" plus a tenant-wide charge_id. The existing execution table admits one owner per declared charge identity; another capture cannot claim its dollars, even in a concurrent write. Structural summary events carry no money and their metadata usage is not priced. Distinct charged attempts remain additive. Comparisons retain charge_ownership counts and declaration status. The debugger reports structural summary_events separately from measured, estimated and unmeasured monetary events; a summary-only run remains incomplete. Legacy amounts remain visible with ownership unverified. These identities are caller assertions, not independent invoice evidence. Use the existing append-only charge-cost revisions and explicit outcome maturity contracts described in the standalone SDK guide; they preserve assertions and interpretation, not independently verified business truth.

Paid project keys and WorkOS browser sessions cannot authenticate the legacy /v1/evaluations/* routes. Those retain the legacy JWT boundary for self-hosted use. Hosted deployment must keep the development token issuer disabled and protect service credentials; local credential tests do not replace deployed security acceptance.

The isolated node replay abstains when pricing is unknown or when asked to prove cost per business outcome or critical-error limits that its evidence cannot establish. It does not yet simulate structural workflow changes such as retry limits, optional verification steps, conditional paths, context reduction, or redundant tool sequences.

Current waste attribution remains narrower than complete causality:

• loop re-execution can attribute dollars to a node;
• failed-run spend is confirmed at run level;
• repeated-attempt cost is priced only when the emitter supplies attempt > 1; it does not prove the step caused the final failure; and
• cache inefficiency is run-wide rather than step-attributed.

Therefore current output may rank opportunities, but must not claim complete step-level causal attribution. Structural backtests come after the event spine can identify versions and attempts reliably.

Initial acceptance evidence

The first vertical slice has been exercised over 10,000 deterministic runs. The verifier reconciled 5,000 successes, 5,000 failures, $100 measured spend, $50 failed-run exposure, and $0.02 measured cost per successful outcome. Its contract tests also cover multiple subjects, typed cohorts, steps, attempts, measurement channels, idempotency, and tenant isolation. The service preserves:

• timeline, cohort, and breakage totals reconcile to the source fixture;
• tenant isolation is enforced on every read and write;
• measured and estimated spend never cross-contaminate;
• unknown or incomplete attribution is visible;
• replaying the same idempotent event cannot double-count spend; and
• the same dataset can be exported without requiring the web UI.

Decision-ready results additionally require an immutable outcome definition for every workflow version in the selected window. Undefined versions remain unresolved and are named in the diagnostic; Zeroth does not infer success from positive numbers, booleans, strings, fraud flags, or other business values.

Debugger windows accept at most 50,000 execution events per request. Larger windows return HTTP 422, so their totals cannot silently describe a truncated population. This is a bounded single-team debugger, not an organization-scale warehouse. Paid rollups should be pre-aggregated rather than increasing that request-time limit indefinitely.

Only after this vertical slice should Zeroth extend the existing right-sizing harness to retry, step-removal, or conditional-path simulations.

Preserved repository surfaces

The graph runtime, service API, governance subsystems, integrations, Studio, console, memory, RAG, sandbox, deployment code, and SDK prototype stay in the repository. Nothing is deleted by this narrowing. Their product roles are:

• runtime and integrations produce economic and governance evidence;
• governance constrains and proves consequential changes;
• the console remains the optional open-source interface;
• the local zeroth.optimization façade exposes existing economic analytics; and
• legacy capabilities remain available without expanding the primary service.

Adversarial review

The strongest objection is that free cost-per-outcome analytics may attract users without converting them. That risk is real: standalone AI observability has consolidated, gateways already optimize model price, and basic cost visibility is widely free. The defensible bet is narrower: use economics as the demand hook, then charge only when data crosses team, finance, security, or audit boundaries. The simpler fallback is to keep the debugger entirely open source and sell only managed hosting plus SSO/RBAC/retention if proof-of-savings backtests do not create willingness to pay.