Economics

zeroth.econ.analytics

Economics analytics -- cost tracking and spend analysis for every LLM call.

Public API: - InstrumentedProviderAdapter: wraps any ProviderAdapter to emit cost events - RegulusClient: thin wrapper around the instrumentation SDK client - CostEstimator: USD cost estimation via litellm pricing data

Every export resolves lazily. Economics analytics read runs, audit records, and provider adapters, so importing this package eagerly would pull the runtime and governance layers into anything that touches it -- historically including zeroth.platform.config.settings, which only needed RegulusSettings (defined in :mod:zeroth.platform.config.models and republished from :mod:zeroth.econ.analytics.models). That made the platform layer transitively import the run domain. Resolving on first attribute access keeps from zeroth.econ.analytics import X working while still letting a submodule be imported on its own.

BudgetCheckResult

Bases: BaseModel

Rich budget status that distinguishes outages from unlimited success.

BudgetEnforcer

BudgetEnforcer(
    regulus_base_url: str | None = None,
    *,
    cache_ttl: int = 30,
    timeout: float = 5.0,
    headers_provider: Callable[[], dict[str, str]]
    | None = None,
    fail_closed: bool = True,
    asgi_app: Any | None = None,
    _transport: Any = None,
)

Pre-execution budget check against Regulus backend (per D-10).

Queries the Regulus /budget/status endpoint for the tenant's current spend and budget cap. Results are cached with a TTL to avoid a network round-trip on every agent call (per D-11).

If the Regulus backend is unreachable or returns an error, the platform and direct construction both use fail_closed=True by default; development callers must explicitly select compatibility mode. The fail-closed switch applies ONLY to the error path — a successfully fetched null cap always stays unlimited.

When asgi_app is supplied the enforcer talks to that ASGI app in-process (the bundled /regulus mount), so a default bundled deploy reaches the plane on Zeroth's own port instead of the external localhost:8000 default. When asgi_app is None the existing external-HTTP path is used unchanged.

check_budget async

check_budget(tenant_id: str) -> tuple[bool, float, float]

Check whether tenant_id is within its budget cap.

Returns (allowed, current_spend, budget_cap).

• allowed is True when the tenant may proceed.
• On a backend error the method returns (False, 0.0, 0.0) when fail-closed, or (True, 0.0, inf) when fail-open. The error path never writes the cache.
• Successful results are cached per tenant for the configured TTL.

check_budget_status async

check_budget_status(tenant_id: str) -> BudgetCheckResult

Return budget status including whether the backend check degraded.

RegulusClient

RegulusClient(
    *,
    base_url: str = "http://localhost:8000/v1",
    timeout: float = 5.0,
    enabled: bool = True,
    headers_provider: Callable[[], dict[str, str]]
    | None = None,
    _asgi_app: Any | None = None,
)

Zeroth-side wrapper around the Regulus InstrumentationClient.

Holds a single InstrumentationClient and exposes track_execution() for fire-and-forget cost event emission, plus stop() for graceful shutdown that flushes the transport buffer.

base_url property

base_url: str

Return the Regulus API base URL.

track_execution

track_execution(event: ExecutionEvent) -> None

Fire-and-forget: enqueue an execution event for delivery to Regulus.

track_execution_confirmed

track_execution_confirmed(event: ExecutionEvent) -> None

Persist one execution before returning or raise on delivery rejection.

stop

stop() -> None

Flush pending events and stop the transport thread.

CostEstimator

Estimates USD cost for LLM calls using litellm's pricing data.

estimate

estimate(
    model_name: str,
    *,
    input_tokens: int,
    output_tokens: int,
) -> Decimal

Return estimated USD cost as a Decimal.

Uses litellm.cost_per_token() internally. Returns Decimal("0") if the model is unknown or any error occurs during lookup.

NodeSpend

Bases: BaseModel

Attributed spend and right-sizing potential for one node.

SpendReport

Bases: BaseModel

Deployment-wide spend attribution, ranked by right-sizing opportunity.

QualityEconomicsReport

Bases: BaseModel

Cost per quality success over the labeled subset of the window.

Every headline travels with its coverage so the number can never be read alone.

RunQualityVerdict

Bases: BaseModel

An externally-attached judgement of whether a run's output was good.

ModelOption

Bases: BaseModel

One cheaper, capability-compatible alternative to the incumbent model.

ref property

ref: str

The provider/model string a node's model_provider field expects.

RightsizingResult

Bases: BaseModel

Cheaper capability-compatible candidates for an incumbent model, or why there are none.

incumbent_known is False when litellm has no pricing for the incumbent (e.g. a model newer than the installed litellm) — the honest signal that this is a pricing gap, not "nothing is cheaper". Never let an unknown incumbent read as "no savings available".

CandidateOutcome

Bases: BaseModel

One model's measured result in a right-sizing experiment.

CorrectnessScorer

CorrectnessScorer(
    provider: ProviderAdapter,
    model_name: str,
    *,
    pass_threshold: float = 0.7,
    name: str = "correctness",
)

LLM judge for absolute correctness against a human-provided answer (ECON-RIGHTSIZE-04).

Unlike :class:EquivalenceScorer (candidate vs the incumbent's own output), this grades the candidate against the reviewer's CORRECT answer — so a cheaper model is judged on whether it is right, not merely on whether it matches the model you're replacing. It is the honest bar for high-stakes nodes: equivalence inherits the incumbent's mistakes; correctness catches them. Same errored-not-zero rail as the equivalence judge.

score async

score(output: object, case: EvalCase) -> Score

Judge whether output is correct against case.expected (the human answer).

EquivalenceScorer

EquivalenceScorer(
    provider: ProviderAdapter,
    model_name: str,
    *,
    pass_threshold: float = 0.7,
    name: str = "equivalence",
)

LLM judge for symmetric equivalence-to-reference (ECON-RIGHTSIZE-02).

Distinct from LLMJudgeScorer (asymmetric graded quality against a rubric): this asks a symmetric "are these two responses to the same request equivalent for the user's purpose?" and normalizes both sides to text first. A provider failure or unparseable verdict yields an errored Score (never a silent zero), so a flaky judge can't read as a quality regression — the same rule run_eval applies to errored cases.

score async

score(output: object, case: EvalCase) -> Score

Judge whether output (candidate) is equivalent to case.expected (incumbent).

ExperimentReport

Bases: BaseModel

Result of a measured right-sizing experiment for one node.

HarvestStats

Bases: BaseModel

What the audit-trail harvest yielded and dropped.

TenantEconomics

Bases: BaseModel

Unit economics for one tenant (customer) within a deployment.

The 'which customer is unprofitable' view: same shape as WorkflowEconomics, keyed by tenant_id instead of workflow_name.

UnitEconomicsReport

Bases: BaseModel

Deployment-wide unit economics: what a successful outcome costs, and the failure tax.

Computed over a bounded, most-recent window of top-level runs (window_runs); the dollar figures are for that window, not all time.

WorkflowEconomics

Bases: BaseModel

Unit economics for one workflow within a deployment.

EconReport

Bases: BaseModel

Per-run economic-waste report built from the run's audit records.

confirmed_waste_usd property

confirmed_waste_usd: float

USD of epistemically-certain waste (e.g. all spend on a failed run).

flagged_waste_usd property

flagged_waste_usd: float

USD flagged for review -- recoverable if unintended (e.g. loop re-runs).

waste_ratio property

waste_ratio: float

Fraction of total spend that is confirmed or flagged as waste.

summary

summary() -> dict[str, Any]

Return a JSON-friendly summary of the report's headline metrics.

EconThresholdError

Bases: Exception

Raised by :func:waste_gate when a report exceeds its configured limits.

WasteFinding

Bases: BaseModel

One detected (or flagged) unit of economic waste in a run.

confirmed separates epistemically-certain waste (a failed run produced no usable output) from spend flagged for review (recoverable only if it was unintended -- e.g. a loop that may be deliberate refinement).

WasteKind

Bases: StrEnum

The category of an economic-waste finding.

WasteKindTotal

Bases: BaseModel

Per-kind waste totals across the window.

WasteRollup

Bases: BaseModel

Deployment-wide economic-waste rollup over a window of runs.

WasteRollupFinding

Bases: WasteFinding

A waste finding tagged with the run it came from, so it stays actionable.

spend_opportunities

spend_opportunities(
    audits: Sequence[NodeAuditRecord],
    *,
    min_savings_pct: float = 20.0,
    limit: int = 20,
    eligible_run_ids: set[str] | None = None,
) -> SpendReport

Attribute spend per node and rank nodes by right-sizing opportunity.

Considers only nodes that actually spent money (LLM/agent nodes). For each, finds the dominant model, whether it used tools, and — via Mode A's recommend — whether a cheaper capable model exists. Nodes are ranked by total spend (biggest bill first), so the top of the list is where a swap saves the most. recommend is memoized per (model, uses_tools) so a deployment with many nodes on one model costs one lookup.

quality_economics

quality_economics(
    runs: Sequence[Run],
    audits: Sequence[NodeAuditRecord],
    *,
    min_coverage: float = 0.2,
) -> QualityEconomicsReport

Cost per quality success over the labeled terminal subset of top-level runs.

Only runs with a good/bad verdict enter the metric; unknown/unlabeled runs are excluded from both numerator and denominator. cost_per_quality_success divides the labeled terminal spend by the good labeled runs. Below min_coverage (or with zero labels) the headline is None with an explanatory state.

read_quality_verdict

read_quality_verdict(run: Run) -> RunQualityVerdict | None

Parse a run's attached quality verdict, or None if absent/malformed.

Never raises and never defaults to good -- a malformed blob is treated as no verdict.

describe

describe(model: str) -> ModelOption | None

Build a :class:ModelOption for a single model (pricing + capability), or None.

Used to represent the incumbent itself with the same shape as its candidates — the measured experiment needs the incumbent's input/output prices to project cost. Returns None when litellm has no pricing for the model. savings_pct is 0 and same_provider is True by definition (it is its own reference).

recommend

recommend(
    incumbent: str,
    *,
    needs_tools: bool = False,
    needs_vision: bool = False,
    min_savings_pct: float = 20.0,
    limit: int = 6,
) -> RightsizingResult

Return cheaper, capability-compatible alternatives to incumbent.

Capability is a gate, applied before price: a candidate that can't call the node's tools (needs_tools) or accept images (needs_vision) is disqualified at any price — cheaper-but-can't-do-the-job is not a saving. Among the survivors, only those at least min_savings_pct cheaper (blended) than the incumbent are returned, sorted same-provider-first then cheapest, capped at limit.

Pure and side-effect-free apart from reading litellm's in-memory model DB. An unknown incumbent yields incumbent_known=False and no candidates (never an exception), so a brand-new model degrades to "we can't price this yet" rather than a 500.

build_experiment_dataset

build_experiment_dataset(
    audits: Sequence[NodeAuditRecord],
    *,
    name: str = "rightsizing",
    incumbent_model: str | None = None,
    max_cases: int | None = None,
) -> tuple[EvalDataset, HarvestStats]

Turn a node's audit records into an equivalence dataset (input + incumbent output).

Keeps only successful, tool-free records with a non-empty output snapshot; each becomes an :class:EvalCase whose input is the real per-node input and whose expected is the incumbent's real output. When incumbent_model is given, records a different model produced (a node whose model changed over time) are dropped, so the self-equivalence ceiling is measured against the incumbent's own outputs — not a mix of configs. Also measures the incumbent's mean token profile for the cost projection. Skips are counted, never silent.

build_labeled_dataset

build_labeled_dataset(
    audits: Sequence[NodeAuditRecord],
    expected_by_run: Mapping[str, str],
    *,
    name: str = "rightsizing-correctness",
    incumbent_model: str | None = None,
    max_cases: int | None = None,
) -> tuple[EvalDataset, HarvestStats]

Turn a node's audit records into a CORRECTNESS dataset using human-labeled answers.

Like :func:build_experiment_dataset, but each case's expected is the reviewer's correct answer (expected_by_run[run_id]) rather than the incumbent's own output — so a candidate is graded against ground truth, not the model being replaced. Only tool-free records whose run carries a human-provided expected answer become cases; success status is NOT required (a labeled run the incumbent got wrong is a valid — and valuable — case).

run_experiment async

run_experiment(
    *,
    incumbent: ModelOption,
    candidates: Sequence[ModelOption],
    dataset: EvalDataset,
    instruction: str,
    replay_provider: ProviderAdapter,
    judge_provider: ProviderAdapter,
    judge_model: str,
    mean_input_tokens: float,
    mean_output_tokens: float,
    harvest: HarvestStats | None = None,
    node_id: str | None = None,
    tolerance_pct: float = 5.0,
    min_cases: int = 5,
    mode: str = "equivalence",
) -> ExperimentReport

Run the measured right-sizing experiment and produce a ranked, honest recommendation.

Replays the harvested cases through the incumbent (to measure the self-equivalence ceiling) and each candidate, scores equivalence, projects cost on the incumbent's real token profile, and recommends the cheapest capability-compatible candidate whose equivalence is within tolerance_pct of the ceiling. The verdict is confirmed only at >= min_cases cases; below that it is flagged — a lead to test, not a switch.

analyze_run

analyze_run(
    run_id: str,
    run_status: RunStatus,
    audits: Sequence[NodeAuditRecord],
) -> EconReport

Build an :class:EconReport from a run's status and its audit records.

Detectors (non-overlapping by construction, so dollars are never counted twice):

• paid_for_failed_run (confirmed): a FAILED run that still incurred cost produced no usable output, so its entire spend is waste.
• loop_reexecution (flagged): a node with more than one audit record re-spent on every execution but its most expensive one (so a repeat served from cache, which attributes $0, adds nothing). Counted as flagged waste only for non-failed runs; in a failed run that spend is already inside paid_for_failed_run and is surfaced as a zero-dollar info finding.

Cost comes from NodeAuditRecord.cost_usd (None is treated as 0), which is populated for instrumented LLM calls -- including calls that were paid for and then failed validation (see the runner's cost-on-failure path).

waste_gate

waste_gate(
    report: EconReport,
    *,
    max_confirmed_usd: float | None = None,
    max_flagged_usd: float | None = None,
    max_waste_ratio: float | None = None,
) -> None

Raise :class:EconThresholdError if the report exceeds any given limit.

The economic analog of the eval harness's quality gate: lets a pipeline fail when a run burns too much money on waste. Thresholds are opt-in -- an unset limit is never enforced.

waste_rollup

waste_rollup(
    runs: Sequence[Run],
    audits: Sequence[NodeAuditRecord],
    *,
    top_n: int = 10,
) -> WasteRollup

Aggregate per-run waste over a window of top-level runs into a deployment rollup.

Calls :func:analyze_run per top-level run (grouping that run's audits, passing its status) and sums the confirmed/flagged buckets. run.status is passed straight through -- :func:analyze_run compares it against RunStatus.FAILED and reads its .value defensively, so an enum or a bare string both work (no coercion that could raise on an unexpected value). Sub-graph children are dropped (parent_run_id).

__getattr__

__getattr__(name: str) -> object

Resolve a public econ symbol from its submodule on first access.

The resolved value is cached in the package namespace. That is not just an optimization: unit_economics names both a submodule and the function it exports, and importing the submodule binds it as an attribute of this package. Caching the function over it reproduces what an eager from ... import unit_economics would do, so zeroth.econ.analytics.unit_economics keeps resolving to the callable.