Economic debugger analytics¶

This local façade exposes currently implemented unit-economics, opportunity, and model-backtest operations. It is not the hosted service contract.

zeroth.optimization ¶

Economic optimization for production AI workflows.

This is Zeroth's primary product surface. It organizes the existing economics engine around one operational flow without duplicating or relocating it:

measure cost per accepted outcome;
find waste and optimization opportunities;
backtest a cheaper candidate against recorded work; and
enforce an economic release gate.

The underlying :mod:zeroth.econ.analytics package remains public and backward-compatible. This module gives new integrations a small, intentional entry point while the broader runtime stays available as supporting infrastructure.

find_optimization_opportunities ¶

find_optimization_opportunities(
    audits: Sequence[NodeAuditRecord],
    *,
    min_savings_pct: float = 20.0,
    limit: int = 20,
    eligible_run_ids: set[str] | None = None,
) -> SpendReport

Attribute spend per node and rank nodes by right-sizing opportunity.

Considers only nodes that actually spent money (LLM/agent nodes). For each, finds the dominant model, whether it used tools, and — via Mode A's recommend — whether a cheaper capable model exists. Nodes are ranked by total spend (biggest bill first), so the top of the list is where a swap saves the most. recommend is memoized per (model, uses_tools) so a deployment with many nodes on one model costs one lookup.

recommend_model_change ¶

recommend_model_change(
    incumbent: str,
    *,
    needs_tools: bool = False,
    needs_vision: bool = False,
    min_savings_pct: float = 20.0,
    limit: int = 6,
) -> RightsizingResult

Return cheaper, capability-compatible alternatives to incumbent.

Capability is a gate, applied before price: a candidate that can't call the node's tools (needs_tools) or accept images (needs_vision) is disqualified at any price — cheaper-but-can't-do-the-job is not a saving. Among the survivors, only those at least min_savings_pct cheaper (blended) than the incumbent are returned, sorted same-provider-first then cheapest, capped at limit.

Pure and side-effect-free apart from reading litellm's in-memory model DB. An unknown incumbent yields incumbent_known=False and no candidates (never an exception), so a brand-new model degrades to "we can't price this yet" rather than a 500.

build_backtest_dataset ¶

build_backtest_dataset(
    audits: Sequence[NodeAuditRecord],
    *,
    name: str = "rightsizing",
    incumbent_model: str | None = None,
    max_cases: int | None = None,
) -> tuple[EvalDataset, HarvestStats]

Turn a node's audit records into an equivalence dataset (input + incumbent output).

Keeps only successful, tool-free records with a non-empty output snapshot; each becomes an :class:EvalCase whose input is the real per-node input and whose expected is the incumbent's real output. When incumbent_model is given, records a different model produced (a node whose model changed over time) are dropped, so the self-equivalence ceiling is measured against the incumbent's own outputs — not a mix of configs. Also measures the incumbent's mean token profile for the cost projection. Skips are counted, never silent.

backtest_model_change `async` ¶

backtest_model_change(
    *,
    incumbent: ModelOption,
    candidates: Sequence[ModelOption],
    dataset: EvalDataset,
    instruction: str,
    replay_provider: ProviderAdapter,
    judge_provider: ProviderAdapter,
    judge_model: str,
    mean_input_tokens: float,
    mean_output_tokens: float,
    harvest: HarvestStats | None = None,
    node_id: str | None = None,
    tolerance_pct: float = 5.0,
    min_cases: int = 5,
    mode: str = "equivalence",
) -> ExperimentReport

Run the measured right-sizing experiment and produce a ranked, honest recommendation.

Replays the harvested cases through the incumbent (to measure the self-equivalence ceiling) and each candidate, scores equivalence, projects cost on the incumbent's real token profile, and recommends the cheapest capability-compatible candidate whose equivalence is within tolerance_pct of the ceiling. The verdict is confirmed only at >= min_cases cases; below that it is flagged — a lead to test, not a switch.

measure_unit_economics ¶

measure_unit_economics(
    runs: Sequence[Run],
    audits: Sequence[NodeAuditRecord],
    *,
    workflow_limit: int = 20,
    tenant_limit: int = 20,
) -> UnitEconomicsReport

Join a window of top-level runs with their audit spend into unit economics.

runs is the most-recent window for a deployment (already deployment-scoped by the caller); audits are that deployment's audit records. Cost is attributed only to runs in runs -- audit records for out-of-window runs are ignored so the window stays internally consistent. Sub-graph children are dropped (parent_run_id), so an outcome is always a top-level invocation.

analyze_economic_waste ¶

analyze_economic_waste(
    run_id: str,
    run_status: RunStatus,
    audits: Sequence[NodeAuditRecord],
) -> EconReport

Build an :class:EconReport from a run's status and its audit records.

Detectors (non-overlapping by construction, so dollars are never counted twice):

paid_for_failed_run (confirmed): a FAILED run that still incurred cost produced no usable output, so its entire spend is waste.
loop_reexecution (flagged): a node with more than one audit record re-spent on every execution but its most expensive one (so a repeat served from cache, which attributes $0, adds nothing). Counted as flagged waste only for non-failed runs; in a failed run that spend is already inside paid_for_failed_run and is surfaced as a zero-dollar info finding.

Cost comes from NodeAuditRecord.cost_usd (None is treated as 0), which is populated for instrumented LLM calls -- including calls that were paid for and then failed validation (see the runner's cost-on-failure path).

enforce_economic_gate ¶

enforce_economic_gate(
    report: EconReport,
    *,
    max_confirmed_usd: float | None = None,
    max_flagged_usd: float | None = None,
    max_waste_ratio: float | None = None,
) -> None

Raise :class:EconThresholdError if the report exceeds any given limit.

The economic analog of the eval harness's quality gate: lets a pipeline fail when a run burns too much money on waste. Thresholds are opt-in -- an unset limit is never enforced.

compare_workflow_versions ¶

compare_workflow_versions(
    baseline_evidence: VersionEvidence,
    candidate_evidence: VersionEvidence,
    *,
    policy: DecisionPolicy | None = None,
) -> EconomicDecision

Compare a candidate to a baseline without manufacturing confidence.

abstain means the evidence contract was not met. fail means the evidence was sufficient and an economic or outcome constraint failed. pass means every declared constraint was satisfied; it does not claim a causal effect beyond the supplied evidence window.

getattr ¶

__getattr__(name: str) -> object

Resolve one product operation from the existing economics engine.