Economic debugger analytics¶

This local façade exposes currently implemented unit-economics, opportunity, and model-backtest operations. It is not the hosted service contract.

zeroth.optimization ¶

Economic optimization for production AI workflows.

This is Zeroth's primary product surface. It organizes the existing economics engine around one operational flow without duplicating or relocating it:

measure cost per accepted outcome;
find waste and optimization opportunities;
backtest a cheaper candidate against recorded work; and
enforce an economic release gate.

The underlying :mod:zeroth.econ.analytics package remains public and backward-compatible. This module gives new integrations a small, intentional entry point while the broader runtime stays available as supporting infrastructure.

find_optimization_opportunities ¶

find_optimization_opportunities(
    audits: Sequence[NodeAuditRecord],
    *,
    min_savings_pct: float = 20.0,
    limit: int = 20,
    eligible_run_ids: set[str] | None = None,
) -> SpendReport

Attribute spend per node and rank nodes by right-sizing opportunity.

Considers only nodes that actually spent money (LLM/agent nodes). For each, finds the dominant model, whether it used tools, and — via Mode A's recommend — whether a cheaper capable model exists. Nodes are ranked by total spend (biggest bill first), so the top of the list is where a swap saves the most. recommend is memoized per (model, uses_tools) so a deployment with many nodes on one model costs one lookup.

recommend_model_change ¶

recommend_model_change(
    incumbent: str,
    *,
    needs_tools: bool = False,
    needs_vision: bool = False,
    min_savings_pct: float = 20.0,
    limit: int = 6,
) -> RightsizingResult

Return cheaper, capability-compatible alternatives to incumbent.

Capability is a gate, applied before price: a candidate that can't call the node's tools (needs_tools) or accept images (needs_vision) is disqualified at any price — cheaper-but-can't-do-the-job is not a saving. Among the survivors, only those at least min_savings_pct cheaper (blended) than the incumbent are returned, sorted same-provider-first then cheapest, capped at limit.

Pure and side-effect-free apart from reading litellm's in-memory model DB. An unknown incumbent yields incumbent_known=False and no candidates (never an exception), so a brand-new model degrades to "we can't price this yet" rather than a 500.

build_backtest_dataset ¶

build_backtest_dataset(
    audits: Sequence[NodeAuditRecord],
    *,
    name: str = "rightsizing",
    incumbent_model: str | None = None,
    max_cases: int | None = None,
) -> tuple[EvalDataset, HarvestStats]

Turn a node's audit records into an equivalence dataset (input + incumbent output).

Keeps only successful, tool-free records with a non-empty output snapshot; each becomes an :class:EvalCase whose input is the real per-node input and whose expected is the incumbent's real output. When incumbent_model is given, records a different model produced (a node whose model changed over time) are dropped, so the self-equivalence ceiling is measured against the incumbent's own outputs — not a mix of configs. Also measures the incumbent's mean token profile for the cost projection. Skips are counted, never silent.

backtest_model_change `async` ¶

backtest_model_change(
    *,
    incumbent: ModelOption,
    candidates: Sequence[ModelOption],
    dataset: EvalDataset,
    instruction: str,
    replay_provider: ProviderAdapter,
    judge_provider: ProviderAdapter,
    judge_model: str,
    mean_input_tokens: float,
    mean_output_tokens: float,
    harvest: HarvestStats | None = None,
    node_id: str | None = None,
    tolerance_pct: float = 5.0,
    min_cases: int = 5,
    mode: str = "equivalence",
) -> ExperimentReport

Run the measured right-sizing experiment and produce a ranked, honest recommendation.

Replays the harvested cases through the incumbent (to measure the self-equivalence ceiling) and each candidate, scores equivalence, projects cost on the incumbent's real token profile, and recommends the cheapest capability-compatible candidate whose equivalence is within tolerance_pct of the ceiling. The verdict is confirmed only at >= min_cases cases; below that it is flagged — a lead to test, not a switch.

measure_unit_economics ¶

measure_unit_economics(
    runs: Sequence[Run],
    audits: Sequence[NodeAuditRecord],
    *,
    workflow_limit: int = 20,
    tenant_limit: int = 20,
) -> UnitEconomicsReport

Join a window of top-level runs with their audit spend into unit economics.

runs is the most-recent window for a deployment (already deployment-scoped by the caller); audits are that deployment's audit records. Cost is attributed only to runs in runs -- audit records for out-of-window runs are ignored so the window stays internally consistent. Sub-graph children are dropped (parent_run_id), so an outcome is always a top-level invocation.

analyze_economic_waste ¶

analyze_economic_waste(
    run_id: str,
    run_status: RunStatus,
    audits: Sequence[NodeAuditRecord],
) -> EconReport

Build an :class:EconReport from a run's status and its audit records.

Detectors (non-overlapping by construction, so dollars are never counted twice):

paid_for_failed_run (confirmed): a FAILED run that still incurred cost produced no usable output, so its entire spend is waste.
loop_reexecution (flagged): a node with more than one audit record re-spent on every execution but its most expensive one (so a repeat served from cache, which attributes $0, adds nothing). Counted as flagged waste only for non-failed runs; in a failed run that spend is already inside paid_for_failed_run and is surfaced as a zero-dollar info finding.

Cost comes from NodeAuditRecord.cost_usd (None is treated as 0), which is populated for instrumented LLM calls -- including calls that were paid for and then failed validation (see the runner's cost-on-failure path).

enforce_economic_gate ¶

enforce_economic_gate(
    report: EconReport,
    *,
    max_confirmed_usd: float | None = None,
    max_flagged_usd: float | None = None,
    max_waste_ratio: float | None = None,
) -> None

Raise :class:EconThresholdError if the report exceeds any given limit.

The economic analog of the eval harness's quality gate: lets a pipeline fail when a run burns too much money on waste. Thresholds are opt-in -- an unset limit is never enforced.

compare_workflow_versions ¶

compare_workflow_versions(
    baseline_evidence: VersionEvidence,
    candidate_evidence: VersionEvidence,
    *,
    policy: DecisionPolicy | None = None,
) -> EconomicDecision

Compare a candidate to a baseline without manufacturing confidence.

abstain means the evidence contract was not met, or that the evidence met the contract but an interval still straddles a limit; the reason codes say which, and additional_runs_required estimates the shortfall. fail means an interval establishes a breach of an economic or outcome constraint. pass means every interval lies within its limit; it does not claim a causal effect beyond the supplied evidence window.

getattr ¶

__getattr__(name: str) -> object

Resolve one product operation from the existing economics engine.