Skip to content

[TS PBT] Evaluate PBT and hybrid configurations on the benchmark #357

Description

@CaelmBleidd

Part of #345. Depends on #354 and #356. #355 is optional, not a blocker.

Goal

Measure the benefit and cost of adding USVM to a mature concrete PBT baseline on the same frozen property corpus.

Protocol

  • Compare PBT_ONLY, SYMBOLIC_ONLY and the sequential HYBRID defined in [TS PBT] Assemble the end-to-end PBT and USVM pipeline #354, initially using fast-check.
  • If [TS PBT] Evaluate runtime-derived PBT hints for symbolic search #355 has a completed, justified hint configuration, evaluate it separately. Otherwise report it as deferred or rejected. Do not delay baseline evaluation for it.
  • Do not claim the sequential portfolio is coverage-guided or uses runtime feedback. Such a claim requires an implemented mode and an ablation against the portfolio.
  • Freeze the corpus, tool revisions, backend/runtime versions, seeds, repetitions, hardware, total/phase budgets and phase allocation before final runs.
  • Use equal total budgets including relevant startup, frontend/adapter work, search, concrete replay and shrinking; account for shared setup consistently.
  • Repeat runs, use a fixed or rotated execution order, and report variability. Fixing a seed does not make timeout-limited symbolic exploration exactly deterministic.

Primary outcomes

  • Number and fraction of distinct validated faulty implementations/mutants found. Count a mutant once even if several properties or inputs expose it. Declare the common denominator and exclusions.
  • Time to the first concrete-confirmed counterexample, with timeouts/unsolved cases treated as censored outcomes rather than dropped.
  • Marginal faults found only, or earlier, with USVM; total hybrid cost.

All reported faults must reproduce in the original TypeScript runtime. A concrete PBT witness must remain reproducible; a symbolic candidate is counted only after replay.

Secondary outcomes

Report replay acceptance, unrepresentable/rejected candidates, minimized counterexample size under a declared metric, optional source coverage, and existing state/solver counters. Unsupported metrics are unavailable, not zero. Separate supported-only comparisons from the full corpus capability inventory.

Use the same instrumented source target definition if reporting coverage. EtsIR reached targets and real TypeScript coverage are separate measures; #382 is required only for claims using its branch mappings.

Definition of Done

  • One command runs the frozen configurations and another regenerates summary tables from raw per-run results.
  • Raw artifacts preserve configuration, phase statuses, candidate/replay distinctions and failures.
  • Baseline findings answer whether and where USVM helps, including no-gain and regression cases.
  • Unsupported cases, selection limits, possible budget bias and threats to validity are explicit.
  • Correctness fixes discovered during evaluation are documented and applied consistently to all configurations before rerunning affected comparisons.
  • No benchmark selection or hidden tuning based on the final results is introduced.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions