Skip to content

[TS PBT] Build a reproducible property benchmark and mutant corpus #356

Description

@CaelmBleidd

Part of #345. Uses #348 and the shared property contract from #384.

Goal

Freeze a meaningful property and mutant corpus before testing optional search hints or drawing comparative conclusions.

Scope

  • Write selection criteria before evaluating engine outcomes. Use curated explanation fixtures and real open-source TypeScript programs.
  • Include round-trip, idempotence, reference-equivalence and collection/relational properties where the selected programs provide meaningful examples. State the mandatory supported categories in the corpus manifest; justify absent categories rather than adding a framework to satisfy a checklist.
  • Author each property once using the existing Kotlin domains and original TypeScript predicate/precondition exports.
  • Keep supported preconditions pure and invocations isolated according to [TS PBT][P0] Align and simplify property execution semantics before integration #384.
  • Record project revision, license, selected files/methods, property rationale and relevant domain limits.
  • Add controlled mutants with stable IDs, a source change, expected violated property and at least one concrete validating witness. Identify equivalent/invalid mutants and exclude them under a documented rule before evaluation.
  • Record capability and reasons for every selected case; preserve unsupported cases in the corpus inventory.
  • Freeze revisions, dependency versions, seeds, budgets, properties and mutants. Separate development/pilot cases from final evaluation cases when tuning hints or phase allocations.
  • Reuse existing loading/execution commands for validation; a simple manifest and runner are sufficient.

Definition of Done

  • Correct fixtures and seeded faulty fixtures are concretely validated.
  • Every mutant counted in the fault denominator is traceable and non-equivalence is supported by a witness.
  • Corpus membership and exclusions do not depend on whether USVM wins.
  • A validation command checks shared-property loading, expected concrete outcomes and essential artifact fields.
  • All configurations consume the same property definitions.
  • The manifest records supported categories, limits, revisions, licenses and default run settings.
  • Selection/authoring may proceed while [TS PBT][P0] Align and simplify property execution semantics before integration #384/[TS PBT] Assemble the end-to-end PBT and USVM pipeline #354 are implemented; final validation uses their shared semantics.

Stateful command-sequence/model-based testing, arbitrary object generators and full coverage of every property category are outside the initial corpus scope.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions