Skip to content

Fail F# tests whose return value xUnit never executes - #20339

Merged
T-Gro merged 5 commits into
mainfrom
t-gro-research-nonvoid-test-methods
Sep 9, 2026
Merged

Fail F# tests whose return value xUnit never executes#20339
T-Gro merged 5 commits into
mainfrom
t-gro-research-nonvoid-test-methods

Conversation

@T-Gro

@T-Gro T-Gro commented Aug 24, 2026

Copy link
Copy Markdown
Member

An F# [<Fact>]/[<Theory>] that is accidentally a partial application — or otherwise returns a value xUnit never runs, an FSharpFunc, a delegate, or a Lazy<_> — executes none of its assertions yet still reports success. A custom test framework now fails such a method at discovery time by swapping it for xUnit's own ExecutionErrorTestCase; only discovery is customized, so parallelism, console capture, and the rest of the runner chain stay stock.

This is expected to turn currently-green tests red where a body silently never ran.

An F# [<Fact>]/[<Theory>] that is accidentally a partial application (or
otherwise returns a value xUnit never runs -- an FSharpFunc, a delegate,
or a Lazy<_>) executes none of its assertions yet still reports success.
A custom test framework fails such a method at discovery time by swapping
it for xUnit's own ExecutionErrorTestCase; only discovery is customized,
so the stock execution/runner chain (parallelism, console capture) is
untouched.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 77be7136-44a4-468f-b8ec-2e5edeb2a76a
@github-actions

Copy link
Copy Markdown
Contributor

✅ No release notes required

…d-test-methods

# Conflicts:
#	tests/FSharp.Test.Utilities/XunitSetup.fs
@github-actions github-actions Bot added the AI-Tooling-Check-Bypassed Tooling check: non-fork PR, not diff-analyzed label Aug 24, 2026
@T-Gro

T-Gro commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

Guard fired on CI (build 1565003): 8 test legs red, 19 FsharpSuiteMigrated_CoreTests methods return FSharpFunc`2 and never ran.

Root cause — two curried helpers drop the trailing sessionIsolation arg, so the call is a partial application xUnit never executes:

let singleTestBuildAndRunVersion folder mode version = singleTestBuildAndRunAuxVersion folder [] mode version           // missing sessionIsolation
let singleTestBuildAndRunAux folder bonusArgs mode    = singleTestBuildAndRunAuxVersion folder bonusArgs mode LangVersion.Latest  // missing sessionIsolation

Dead tests exposed:

  • array-FSC_DEBUG-8_0, array-FSC_OPTIMIZED-8_0, array-FSC_OPTIMIZED-preview, array-FSI-8_0
  • array-no-dot-FSC_DEBUG, array-no-dot-FSC_OPTIMIZED, array-no-dot-FSI
  • auto-widen-minimal, auto-widen-version-FSC_DEBUG-preview, auto-widen-version-FSC_OPTIMIZED-preview
  • controlChamenos-FSC_OPTIMIZED, controlChamenos-FSI
  • control --tailcalls, controlMailbox --tailcalls
  • namespaceAttributes-FSC_DEBUG, namespaceAttributes-FSC_OPTIMIZED
  • patterns-FSC_OPTIMIZED
  • printf, printf-interpolated

@T-Gro

T-Gro commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@jwosty :
Great work with spotting those initially 👍 .
Doing the "custom xunit analyzer" reveled one more helper not covered. We should have had this ages ago.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 77be7136-44a4-468f-b8ec-2e5edeb2a76a
@T-Gro
T-Gro requested a review from abonie August 25, 2026 11:47
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 77be7136-44a4-468f-b8ec-2e5edeb2a76a
Comment thread tests/FSharp.Test.Utilities/FnGuard.fs
Comment thread tests/FSharp.Test.Utilities/FnGuard.fs
@T-Gro
T-Gro requested a review from abonie August 27, 2026 11:32
@T-Gro
T-Gro enabled auto-merge (squash) August 27, 2026 11:32
@github-project-automation github-project-automation Bot moved this from New to In Progress in F# Compiler and Tooling Sep 9, 2026
@T-Gro
T-Gro merged commit 2dea695 into main Sep 9, 2026
108 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in F# Compiler and Tooling Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-Tooling-Check-Bypassed Tooling check: non-fork PR, not diff-analyzed

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants