Fold the code-coverage job into the test matrix: 7 CI jobs per PR to 5

[1-Bort-1]

TL;DR

test-with-code-coverage ran the same suite as the two ubuntu test entries, only instrumented, so every non-draft pull request built and ran the full suite on ubuntu four times. Coverage now rides on the ubuntu 1.12 entry of test and the second job is gone: 7 jobs per non-draft pull request become 5, with the matrix unchanged — ubuntu 1.11, ubuntu 1.12 with coverage, windows 1.12, macOS-latest/aarch64 1.12, plus Documentation.

What the second job was actually adding

BUILD_IS_PRODUCTION_BUILD is read in one place, test/runtests.jl:27-34, where false skips test/bench.jl and nothing else. So test-with-code-coverage differed from the ubuntu entries of test by three things: --code-coverage, not running bench.jl, and caching with julia-actions/cache@v2 rather than actions/cache@v6 — which cached only ~/.julia/artifacts, keyed on Project.toml. Coverage is now a matrix flag on the ubuntu 1.12 entry, julia-processcoverage and codecov are guarded by if: matrix.coverage, every entry caches with julia-actions/cache@v2, and the job carries the 50-minute cap #289 gave the coverage job rather than the 40 test had.

Folding it in also puts coverage behind the github.event.pull_request.draft == false guard that only test carried, so a draft pull request drops from 3 jobs to 1.

Does the fold break bench.jl? — asked on the thread

No, and it is a fair thing to have asked: I assumed the same on the first read. The old job set BUILD_IS_PRODUCTION_BUILD: false, and bench.jl asserts hard allocation ceilings (result.allocs ≤ 50, ≤ 10, ≤ 700, …) that instrumentation looks certain to blow, so I was going to carry that skip across. It does not blow them. Running test/bench.jl under --code-coverage=user and under --code-coverage=none back to back gives 21/21 passing both ways and identical allocation counts at every assertion — 23, 7, 6, 330, 69, 92, 4, 132, 0 — in 26.0 s against 29.5 s. Coverage counters are not heap allocations, so the numbers those tests assert on do not move.

The folded entry therefore runs bench.jl exactly like the other ubuntu entries — BUILD_IS_PRODUCTION_BUILD becomes the constant true rather than a matrix axis with one value in it — and that entry is green in CI with bench.jl in it, not just on this box.

macOS stays

#309 asked for 4 jobs on a pull request, by giving the macOS-aarch64 entry a conditional include that left it out on pull requests and kept it on push-to-main and tags. Asked on the thread to keep macOS, I took that out: the include is a plain three-entry list again, every event runs the same matrix, and a pull request gets 5 jobs rather than 4. The gain is the ubuntu duplication, which was the larger half of it — a pull request now builds and runs the suite on ubuntu twice rather than four times.

Flagged, not fixed: the declared minimum Julia is never tested

Project.toml declares julia = "1.10, 1.11" while CI tests 1.11 and 1.12, so 1.10 — the declared minimum — is untested, before this change and after it. It reads as stale rather than deliberate: only Manifest-v1.11.toml.default and Manifest-v1.12.toml.default are tracked, and bin/install:140-141 resolves Manifest-v${major}.toml.default, so ./bin/install on 1.10 has no manifest to instantiate from at all. The bound is redundant as written too, since ^1.10 already covers 1.11. My recommendation is to bump [compat] julia to "1.11" in a pull request of its own — say the word and I will open it. If 1.10 really is supported, then a 1.10 entry and its tracked manifest are what belong here instead, and that is the more expensive answer.

fail-fast: false is still on the job. That is #279, left alone so this diff holds one idea.

Verification

• Reproduced first: n/a — this is CI configuration, and the job list is the measurement.
• This pull request's own run, all green: run 34686873009 on 5000fb8c — exactly 5 jobs, ubuntu 1.11 (29 min), ubuntu 1.12 with coverage (21 min, inside the 50-minute cap), windows 1.12 (22 min), macOS aarch64 (17 min), Documentation (11 min).
• julia-processcoverage and codecov-action report success on the ubuntu 1.12 job and skipped on the other three, so the if: matrix.coverage guard picks out the one entry; and the coverage: true include entry merges into the existing ubuntu/1.12 combination rather than adding a second ubuntu 1.12 job, which was the one thing here I could not settle by reading.
• test/bench.jl: 21/21 under --code-coverage=user and under --code-coverage=none, identical allocation counts (juliaserver, both exit 0)
• Local CI mirror: PASS (10 min, Julia 1.12.7, one cell of the matrix) · merged up to date with main at 50b1454; main has since released v5.1.0 (0f958d0), which this branch neither needs nor conflicts with.
• Test end-user and developer setup went red on an earlier head of this branch and is green now. It was never this diff: it is the separate setup-test.yml workflow, which CI.yml does not touch, failing in examples/ram_air_kite.jl with argmin over an empty contour at src/airfoil_aero/kulfan.jl:71 — the same exception on release/v5.1.0 and fix/panel-normal-quarter-chord, branches that do not carry this commit. Setup Test is failing #312 fixed it on main and that fix is merged in here.
• Benchmark: n/a
• Risk: coverage: ${{ matrix.coverage || false }} reads matrix.coverage on entries that never define it; GitHub resolves the undefined property to an empty string, so the || falls through to false — that is what the three non-coverage jobs in the run above did, but it is the line I would check first if coverage ever showed up where it should not.

Scope

+16 / -54, all of it .github/workflows/CI.yml.

Closes #309 · task VortexStepMethod.jl-309

[1-Bort-1]

Local full suite: PASS (6 min, Julia 1.12.7, one cell of the matrix)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[1-Bart-1]

Keep macos. Doesn't the fold cause the benchmark tests to fail?