chore: render coverage numbers into the job summary - #577
Open
runpod-Henrik wants to merge 6 commits into
Open
Conversation
The coverage report was only readable by downloading the artifact. Write a Test Results + Coverage Summary table to $GITHUB_STEP_SUMMARY so the numbers show up on the run page, matching what the ai-api component workflow does. Adds --junitxml so the run's test counts can be reported alongside coverage; pytest-results.xml is gitignored. Stdlib only, so there is no extra install step, and `if: always()` means the summary still renders when tests fail — which is when it is most useful. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review feedback from #373. Three defects, all reachable because this step runs under `if: always()`: * the coverage-XML parse was unguarded while the junit parse beside it was. A report truncated by a timeout, OOM or crashed xdist worker made ET.parse raise, so the summary step exited non-zero and stacked a spurious failure on top of the real one. Verified: the old script exits 1 on a truncated report, the new one exits 0 and says so in the summary. * hits/lines attributes were parsed with bare int(), so a malformed value raised rather than degrading. * the status cell treated `0 failures` as passing even when no tests ran at all. A suite dying at import reports errors>0 with tests possibly 0, so the check is now `no failures AND at least one test`. Also switch the artifact upload to `if-no-files-found: warn`. With `error` a run that never produced coverage.xml — pytest erroring at collection, before pytest-cov writes anything — failed the upload step too, red-flagging the job and masking the root cause. This upload hangs off the PR-gating test job, so it should not be able to fail a PR on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Branch coverage was never collected, so the job summary could only ever show line coverage. Adds --cov-branch. Line coverage is unchanged at 94.3% (3299/3499), so the weekly coverage trend, which reads line coverage, is unaffected. The Cobertura report now carries branch data, which the summary renders as its own row: 88.1% (796/904). One thing to be aware of: --cov-fail-under in pytest.ini gates on coverage.py's total, which now blends lines and branches. That total moves 94.28% -> 93.00% against the 90% gate, so the margin narrows from 4.28 to 3.00 points. It passes, but the gate now measures something slightly different than it did. Happy to lower it to ~88 to preserve the original strictness, or to skip branch collection here, if the tighter margin is unwelcome. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the inline summary script with runpod/internal-workflows/.github/actions/coverage-summary, which now owns both the summary rendering and the artifact upload. The same ~90-line script had been copy-pasted into six repos, and review found the same bugs in every copy — four rounds of a single class where "passed" was the default state and each new way of losing a report had to be patched out of it separately. The shared version computes it from positive evidence instead, and has 59 unit tests plus a smoke job behind it. Pinned by commit rather than tag, so a change to the action cannot reach this repo until someone bumps the SHA. Behaviour was proven on runpod/github-image-builder first: green run, artifact uploaded, and the weekly collector parsed it back at the expected number before the remaining repos followed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A public repository cannot consume an action from a private one, and
runpod/internal-workflows is private. Every matrix leg failed at "Set up
job" with:
Unable to resolve action `runpod/internal-workflows`, not found
This reverts the migration to the shared action for this repo. The inline
script is restored, including all the hardening from review.
The shared action still applies to the private repos — github-image-builder
is migrated and green, main-ui and RunPod follow. Sharing it here needs
internal-workflows to be public, or the action published somewhere public.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the ~90-line inline heredoc summary with the shared action, now that there is a host this repo can actually consume. The first attempt at this pointed at a private repo and failed at `Set up job` on every matrix leg with "Unable to resolve action, not found", so no tests ran at all and it was reverted. A public repository cannot resolve an action from a private one, and this repo is public. The action now lives in runpod/coverage-summary-action, and the same commit is already green in three private consumers plus this repo's sibling. The action owns the coverage artifact upload, so the separate "Upload coverage report" step is gone -- that is the same artifact, not a second one. Note for whoever revisits the gate: --cov-branch narrowed the margin against --cov-fail-under=90 in pytest.ini from 4.28 to 3.00 points. Unchanged here, but it is thinner than it looks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to #576, which published
coverage.xmlas an artifact. That made the number collectable but still only readable by downloading the artifact. This writes a Test Results + Coverage Summary table to$GITHUB_STEP_SUMMARYso it shows up on the run page, matching what the ai-api component workflow already does.Changes
Makefile: add--junitxml=pytest-results.xmltotest-coverage, so the run's test counts can be reported alongside coverage.github/workflows/ci.yml: aCoverage summarystep that renders the tables.gitignore:pytest-results.xmlStdlib only — no extra install step.
if: always()so the summary still renders when tests fail, which is when it's most useful.Rendered output
Verified by running
make test-coveragelocally and executing the step's own script:Plus a collapsible per-file table, least-covered first, so the summary points at what to test next rather than just reporting a number.
Verification
The step was validated by parsing the workflow YAML, extracting the
runblock, and executing that exact block withGITHUB_STEP_SUMMARYset — so what was tested is the heredoc the runner will use, not a copy of it. The embedded Python was also syntax-checked viaast.parse.No production code touched — CI config, one Makefile flag, one gitignore line.
🤖 Generated with Claude Code