Force-include rules and a --include flag for the scan packager (SAST-01)

[Ibrahimrahhal]

Description

A file the CLI leaves out of the archive cannot be scanned whatever the engine later decides about it, so DEFAULT_EXCLUDE_GLOBS and .gitignore silently overruled any platform-side attempt to force-scan misclassified proprietary code. This makes the packager honor force-include rules.

New src/include_rules.rs. Reads the project's rules from GET /api/v1/scan-settings and unions them with the new repeatable --include flag. Patterns accept a path, a directory, or a glob; a bare path also matches its descendants, so the same pattern means the same thing here as in the platform's ignore rules. Expansion walks with the standard ignore filters off — the point is to reach files .gitignore and the default excludes hide — skipping .git, sorted, then capped at 5 000 files so a rule like **/*.js cannot drag a whole node_modules tree into the upload.

Rules are resolved in run(), before the reuse decision. A reusable scan predates any include rule, so its results omit the files the rule forces in. --include refuses --skip-if-commit-scanned-recently at the clap level; project rules are invisible to clap, so they are checked here and refuse reuse too — as does a failed lookup, since not knowing whether the project has rules is not the same as knowing it has none. The refusal still prints CORGEA_SCAN_SKIPPED=false, which the flag promises on every run.

Packaging. create_zip_from_target takes the matched repo-relative paths and exempts them from the default excludes, --exclude, and the walk's gitignore pruning, appending any the walk never reached. Same shape as the existing extra_files path used by --include-image. A force-included file is a complete payload on its own, so a --target that matches nothing no longer aborts the run. The forced paths are printed, not just counted: they override .gitignore and the default excludes, which cover **/*.env among others.

Incremental. IncrementalPlan::including adds force-included files to the changed-file list. The server carries findings forward for every file the diff omits, and an include rule exists precisely because the file was never scanned — so there is nothing to carry forward, and without this the rule would never get the file looked at. Blowing past MAX_CHANGED_FILES falls back to a full scan, which is only slower.

Upload. Only the --include values travel with the archive (include_paths, JSON array); the project's own rules are already stored server-side.

Validation and failure handling. Invalid and repo-wide patterns are refused up front rather than dropped later: a pattern globset cannot compile packages nothing, so the command would otherwise appear to succeed having ignored what was asked for, and ** would disable every exclude rule including .gitignore. Server-supplied patterns stay tolerant, since one bad rule in the web app must not stop a scan. A failed /scan-settings lookup warns and proceeds on the flag alone; a 404 is a backend without the endpoint, which has no rules to apply either. Traversal errors are reported rather than swallowed, so an unreadable directory cannot turn an explicit include into a silent no-match. The lookup is capped at 15s, well under the 150s default, because the run continues without it.

Credentials. A git origin can embed a token (https://oauth2:glpat-x@host/...), so userinfo is stripped before the remote reaches the query string or --verbose output. Only scheme URLs are touched: scp-style git@host:path carries no secret, and stripping its git@ would stop the server recognising the shape and normalizing it to the stored URL.

Related PRs

All three are needed for the feature to work end to end:

• Corgea/doghouse#1983 — stores project include rules, serves /scan-settings, accepts include_paths on upload.
• Corgea/fusion#500 — honors include_paths when classifying extracted files.

Requirement ID: SAST-01 (Post-Migration).

Review feedback addressed

Finding	Fix
Project rules bypassed on scan reuse	Rules resolved in run() before resolve_reusable_scan; project rules and a failed lookup both refuse reuse
Empty --target blocked force-included files	Guard now treats a non-empty force_included as a valid payload
Credentials in the settings query and debug log	strip_remote_credentials on scheme URLs before the request
Traversal errors silently weakened rules	Errors counted and warned rather than dropped by flatten()
Invalid --include globs uploaded but not applied	Rejected at argument handling; server patterns stay tolerant
5 000-file cap applied in traversal order	Sorted before truncation
Broad rules could sweep secrets into the archive	Repo-wide patterns refused; forced paths printed for review
Settings lookup inherited the 150s timeout	15s per-request budget

Testing

tests/cli_scan_include.rs drives the real binary against a stubbed API: --include bundles a node_modules file while its siblings stay excluded, platform-configured rules apply with nothing sent back, an include rule beats --exclude, a target matching nothing still scans the forced file, a repo-wide or invalid pattern is refused before anything is packaged, and a rule matching nothing warns without failing the scan. Unit tests cover reuse refusal, pattern validation, sort-before-cap, credential stripping, reaching gitignored paths, and the incremental changed-file union. The cloud_commands_e2e contract asserts the /scan-settings request in every BLAST plan, that a project rule starts a new scan instead of reusing one, and the flag conflict.

./harness check → Clippy fix, Format, Clippy (strict), Tests (833 passed), Deps skill drift — all OK

One suppression added: #[allow(clippy::large_enum_variant)] on Commands. Scan carries by far the largest flag set, and exactly one Commands value exists per process — parsed at startup and destructured immediately — so boxing would not recover anything real.

  

[cursor]

Stale comment

Project include rules never suppress --skip-if-commit-scanned-recently. The clap conflict only covers the --include flag, so the primary SAST-01 path (rules from the web app, no flag) can reuse a scan that never packaged those files — the same under-reporting this PR already refuses for the flag.

One inline comment with the fix. I did not find a second merge-blocking issue in the packager exemption, the incremental union, or the --include upload field.

  

_{Sent by Cursor Automation: pr-flow}

[corgea-security]

Automated review risk: 5/5.

High-risk under-scanning paths and repository credential exposure must be fixed before merge.

Critical or high-priority changes must be addressed.

Automatic approval was not submitted: automated review found critical or high-priority findings.

[corgea-security]

high: Settings lookup failure silently drops mandatory project rules

A timeout or HTTP error sets lookup_failed but continues with only CLI patterns. A fresh scan therefore still omits files mandated by project include rules while exiting successfully. Preventing scan reuse does not fix this under-reporting; the scan must fail when required settings cannot be retrieved.

Proof or reproduction:

Given a project rule "node_modules/internal/**":
1. Make GET /scan-settings return HTTP 500.
2. Run `corgea scan` with no --include flag.
3. The command warns and succeeds, but node_modules/internal files are absent from the archive.

[corgea-security]

high: Include match cap produces a successful partial scan

When rules match more than 5,000 files, the implementation truncates the matched set and continues successfully. Files after the deterministic cutoff are excluded despite explicitly matching an include rule, contradicting the documented guarantee that force-included files are analyzed. Exceeding the safety limit should fail the scan rather than produce false-negative results.

Proof or reproduction:

Create 5,001 files under node_modules/internal and apply `--include 'node_modules/internal/**'`. `matching_files` returns only 5,000 paths, and the remaining matching file is never added to the archive.

[corgea-security]

nitpick: Timeout documentation is attached to the preview constant

The existing documentation for the scan timeout now precedes FORCE_INCLUDE_PREVIEW, so it incorrectly claims that this preview-size constant controls wait_for_scan polling. Move that sentence back above SCAN_TIMEOUT_ENV.

Proof or reproduction:

/// Overrides how long `wait_for_scan` polls before giving up.
/// How many force-included paths to name...
const FORCE_INCLUDE_PREVIEW: usize = 20;

[corgea-security]

Automated review risk: 4/5.

Include-rule failures can produce successful but incomplete security scans. Previously reported issues appear addressed.

Critical or high-priority changes must be addressed.

Automatic approval was not submitted: automated review found critical or high-priority findings.

[cursor]

Re-reviewed 18a84a8 after the force-include follow-up. The skip-reuse hole I flagged is gone, and I did not find a remaining merge blocker.

Prior finding — addressed. include_rules::resolve now runs in blast::run before resolve_reusable_scan. IncludeRules::reuse_refusal refuses reuse when project rules exist or the lookup failed, and still prints CORGEA_SCAN_SKIPPED=false. a_project_include_rule_refuses_reuse_and_starts_a_new_scan drives that path through the real upload plan (no commit lookup). Clap still blocks --include + the skip flag.

Also verified on this HEAD (not re-raised):

• Empty --target + nonempty force_included is a valid payload (cli_scan_include e2e).
• Packager exemption + append of gitignored / default-excluded paths (create_zip_from_target, project-rules e2e asserts the zip entry and that include_paths is not resent).
• Incremental including() unions forced paths; overflow falls back to a full scan.
• GET /scan-settings is authenticated, 15s-bounded, 404 → no rules, other errors → lookup_failed.
• strip_remote_credentials on scheme URLs before the query/debug log; scp-style remotes left intact.
• Invalid / repo-wide --include rejected in validate_cli_patterns before packaging.
• Matches sorted before the 5k cap; traversal errors counted and warned; forced paths printed.

CI on this SHA is green (rust-tests, platform matrix, native binaries). Remaining related work is the documented doghouse #1983 / fusion #500 pairing, not a CLI defect.

  

_{Sent by Cursor Automation: pr-flow}

[juangaitanv]

./-prefixed patterns never match root-relative paths, so requested files are skipped; could we normalize leading ./ before matching and uploading and cover it with a test?

[juangaitanv]

get_repo_info_for_scan runs a whole-worktree status check although this path only needs repo_url; could we use get_repo_info here?