feat(quantization): compile algo_cfg into a validated calibration plan [3/4]

[Fridah-nv]

Brief

What changes: Configs can express per-scope algorithm pipelines; invalid ones are rejected with a reason. Nothing runs them until 4/4 (#2537).

How: AlgoCfgEntry mirrors the shape of a quant_cfg entry — one selector plus a cfg — and compile_algo_cfg lowers entries plus algorithm into an ordered list of stages. Compilation is a pure function of the config and the model's structure: no mutation, no forward pass. Eight validation rules, one function each, all reported in a single pass.

---

What does this PR do?

Type of change: new feature

Stack 3 of 4, splitting #2292. Base: #2535. This is the design review — the other three are mechanical by comparison.

#	PR
1	capabilities	#2534
2	write-mask	#2535
3	compile algo_cfg	← you are here
4	execute the plan

The config surface

config = {
    "quant_cfg": [...],                                    # unchanged
    "algo_cfg": [
        {"module_name": "*self_attn*",         "cfg": ["awq_lite", "mse"]},
        {"module_name": "*mlp*",               "cfg": ["mse", {"method": "gptq", "block_size": 64}]},
        {"quantizer_name": "*input_quantizer", "cfg": ["max"]},
    ],
    "algorithm": "max",   # fallback for anything no entry matches
}

An algo_cfg entry has the same {<selector>, "cfg": ...} shape as a quant_cfg entry: quant_cfg entries carry quantizer attributes, algo_cfg entries carry the ordered algorithms for the matched targets. Exactly one selector per entry — module_name for module-level algorithms where the role is implied, quantizer_name when the role must be chosen explicitly — enforced at config-construction time by a pydantic before-validator, so a malformed entry fails before the model is ever consulted.

This expresses two things the current surface cannot: different algorithms for different parts of the model, and an ordered pipeline on the same targets where each stage consumes the previous one's mutated weights and scales.

Compile is a pure function

compile_algo_cfg(config, model) reads the quantized model's structure to resolve globs and validate, but mutates nothing, runs no forward and touches no data. So a bad config fails before any expensive calibration, it is testable without running a model, and the plan is a pure function of (config, structure) — hence identical on every rank, which is what will keep predicate scoping from desynchronizing collectives.

Nothing executes a plan in this PR. That is PR 4. Reviewed on its own, this is a function and the rules that reject a plan that cannot be right.

The rules

Eight, one named function each, all reported in a single pass rather than failing at the first:

• an empty scope (almost always a typo), while a scope matching only disabled quantizers warns instead — an algo_cfg shared across numerics may name a role one of them turns off
• an unscopable algorithm given a scope
• a whole-module algorithm whose scope is not closed over its modules
• an algorithm aimed at a role it does not improve
• fusible siblings split across pipelines (they export to one kernel)
• an algorithm whose own output violates its precondition (awq_lite twice folds a scale into an already-smoothed weight)
• a stage whose every write is overwritten before anyone reads it

Overlap is judged per state token and per quantizer role, so two stages sharing a module do not conflict if they write different roles.

derive_handoff reports when an earlier stage already produced what a later one needs, using coverage rather than overlap: a narrow producer must not let a wider consumer skip its own initialization.

algorithm lowers through the same path as its all-"*" case, so there is no second engine, and an algo_cfg that does not cover the whole model leaves the rest to algorithm via an explicit exclusion rather than an implicit fallthrough.

What validation was worth

Exhaustive compile sweeps over the finished stack — every 2- and 3-algorithm chain across three quantizer layouts (8,712 compiles) plus the scoping surface (4,422) — found four defects, all in what state a rule is evaluated against rather than in the rule logic. Three are in PR 4 with the grid contract they belong to; the fourth (disabled quantizers counting as targets) is here.

Testing

tests/unit/torch/quantization/test_algo_cfg.py — 43 tests: lowering, every rule, derived handoff, coverage, and three fixture shapes (single-level quantizer, SequentialQuantizer, weight-only/no-forward).

pytest tests/unit/torch/quantization/ → 1178 passed, 8 skipped, plus the same two pre-existing test_huggingface.py failures and one pre-existing collection error described in #2534.

Before your PR is "Ready for review"

• Is this change backward compatible?: ✅ — algo_cfg is opt-in; without it nothing changes.
• If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: N/A
• Did you write any new necessary tests?: ✅
• Did you update Changelog?: ❌ — deferred while the stack is under review.
• Did you get Claude approval on this PR?: ❌ — draft.

Additional Information

The config surface and how much of the capability contract belongs in a first cut are what I would most like feedback on.

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA.github.io/Model-Optimizer/pr-preview/pr-2536/
Built to branch gh-pages at 2026-09-23 22:07 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[codecov]

Codecov Report

❌ Patch coverage is 93.52751% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.08%. Comparing base (4178782) to head (9e617c8).

Files with missing lines	Patch %	Lines
modelopt/torch/quantization/algo_cfg.py	93.68%	18 Missing ⚠️
modelopt/torch/quantization/config.py	91.66%	2 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                        @@
##           fridah/algo-cfg-2-write-mask    #2536      +/-   ##
================================================================
+ Coverage                         68.97%   69.08%   +0.11%     
================================================================
  Files                               606      606              
  Lines                             67315    67623     +308     
================================================================
+ Hits                              46429    46717     +288     
- Misses                            20886    20906      +20     

Flag	Coverage Δ
unit	58.81% <93.52%> (+0.16%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.