Skip to content

AutoTPPR: minimise Test Top 20 DE MSE, do not maximise it - #27

Open
GuoCheng24 wants to merge 1 commit into
InternScience:mainfrom
GuoCheng24:fix-autotppr-metric-direction
Open

AutoTPPR: minimise Test Top 20 DE MSE, do not maximise it#27
GuoCheng24 wants to merge 1 commit into
InternScience:mainfrom
GuoCheng24:fix-autotppr-metric-direction

Conversation

@GuoCheng24

Copy link
Copy Markdown

tasks/metric_config.json marks Gears / "Test Top 20 DE MSE" as true, which the loader reads as maximize:

"Gears": {
  "Test Top 20 DE MSE": true
}

The value comes from gears_model.test_metrics['mse_de'] (tasks/AutoTPPR/code/experiment.py:1409) — a mean squared error. The flag is read into MetricValue(..., maximize=maximize) at mcts_experiments_utils_claude.py:186-188 and then drives both node selection (improvement = new_metric - local_best_metric, :466) and reward (:516). So a worse test MSE is scored as an improvement and rewarded.

Worked through with the file's own values:

test MSE 0.20 -> 0.35 (worse)
  current config (maximize=true):  improvement = +0.15   scored as progress, rewarded
  with false:                      improvement = -0.15   scored as a regression

Two things make this look like a typo rather than intent:

  1. Every sibling error metric in the same file uses falseIEEE39: {"val/PQ_Vm_rmse": false}, AutoMolecule3D: {"Forces MAE": false}, AutoScale: {"RMSE": false}.
  2. The module's own fallback heuristic, one branch away at :193, infers the direction from the metric name and gets it right — default_maximize = not any(k in metric_lower for k in ["loss", "rmse", "error", "mse", "mae"]) returns False here. The explicit entry is strictly worse than having no entry at all.

One character.

Separately and not changed here: _get_metric_config at :38-46 has its return inside the for loop, so only the first metric of a multi-metric task config is ever reachable. Every task in the file currently has exactly one metric, so it is latent rather than active — mentioning it in case you want it fixed in the same pass.

tasks/metric_config.json marks Gears/"Test Top 20 DE MSE" as maximize=true.
The value comes from gears_model.test_metrics['mse_de'] (experiment.py:1409),
a mean squared error, and the flag drives both node selection and reward in
mcts_experiments_utils_claude.py, so a worse test MSE is scored as progress.
The sibling error metrics in the same file (IEEE39 rmse, AutoMolecule3D MAE,
AutoScale RMSE) all use false, and the module's own fallback heuristic would
also infer false from the metric name.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant