AutoTPPR: minimise Test Top 20 DE MSE, do not maximise it - #27
Open
GuoCheng24 wants to merge 1 commit into
Open
Conversation
tasks/metric_config.json marks Gears/"Test Top 20 DE MSE" as maximize=true. The value comes from gears_model.test_metrics['mse_de'] (experiment.py:1409), a mean squared error, and the flag drives both node selection and reward in mcts_experiments_utils_claude.py, so a worse test MSE is scored as progress. The sibling error metrics in the same file (IEEE39 rmse, AutoMolecule3D MAE, AutoScale RMSE) all use false, and the module's own fallback heuristic would also infer false from the metric name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tasks/metric_config.jsonmarksGears/"Test Top 20 DE MSE"astrue, which the loader reads as maximize:The value comes from
gears_model.test_metrics['mse_de'](tasks/AutoTPPR/code/experiment.py:1409) — a mean squared error. The flag is read intoMetricValue(..., maximize=maximize)atmcts_experiments_utils_claude.py:186-188and then drives both node selection (improvement = new_metric - local_best_metric, :466) and reward (:516). So a worse test MSE is scored as an improvement and rewarded.Worked through with the file's own values:
Two things make this look like a typo rather than intent:
false—IEEE39: {"val/PQ_Vm_rmse": false},AutoMolecule3D: {"Forces MAE": false},AutoScale: {"RMSE": false}.:193, infers the direction from the metric name and gets it right —default_maximize = not any(k in metric_lower for k in ["loss", "rmse", "error", "mse", "mae"])returnsFalsehere. The explicit entry is strictly worse than having no entry at all.One character.
Separately and not changed here:
_get_metric_configat:38-46has itsreturninside theforloop, so only the first metric of a multi-metric task config is ever reachable. Every task in the file currently has exactly one metric, so it is latent rather than active — mentioning it in case you want it fixed in the same pass.