feat(operator): export the estimators and the Hugging Face models

[kz930]

What changes were proposed in this PR?

The scikit-learn estimators are fitted on one port and score on the other, so
the script holds both frames and narrows each by the same rule: a fit and a
score taken on different columns would compare two different models.

The four Hugging Face models declare the column types they take, and the iris
regression keeps the row when a petal measurement is empty rather than ending
the run on it.

Sklearn Prediction and Sklearn Testing are reported as unverifiable rather
than exported blind: each consumes a fitted model on an input port, and a
fixture written from the JVM cannot carry a live Python object.

Review also turned up two defects that this PR fixes rather than mirrors. SklearnTesting squeezed the feature frame before predicting, so a single feature column, or the single row a drop left behind, reached scikit-learn as one dimension and was rejected. That was the executor's own behaviour and the export had copied it, so both paths pass the frame itself now. Separately, the three HuggingFace text operators guarded a missing cell with is None. Arrow normalizes an empty cell to null, so that held for the executor as long as the column really was a string, but attributeTypeRules is a warning the editor prints rather than a filter, and the exported script reads a frame where a column holding nothing else comes back as float64. Both paths ask pandas now.

Any related issues, documentation, discussions?

Part of #8325, 20 of 27; that issue lists the set in order. It needs #8327 for the trait, so it does not compile until that lands, and the rows these operators add to the verification runner follow with the harness rather than as whole new files here.

Closes #8316, closes #8056.

Closes #8422, the task this change is the whole of.

How was this PR tested?

Each operator asserts the block it emits in its own spec. Once the harness lands, each is also run through the engine and through its generated script, on every configuration its schema offers, and the two answers compared.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @PG1204, @aglinxinyuan, @carloea2
  You can notify them by mentioning @PG1204, @aglinxinyuan, @carloea2 in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.77%. Comparing base (5264df2) to head (1d16b61).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8368      +/-   ##
============================================
- Coverage     92.78%   92.77%   -0.01%     
+ Complexity     4899     4890       -9     
============================================
  Files          1236     1236              
  Lines         52130    52103      -27     
  Branches       6407     6405       -2     
============================================
- Hits          48369    48340      -29     
- Misses         2185     2186       +1     
- Partials       1576     1577       +1     

Flag	Coverage Δ		*Carryforward flag
access-control-service	71.78% <ø> (ø)
agent-service	99.32% <ø> (ø)		Carriedforward from d118784
amber	88.60% <ø> (-0.02%)	⬇️	Carriedforward from d118784
computing-unit-managing-service	55.20% <ø> (ø)
config-service	87.37% <ø> (ø)
file-service	81.53% <ø> (ø)		Carriedforward from d118784
frontend	96.57% <ø> (-0.01%)	⬇️	Carriedforward from d118784
notebook-migration-service	83.73% <ø> (ø)
pyamber	98.48% <ø> (ø)		Carriedforward from d118784
workflow-compiling-service	74.09% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 5264df2 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	522	0.319	18,737/25,031/25,031 us	🔴 -9.8% / 🔴 +61.9%
🔴	bs=100 sw=10 sl=64	1,197	0.731	78,959/114,298/114,298 us	🔴 +18.7% / 🟢 -21.1%
⚪	bs=1000 sw=10 sl=64	1,410	0.861	706,280/778,035/778,035 us	⚪ within ±5% / 🟢 +33.6%

Baseline details

Latest main 5264df2 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	522 tuples/sec	579 tuples/sec	796.29 tuples/sec	-9.8%	-34.4%
bs=10 sw=10 sl=64	MB/s	0.319 MB/s	0.353 MB/s	0.486 MB/s	-9.6%	-34.4%
bs=10 sw=10 sl=64	p50	18,737 us	17,436 us	12,549 us	+7.5%	+49.3%
bs=10 sw=10 sl=64	p95	25,031 us	23,800 us	15,458 us	+5.2%	+61.9%
bs=10 sw=10 sl=64	p99	25,031 us	23,800 us	19,275 us	+5.2%	+29.9%
bs=100 sw=10 sl=64	throughput	1,197 tuples/sec	1,227 tuples/sec	1,027 tuples/sec	-2.4%	+16.6%
bs=100 sw=10 sl=64	MB/s	0.731 MB/s	0.749 MB/s	0.627 MB/s	-2.4%	+16.6%
bs=100 sw=10 sl=64	p50	78,959 us	81,805 us	100,085 us	-3.5%	-21.1%
bs=100 sw=10 sl=64	p95	114,298 us	96,317 us	105,895 us	+18.7%	+7.9%
bs=100 sw=10 sl=64	p99	114,298 us	96,317 us	116,076 us	+18.7%	-1.5%
bs=1000 sw=10 sl=64	throughput	1,410 tuples/sec	1,366 tuples/sec	1,055 tuples/sec	+3.2%	+33.6%
bs=1000 sw=10 sl=64	MB/s	0.861 MB/s	0.833 MB/s	0.644 MB/s	+3.4%	+33.6%
bs=1000 sw=10 sl=64	p50	706,280 us	731,549 us	979,292 us	-3.5%	-27.9%
bs=1000 sw=10 sl=64	p95	778,035 us	785,016 us	1,018,835 us	-0.9%	-23.6%
bs=1000 sw=10 sl=64	p99	778,035 us	785,016 us	1,039,692 us	-0.9%	-25.2%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,382.95,200,128000,522,0.319,18737.39,25030.72,25030.72
1,100,10,64,20,1670.75,2000,1280000,1197,0.731,78958.66,114298.22,114298.22
2,1000,10,64,20,14182.18,20000,12800000,1410,0.861,706279.54,778035.12,778035.12

[carloea2]

Sklearn Testing drops rows with missing values in the engine path, but the standalone code uses all rows. Missing values can make the script fail or produce different scores. Please apply the same filtering before building X and Y, and add a parity test with a missing row.

[kz930]

Fixed. The script drops the rows the executor drops before it builds X and Y. The test pins that a dropped row cannot move the score, since this operator cannot be run against the other path (a JVM fixture cannot carry a fitted model on an input port).

[carloea2]

The generated code now drops the same missing rows as the executor. The test covers it.

[carloea2]

Reviewed alongside the related export and verification PRs. These findings are based on code inspection and focused Python checks, not a full Scala suite run.

[kz930]

@carloea2 the finding is fixed: prediction runs per row, so a row with a missing feature keeps its place with an empty result. Would you take another look when you have a moment?

[carloea2]

Missing feature rows now stay in place with an empty prediction. Complete rows are predicted together. Looks good.

[carloea2]

The sentiment, spam, and summarization exports only check None and blank strings. A missing pandas text value can be NaN or pd.NA, which is passed to the model and fails. Please handle all pandas missing values and add tests.

[carloea2]

Sklearn Testing calls predict on X.squeeze. One feature across several rows or one remaining row across several features becomes one dimensional and sklearn rejects it. Please keep X two dimensional in both paths and test both shapes.

[kz930]

Both paths now ask pd.isna, which catches NaN and pd.NA as well as None. The type rule naming the column string is a warning the editor prints rather than a filter, so the executor can be handed a numeric column carrying its own NaN.

[kz930]

Both paths now hand predict the frame itself rather than squeezing it first. The spec scores a single-feature table and a table with one surviving row, and asserts that neither path still squeezes.

[carloea2]

Sklearn Testing scores only the first model in the exported path and copies that score to every model row. With two classifiers where one predicts 0 and one predicts 1 against labels 0 and 0, native returns accuracy 1.0 and 0.0, but the export returns 1.0 for both. Please score each model row separately and add a two model test.

[kz930]

Fixed in f9835ba: the exported script now predicts once per model row, and the new two model test reads 1.0 and 0.0 where it used to read 1.0 twice. Sklearn Prediction read the first model too, where the executor keeps the last one the port hands it, so that is in the same commit.

[carloea2]

The missing value, input shape, and multiple model issues are fixed. Each model now gets its own scores. I checked the two model case and found no remaining blocker in this review.

[carloea2]

The added reporting matches the native handling of skipped test rows. The previously reviewed model fixes remain intact. Looks good.