Add hosted auto-label support to the SDK and CLI

[lucas-fochesatto]

Exposes the hosted Auto Label endpoints already used by the Roboflow MCP in the Python SDK and CLI, so scripts and agents using roboflow-python can list models, preview, start and track auto-label jobs.

• Workspace.autolabel_models() lists the foundation-model catalog with availability, guidance and credits per image
• Project.autolabel_preview(model, image, ontology=...) runs a free single-image preview; image accepts an HTTPS URL, a local file or base64
• Project.autolabel(batch_id, model, model_type="foundational" | "roboflow", ...) starts a job over a batch and returns jobId / annotationJobId
• Project.autolabel_job(job_id) / Workspace.autolabel_job(job_id) poll per-subjob progress
• CLI: roboflow autolabel models | preview | start | job; ontology via repeated --class or --ontology JSON
• Model ids are passed through as-is (no client-side whitelist); model_type="roboflow" is sent as custom_roboflow with the id in modelOptions.modelId, matching the MCP

Testing

• Unit tests for the adapter contracts, SDK wrappers, shared helpers and CLI (29 new; full suite green)
• Validated end to end against staging (lucas-fochesatto/gemini-autolabel-test): models list, preview with sam3-rle and gpt-6-astra-boxes, jobs started with gpt-6-astra-boxes and gemini-boxes ran to done

🤖 Generated with Claude Code

[iurisilvio]

Reviewed this end to end against the current backend (app/functions on master, plus the queue worker). The shape of the feature is good and the test coverage is unusually thorough for an SDK addition. One blocking correctness issue, which I've pushed three commits to fix, plus a list of smaller things below.

The ontology direction was inverted

This is the one that mattered. The SDK, CLI and docs all specified {"class name": "text prompt"} and passed that dict straight through, but the API's ontology is keyed the other way round: key = prompt, value = class name. Four independent confirmations:

• autoLabelPreviewService.ts maps entries as ([prompt, cls]) => ({class: cls, prompt})
• autoLabelService.normalizeOntology normalizes the array form [{class, prompt}] to acc[prompt] = className
• startAutoLabelJobService.deriveConfidenceThresholds keys its output off Object.values(ontology), with the variable literally named className
• the queue worker does CaptionOntology(data["ontology"]) and class_map = sorted(set(ontology.values()))

So project.autolabel("batch", "sam3-rle", ontology={"cat": "a cat"}) prompted the model with "cat" and wrote every annotation under the class name "a cat" — on a job that spends credits and pollutes the project's class list. The --class cat path builds the identity map {"cat": "cat"}, which is symmetric, which is why the unit tests and the staging run didn't catch it.

Fixed in the three commits I pushed. Read the final state (1dd4b4b), not the intermediate ones — the first two took a different approach (translating to an explicit [{class, prompt}] wire form) before we settled on taking the API's direction directly.

Why prompt-keyed rather than translating

It reads backwards at first, but it's the useful direction rather than an accident of the API: prompts are the unique side, so several of them can collapse onto one output class.

ontology={"kitten": "cat", "tabby": "cat", "puppy": "dog"}

A class-keyed object has room for exactly one prompt per class, since its keys have to be unique — so it can't express the case the ontology exists for. Taking the API's shape directly also removes the translation step and with it this whole class of bug.

A plain list of class names still works and still means "prompt each class with its own name".

The tradeoff, and the reason I'm flagging it rather than just merging: {"cat": "a cat"} is now silently accepted and will label everything as "a cat". The direction lives only in the docstrings and the flag help. If you'd rather keep the class-keyed public API and translate, say so and I'll flip it back — the first two commits show what that looks like.

roboflow/roboflow-mcp#166 carries the same fix for the MCP, which has the identical inversion live in production today.

Worth fixing before merge

A mistyped --image path is silently sent as base64. image_payload falls through to {"type": "base64", "value": image} for anything that's neither an http(s) URL nor an existing file. image_payload("smaple.jpg") returns {'type': 'base64', 'value': 'smaple.jpg'}, and ~ isn't expanded either (unlike _parse_json_flag, which does call expanduser). The user gets a generic inference failure instead of "file not found". roboflow/util/image_utils.py already has check_image_path / validate_image_path.

An unreadable image file escapes as a raw traceback. image_payload(image) runs inside the operation passed to _run, which only catches RoboflowError and ValueError, so an OSError isn't handled. Verified: --image <mode-000 file> exits with an uncaught PermissionError traceback. CLAUDE.md requires structured error output on stderr for every command. Building the payload before _project_command fixes it and fails faster.

autolabel job can't poll a job autolabel start just created. start derives the workspace from the -p ws/proj shorthand via resolve_resource; job uses _resolve_workspace, which only reads --workspace or the account default. So autolabel start -p other-ws/proj ... prints a jobId that autolabel job <id> then looks for in the default workspace — getAutoLabelJobHandler checks job.workspaceId !== req.workspaceId and 404s.

preserveExistingAnnotations isn't exposed, and the server default destroys annotations. buildHostedStartParams does preserveExistingAnnotations: req.body.preserveExistingAnnotations === true, so omitting it means false. In autoLabelService.js:613 the non-preserve path writes the full labels set, versus preserve mode which only adds updateResult.addedLabels. Auto-labeling a batch that already has manually reviewed annotations replaces them, and unlike the web UI there's no toggle. Worth a preserve_existing_annotations param + --preserve-existing flag.

Exit code 1 where the CLI contract says 2. _resolve_workspace re-implements roboflow/cli/_resolver.py:resolve_ws_and_key and diverges from it: with no default workspace, roboflow autolabel models exits 1 while roboflow workflow list exits 2 for the identical condition. CLAUDE.md pins 2 = auth error, so an agent branching on $? == 2 to re-auth mis-handles every autolabel command. _resolve_project likewise re-copies annotation.py:_resolve_project_context verbatim — worth lifting into _resolver.py rather than adding a fourth copy.

Smaller things (8)

• MODEL_TYPES is dead and --model-type is validated too late. It's defined in autolabel_utils.py and referenced nowhere; resolve_model re-hardcodes both strings. Because --model-type is a bare str, a typo is only rejected after workspace and API-key resolution (which can hit the network). Making it a str, Enum built from MODEL_TYPES gets typer to reject it at parse time — matching how the MCP declares the same parameter as Literal["foundational", "roboflow"].
• The models table hides why a model is unavailable. listAutoLabelModelsHandler returns unavailableReason and a human unavailableMessage for every plan-gated entry (gpt-6-astra-boxes and gemini-boxes are both planGated: true). The table shows the default model as available: no with no explanation, so you have to re-run with --json to learn it's a plan gate and not an outage.
• API errors reach the user phrased for raw HTTP callers. _run calls output_api_error(args, exc) with no hint. A bad model id returns Unknown model "x". Call GET /:workspace/autolabel/models for the available model ids. — _translate_api_hints only rewrites messages starting with "You can ... request", so this passes through telling a CLI user to make an HTTP call instead of running roboflow autolabel models.
• --confidence is unbounded despite help text saying 0.0 to 1.0. --confidence 50 (thinking in percent) is accepted and fanned out across every class; the job runs, burns credits, returns nothing. batch.py:76 already uses typer.Option(min=..., max=...).
• _autolabel_response and _annotation_administration_response are both one-line pass-throughs to _json_response_or_raise — three names for one behavior.
• _models duplicates _workspace_command just to pass a text renderer, and the two have already drifted (if not workspace_url vs if api_key is None or workspace_url is None). _run already takes a text param, so forwarding it collapses _models to one call.
• _raise_for_trash_response (rfapi.py:1993) is a third copy of the logic this PR extracts as _json_response_or_raise, and a weaker one: it does msg = body.get("error") with no isinstance(..., dict) check, so a nested {"error": {"message": ...}} stringifies a dict into the message. Since the extraction is happening here anyway, folding it in fixes that path too.
• Docs policy. CLAUDE.md: "CLI-COMMANDS.md in this repo is a quickstart only. The full command reference lives in roboflow-product-docs... When adding commands, update both." Four new commands here, only the quickstart updated — and the product docs will need the ontology direction too.

Everything green after the three commits: 1026 tests, ruff clean, mypy no new errors, and CI is passing on all 15 checks.

[lucas-fochesatto]

Let's keep it prompt-keyed, same as the MCP. Thanks for catching the inversion.

The five merge-blocking items are fixed in 3d4a4f3 (CI green).

[iurisilvio]

Re-reviewed 3d4a4f3. All five merge-blocking items are properly fixed, and I verified each one rather than reading for it:

• roboflow autolabel models now exits 2 with no default workspace, same as workflow list. Contract restored.
• An unreadable --image now exits 1 with {"error": {"message": "Cannot read image ...: Permission denied", ...}} on stderr and no exception. No traceback.
• job job-1 -p other-ws/proj calls the API with other-ws; omitting -p still uses the default workspace, so nothing existing breaks.
• --preserve-existing sends preserveExistingAnnotations: true, and the key is omitted entirely when the flag is absent.
• smaple.jpg is rejected before credentials are resolved.

1038 tests (up from 1026), ruff clean, mypy no new errors. Lifting resolve_project_context into _resolver.py rather than just calling the existing resolve_ws_and_key was the right call — that removed the third copy in annotation.py too, which I'd flagged but hadn't asked you to fix. The new tests are well aimed; mock_resolve.assert_not_called() in the mistyped-path test locks down the "fails before any network call" property and not just the error message.

Three small things left over from this commit. None are blocking.

_is_base64 still lets some mistyped paths through. The dotted cases are all fixed, but / is in the base64 alphabet, so an extensionless path whose length happens to be a multiple of 4 still validates:

rejected   'smaple.jpg'   'cat.jpeg'   './img/cat.png'   '~/nope.jpg'   'img/cat'
ACCEPTED   'data/img'  →  base64:data/img
ACCEPTED   'photos/a'  →  base64:photos/a
ACCEPTED   'test'      →  base64:test

img/cat gets rejected and data/img doesn't, purely on length parity, which is a confusing thing to explain to whoever hits it. A minimum length is a sturdier discriminator than alphabet validity here: the smallest real image is still hundreds of base64 characters, so something like len(compact) >= 64 before the _is_base64 check would close all of these without touching the legitimate path.

Unpadded base64 is now rejected, and data URIs get a misleading message. b64decode(validate=True) requires padding, so iVBORw0KGgo fails where iVBORw0KGgo= passes. That's a narrowing of an input the docstring still advertises. And data:image/jpeg;base64,... — a plausible thing to paste — now fails with "Image file not found", which sends the user looking for a file that was never meant to exist. Stripping a data:...;base64, prefix, or just distinguishing "not valid base64" from "file not found" in the message, would cover both.

Project.autolabel_preview gained exceptions its docstring doesn't mention. image_payload documents its Raises:, but SDK callers read the Project method, and that one still says only "image: HTTPS URL, local file path, or base64-encoded image". It can now raise ValueError and OSError, which is a behavior change for anyone already calling it. Worth a Raises: block there.

Also, annotation.py:_resolve_project_context is now a one-line pass-through to the shared function. Same pattern as the _autolabel_response alias I mentioned before — the call sites can import resolve_project_context directly and the wrapper can go.

The seven other minor items from my first pass are untouched, which is fine — none of them block. The one I'd still like before this ships is --confidence bounds, since --confidence 50 silently burns credits for zero detections and min=/max= is a one-line typer change.

Nothing here blocks a merge from my side. Ship it and take the leftovers in a follow-up if you'd rather.

[iurisilvio]

Note on the preview docstring, expanded in my approval below.

[iurisilvio]

Not totally true, we charge for third-party APIs.

[iurisilvio]

Approving. The five merge-blocking items are fixed and verified, CI is green, and nothing outstanding is worth holding the PR for.

For the record, I pushed three commits to this branch myself (the ontology direction fix), so treat my approval as covering Lucas's work rather than my own.

On my inline comment about the "free" wording — it's right, and the claim is in four places, not one: rfapi.py:1367, project.py:1171, the preview command docstring in handlers/autolabel.py:42, and CLI-COMMANDS.md:257. gpt-6-astra-boxes and gemini-boxes are both planGated: true, and a preview of either runs real inference through workflowsAdapter.runPreview on the managed provider key, which is third-party spend Roboflow pays and passes through. What's actually true is narrower: no auto-label job is created and no auto-label credits are deducted. Worth rewording to that in all four spots rather than dropping the sentence, since "no job is created" is the part users need.

Everything else from my last two comments is non-blocking and we'll pick it up in a follow-up:

• _is_base64 still accepts an extensionless path whose length is a multiple of 4 (data/img, photos/a). A minimum length before the base64 check closes it.
• Unpadded base64 is rejected, and a data:...;base64, URI reports "Image file not found".
• Project.autolabel_preview needs a Raises: block for the new ValueError / OSError.
• annotation.py:_resolve_project_context is now a one-line pass-through and can go.
• --confidence has no min/max, so --confidence 50 burns credits for zero detections.
• The rest of the minor list: MODEL_TYPES unused and --model-type not a typer enum, unavailableReason dropped from the models table, output_api_error called with no CLI-shaped hint, the _autolabel_response / _annotation_administration_response aliases, and _raise_for_trash_response as a third copy of _json_response_or_raise.

Two that outlive this PR and shouldn't get lost in it:

roboflow-product-docs still needs the new command group, per the docs policy in CLAUDE.md, including the prompt-keyed ontology direction. CLI-COMMANDS.md is only the quickstart.

The same ontology inversion is live in production in the MCP right now and doesn't depend on this merging. roboflow/roboflow-mcp#166 fixes it and is ready for review.

Nice work on the fixes, especially lifting resolve_project_context into _resolver.py instead of just wiring up the existing helper.