feat: a double-blind eval demo on a Tinfoil enclave

[koenvanderveen]

Stacked on #9515. Two Colab notebooks that rebuild the tinfoilsh/double-blind-eval flow over PySyft, with no dbe involved. A benchmark owner and a model owner each upload half of an evaluation, both attest the enclave against a pinned image digest, both approve the job, and only the benchmark owner reads the results. No receipt step.

notebooks/enclave/double-blind-eval/

File	Party
1. DO-benchmark-owner-dbe.ipynb	uploads the prompts, submits the job, reads the results
2. DO-model-owner-dbe.ipynb	uploads the adapter, approves the job, sees no results
SETUP.md	the operator side: reset, deploy, run, republish

Two decisions worth reviewing

The benchmark owner submits the job, not the model owner. distribute_results always forwards output to job.submitted_by, and share_results_with_do=True forwards to every data owner named in datasets, so there is no way to have the model owner submit without also receiving. Moving the submit step gives the asymmetry the demo is about, and matches dbe's output policy. The model owner approves the run and its copy of the job never reaches done.

The model runs on CPU. tinfoil-config.yml sets gpus: 0, so the demo runs a TinyLlama-1.1B adapter and caps generation at 32 new tokens to stay inside the 600s job timeout. The model owner notebook carries a commented ADAPTER_REPO line for the gemma-4-31B adapter dbe uses; switching to it needs gpus: raised, which is a measured config change.

Also here

• OpenMined/double-blind-eval-bench, a new public repo holding the AILuminate prompt sample the benchmark owner curls. Its CSVs are well-formed, so nothing downstream repairs them.
• The notebooks pin TINFOIL_TAG=v0.1.14 and the digest that release pins, so attest_peer is called the way this branch now requires: with the image digest, the data owners and the enclave email all set.
• transformers is pinned to 4.57.6. Unpinned, the job venv resolved 5.x, whose import chain fails.

Verification

• The job code, extracted from the checked-in notebook, ran end to end through the in-memory enclave: dual-approval gate held (pending after one approval), 5 prompts evaluated with completions and timings, benchmark owner got the output, model owner got none.
• Every code cell compiles, and the model card builds against both the small adapter and dbe's.
• pre-commit passes on all four files.

Attestation is not verified against a live enclave — nothing is deployed at syft-enclave.openmined.containers.tinfoil.dev right now. The policy call is the shape docs/tinfoil_deployment.md and scripts/tinfoil_e2e_check.py use.

🤖 Generated with Claude Code

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[koenvanderveen]

Tested end to end against a live Tinfoil enclave: deployed published v0.1.14, ran both notebooks' own code cells with only the Colab login swapped for a token login, then deleted the container. It passed — real completions from the adapter inside the enclave, the benchmark owner collecting them, the model owner left with [].

Five fixes came out of it, three of which nothing but a live run would have caught.

The job failed twice in the enclave before it ran.

1. No space left on device. Plain torch on Linux drags in the whole CUDA stack — several GB — which overran the enclave's ramdisk. The enclave has gpus: 0, so none of it could ever be used. Now pinned to the CPU-only wheel by URL (176 MB), with a marker so the notebook still runs off Linux. Resolving that list for linux/cp312 pulls zero nvidia packages.
2. rc=137, Killed — the OOM killer. float32 weights peak at ~4.4 GB, on top of a venv and a 2.2 GB model download that both live in the same 8 GB of RAM. Now dtype=torch.bfloat16 with low_cpu_mem_usage=True (which needs accelerate, added), and MAX_NEW_TOKENS down from 32 to 16. Measured 2.48 GB peak against 4.36 GB.

Reading the model card could never have worked. SyftEnclaveClient.datasets returns self._rds.dataset_manager directly (client.py:218), bypassing the RDS property that syncs — unlike jobs and peers, which both go through theirs. So step 4 pulled nothing however long you re-ran it. The cell now syncs explicitly.

Step 7 asked you to keep waiting for a job that had already failed. It now reports a failed job and prints its stderr — which is how I diagnosed both failures above.

verify_tinfoil.py could not verify anything. It never passed the host into the evidence or the policy, so the appraisal raised no host to reach it on before running a single check, and main() swallowed the error: exit 1, zero output. Fixed, and the tinfoil-verify recipe now installs the optional extra its own comment says it needs. It passes all ten checks against a live enclave.

What the run proved

• Attestation from the notebook's own cell, both parties: all ten checks green, including image digest, data owners and enclave email, with the key bundle bound to the report. It resolved the host from the Drive-published evidence, so no host= needed.
• The dual-approval gate: both parties saw status=pending after the first approval, approved only after the second.
• Output asymmetry: benchmark owner got dbe_results.json; the model owner's copy stayed at approved with no output files and never reached done.
• The 554-character prompt arrived intact, so the bench repo's re-quoting holds over the wire.
• Job runtime 2 min 17 s inside the enclave (received_at → completed_at), at 2.0–5.4 tok/s. SETUP.md's estimate was four to six minutes; corrected to the measured figure.

Emails now come from credentials/CLAUDE.md (enclave@, benchmark_owner@, model_owner@openmined.org) instead of my placeholders.

One thing left for you to decide. I fixed the sync in the notebook rather than in SyftEnclaveClient.datasets, since changing that property affects every enclave user. But the asymmetry is a trap — jobs and peers sync on access and datasets does not. notebooks/enclave/gemma/colab has the same bare benchmark_owner.datasets call; it only displays a list so it never raises, but it can show stale data. Worth fixing at the source in a separate PR.

Also worth knowing: just delete-syftbox failed with a transient Drive error on roughly one call in four and worked on retry every time.