cabi: compound element types recompute layout per element and per field — ~5 µs/element for a stream<variant-of-records>; 2.6x from memoizing four pure functions

[lannbot]

Found while evaluating an explicit-WIT-schema alternative to a hand-rolled byte protocol in a consumer (polyengine-dioxus, which streams DOM mutations guest→host). The schema is a variant operation with 17 cases over records — several carrying string, option<u16>, list<u8>, and a nested variant — delivered as stream<operation>.

A stream<operation> element lifts at ~5.0 µs. The same batch as stream<u8> decodes at ~76 ns/op. Linear in element count across a 100→4000-row sweep (5225 / 4920 / 4310 / 4266 ns per element), so this is a per-element constant factor, not a size effect.

Most of it is layout recomputation the runtime already has enough information to cache. Memoizing four functions by type identity — 71 lines, no contract change, no emission machinery — takes the same workload from 4981 → 1909 ns/element (2.6x) in one sitting, and end-to-end in the consumer's benchmark from 5.07x → 1.74x slower than the byte channel on its heaviest operation.

Root cause

loadListFromValidRange (runtime/src/cabi/load.ts:127) hoists elemSize out of the loop, but everything below it recomputes from scratch, per element and per field:

• loadVariant (load.ts:169) calls maxCaseAlignment(cases) (layout.ts:80) on every element. That walks all 17 cases, and alignment on each case recurses through its record's fields into nested variants and options. One full type-tree walk per element.
• loadRecord (load.ts:153) calls both alignment(field.type) and elemSize(field.type) per field, per element — each an independent recursive walk of that field's subtree.
• despecialize (types.ts:267) allocates on every call for option/tuple/enum/result — a fresh {kind:"variant", cases:[…]} object. It sits at the top of load, alignment and elemSize, so every option<u16> field allocates several times per element just to be measured.
• discriminantType (types.ts:306) allocates a fresh {kind:"u8"} per call, and it is called from inside loadVariant, alignmentVariant and elemSizeVariant.

None of this is data-dependent. alignment/elemSize are pure functions of (ValType, PtrType); the descriptor's type objects are stable for the lifetime of a plan.

Second contributor, in the embedder adapter rather than the CABI: toHost (runtime/src/embedder/values.ts) calls camelCase(f.label) — a split/map/join, allocating — per field per element (:193, :202), and resolves the variant case by linear scan over all 17 (:212, :427). (Its checkNoCollisions is correctly memoised per type and is not a per-element cost.)

Measurements

Box: linux-arm64 dev box, Deno 2.9.5, guest wit-bindgen 0.60, polyengine @ 9e17dc9. One batch of 8009 operations (a synthetic create-1000-rows render), decoded into the same OpSink both ways, no DOM in the loop. Box-relative; read the ratio.

path	ms (median of 9)	ns/element
stream<u8> readDirect + hand-written decode	0.610	76
stream<u8> read + hand-written decode	0.469	59
stream<operation> read + switch on the lifted value	39.890	4981
(guest-side batch construction, byte encoder)	0.548	68
(guest-side batch construction, typed encoder)	0.339	42

Guest-side construction is cheaper for the typed encoder — pushing structs beats byte-serialising them — so the cost is entirely in lowering, lifting and adapting.

Attribution, with the layout memoization already applied (i.e. these are the residual shares, and cabi/load is still the majority after the cheap fix is in):

stage	share
per-element load() in runtime/src/cabi/	~64%
toHost adaptation in runtime/src/embedder/values.ts	~20%
rendezvous, guest-side lowering, promise plumbing	~10%
the consumer's own switch over the lifted values	~1–4%

The cheap fix, measured

WeakMap caches keyed by type identity on alignment, elemSize, maxCaseAlignment and despecialize (plus module-level singletons for discriminantType's three return values). 71 lines across runtime/src/cabi/layout.ts and runtime/src/cabi/types.ts; nothing else touched.

workload	stock	memoized
stream<operation>, 8009 elements (above)	4981 ns/el	1909 ns/el	2.6x
consumer bench create-10k (90 006 ops) vs its byte channel	5.07x slower	1.74x slower
consumer bench clear (10 002 ops) vs its byte channel	12.18x slower	2.08x slower

(A second sitting on a noisier box measured 5426 → 1464 ns/el, 3.7x. Treat the range as 2.5–3.5x, not the third digit.)

runtime's suite passes unchanged with it: 661 passed, 0 failed, 24 ignored — including tests/conventions/ (the golden host-ABI transcripts, which is what would catch an observable change), layout_flatten_test.ts, bulk_list_test.ts, nan_test.ts, values_test.ts. The corpus lane was not run (the third_party/component-model submodule is not checked out in my dep checkout), so treat that as unverified.

The soundness condition is that despecialized results are never mutated by callers, since the memo returns a shared object where each call previously got a fresh one. I checked all 23 despecialize( call sites in runtime/src: every one reads .kind/.fields/.cases and none mutates. Worth making that an explicit invariant in the doc comment (or freezing the cached value) rather than leaving it implicit, if this is taken.

I have the patch and can open a PR if wanted — happy for someone to write it differently, the numbers are the point.

Why it stayed invisible

bench/boundary's shape table is list<u8>, u32 and stream<u8> — every payload is a flat scalar type, which is exactly the set #63/#67 moved onto bulk paths. There is no compound-element-type shape anywhere in the instrument. A stream<record-with-strings> or a list<variant> lane would have shown this; it is the natural companion row to #68's stream shapes.

Scope, against the neighbouring issues

• cabi: bulk copies for the remaining flat element types (list<u16|u32|u64|f32|f64|...>, and their streams) #67 deliberately covers flat element types and its checklist stops there; compound element types stay on the interpreted per-element path by design. This is the constant factor of that path, and the memoization is orthogonal to cabi: bulk copies for the remaining flat element types (list<u16|u32|u64|f32|f64|...>, and their streams) #67's two open policy calls (host value shape, endianness) — it needs neither.
• P1: emitted specialized-JS host-boundary executor (AOT-shaped, no eval) #8 is the real answer: a per-type compiled lift removes the interpretation, not just the redundant layout walks. This is evidence for it and a datapoint for its cost ledger — note the attribution table above says ~64% of the post-memoization cost is still interpreted load(), so the cheap fix does not remove P1: emitted specialized-JS host-boundary executor (AOT-shaped, no eval) #8's motivation, it just stops the runtime paying for the same arithmetic tens of thousands of times per batch. P1: emitted specialized-JS host-boundary executor (AOT-shaped, no eval) #8 is gated on "a measured gap from Perf baseline vs the jco legs (and consumer case-budget fit) #17, not the calendar"; this is a measured gap from a real consumer, on a shape Perf baseline vs the jco legs (and consumer case-budget fit) #17's lanes do not cover.
• cabi: host->guest list<u8> lift is per-element (~22 MB/s ceiling); guest->host lower is bulk (~6.4 GB/s) #54 is the same species of finding one level down (per-element where bulk was possible), and the same fix shape: notice the loop is recomputing something invariant.

There is also a floor worth recording for #8's ledger: reconstructing the lifted {kind, value} object graph in plain JS — no runtime, no ABI, just allocating the objects an embedder must hand out and walking them — costs ~190 ns/element against ~45 ns/element to decode the equivalent bytes. A compiled lift can approach that; it cannot go below it, because the host value shape mandates the allocations. For consumers with tens of thousands of elements per batch that gap is the honest cost of a typed boundary, and it is worth having in the contract's performance notes so consumers can design around it rather than discover it.

Checklist for whoever takes this

• memoize alignment / elemSize / maxCaseAlignment by (ValType, PtrType) identity; despecialize by input identity; singleton discriminantType returns
• state (and ideally enforce) the "despecialized results are read-only" invariant the despecialize memo relies on
• hoist toHost's per-field camelCase and per-element variant-case lookup to per-type tables
• add a compound-element-type lane to bench/boundary — a stream<T> and a list<T> where T is a variant of records with a string — so this has a regression instrument
• re-run the corpus lane, which I could not

[lannbot]

Follow-up: the memoization I proposed above is the wrong fix — it is worth ~2.6x, and closure specialization is worth ~17x for the same constraint set (no eval, no emission, no build step, no CSP question). Correcting my own filing, with measurements.

I hooked GuestBuffer.read (runtime/src/task/streams.ts:143) so several lift strategies run against the same guest memory at the same rendezvous, interleaved in one timed loop so no strategy is measured under different JIT/GC state. Same box, same batch (8009 operation elements), medians of 11 after 2 warmups, three independent runs.

strategy	ns/element	vs byte protocol
1. interpreted load() + toHost — today	~4080	~300x
2. compiled lift, identical JS values	~232	~17x
3. compiled visitor, operands as arguments	~127	~9x
4. (3) + number-domain string arithmetic	~135	~9x
5. the consumer's hand-written byte decoder, same sink	~14	1.0x

Correctness first, since a fast wrong answer is worthless. Every run asserts, over all 8009 elements: the compiled lift's output is JSON-identical to toHost(loadListFromValidRange(…)) element by element (all 17 variant cases, including the nested list<variant>, option<u16> present and absent, and list<u8>); and both visitors agree with an independent implementation (the consumer's own applier walking the compiled values) on call count, operand checksum and total string length. The compiled readers keep the interpreter's traps — invalid discriminant, list too long, misaligned list pointer, list and string out of bounds. Adding those traps cost nothing measurable.

2 — compile the descriptor into closures, once per type

This is just specialising the interpreter. Walk the ValType once and build a tree of closures (mem, view, ptr) => value; every call site inside it is monomorphic and every offset is a constant the JIT can fold. That collapses, at compile time rather than per element:

• despecialize / alignment / elemSize / maxCaseAlignment — the subject of the original report
• record field offsets and their camelCase(label) strings
• the variant case lookup, from t.cases.find(…) over 17 cases to an array index by discriminant
• load()'s own switch dispatch, from a megamorphic recursive walk to a direct call

Nothing about the contract, the host value shape, or CSP changes. It is not #8: no emitted modules, no AOT step, no cache directory, no new Function. Just closures built at plan-load time. ~17x, and it makes the memoization in the original report redundant — memoization removes the redundant arithmetic, closure specialization removes the interpretation, and the second subsumes the first.

I would put this ahead of #8 rather than inside it: it is small, it needs none of #8's rescoped delivery machinery, and it changes the shape of #8's remaining case (see below).

3 — a visitor-shaped read for streams and lists

A further ~1.8x on top of (2), and this one is a contract addition.

The remaining cost after (2) is almost entirely allocation the consumer did not ask for: today every element materialises a {kind, value} wrapper plus a payload record, and the consumer immediately destructures and discards both. contracts/embedder-api.md's value mapping mandates those objects, so a lift cannot skip them.

The alternative is to hand the operands over as arguments rather than as an object — the shape a consumer with a switch over cases already wants:

// instead of: for (const op of await stream.read(max)) switch (op.kind) { … }
await stream.readVisit({
  "set-attribute": (id, name, ns, value) => applier.setAttribute(id, name, ns, value),
  "create-text-node": (id, text) => applier.createTextNode(id, text),
  …
});

The runtime already knows each case's payload layout, so the dispatch is a compiled sink[case](read0(p), read1(p), …). Zero allocation per element except the strings the consumer actually asked for. This is the same instinct as amendment A21's readDirect — let the consumer consume in place instead of materialising a chunk — generalised past stream<u8> to any element type, and it is what A21 could offer typed streams without exposing raw guest memory or the ABI layout to embedders.

For a consumer whose batch is tens of thousands of elements this is the difference between allocating ~160k throwaway objects per render and allocating none.

What is left, and why it is not lift

The gap from (3) to the byte protocol (~9x here) is layout, not interpretation: the lowered list<operation> is 24 bytes/element fixed-stride (~192 KB for this batch) with strings, list<u8> paths and list<template-node> all out of line, against a 102 KB packed variable-length frame decoded with one TextDecoder pass over a single contiguous string segment and substring per operand. The typed path chases pointers and calls TextDecoder per string. That part is inherent to the canonical ABI's memory layout and no runtime change removes it.

Note (5) is a tight-loop best case — the same 102 KB buffer re-decoded 13 times, entirely cache-resident — so treat ~9x as the pessimistic end of the range rather than a number to quote. The consumer-level number is more useful: end-to-end, today's typed channel costs ~4.9 µs/op against its byte channel, of which ~4.08 µs is this lift. Substituting (3) leaves ~0.9 µs/op, which would take that consumer's create-10k from 5.8x to roughly 1.9x. That residual is not lift — it is guest-side lowering (a String per dynamic operand, a Vec per list) plus the rendezvous copy plus the absence of readDirect for typed streams. Whoever picks this up should measure the lowering side before assuming the lift is the whole story; I have not.

Two smaller things noticed along the way

• loadStringFromRange (runtime/src/cabi/strings.ts:94) does its entire length/bounds ladder in BigInt — BigInt(taggedCodeUnits), BigInt(ptr), BigInt(MAX_STRING_BYTE_LENGTH), the tag mask — on every string, for every memory. It is only needed for memory64; on a 32-bit memory every quantity fits in a Number. I measured a variant with a ptrSize() === 4 fast path (row 4) and it came out inside the noise for this workload (~3 strings per 8 elements), so I am not claiming a win — but it is free to do and would matter for a string-dense element type. loadListFromRange (load.ts:110) has the same BigInt ladder per list.
• bench/boundary has no compound-element-type lane, which is why this was invisible; that checklist item from the original report stands, and rows 1-5 above are roughly the shape it wants.

Happy to open a separate issue for the visitor API if it is worth discussing on its own — it is a contract change and does not belong buried in a cabi perf issue. The closure-specialization result belongs here, since it supersedes what I originally proposed.