feat(grpc-web): Pyodide/WASM grpc-web transport for the async client

[g-despot]

What & why

Lets the async client run inside Pyodide/WebAssembly (marimo notebooks, browser, WASM workers), where grpcio has no wheel and sockets don't exist. gRPC is re-routed over grpc-web (fetch), REST over the browser's fetch — against Weaviate core's native /v1/grpc-web endpoint (default-on since 1.38.3).

This is the trimmed successor of #2056 (kept open for reference): only what the WASM goal actually needs. The general fixes that were bundled there now ship separately:

• fix(connect): token refresh lifecycle, backoff and error-taxonomy follow-ups #2140 — token refresh lifecycle fixes. This PR is stacked on it (base branch fix/token-refresh-lifecycle): the async OIDC refresher must be an asyncio task because threads cannot start under Pyodide.
• fix(batch): surface background stream failures on exit instead of returning partial results #2141 — batch stream exit semantics.

Quickstart (Pyodide / marimo)

import micropip
await micropip.install("weaviate-client[grpc-web]")  # until release: wheels built from this branch

import weaviate  # a bare import bootstraps weaviate_client_web under Emscripten

client = weaviate.use_async_with_custom(
    http_host="weaviate.example.com", http_port=443, http_secure=True,
    grpc_host="weaviate.example.com", grpc_port=443, grpc_secure=True,
)
await client.connect()

Under Emscripten all three async helpers (use_async_with_local / _weaviate_cloud / _custom) pin gRPC to the REST endpoint under /v1/grpc-web themselves — the same contract as the TS @weaviate/web client's webify(). There is no grpc_path_prefix parameter on the helpers: pass gRPC arguments equal to the HTTP ones; anything else is discarded with a Con006 warning. A grpc-web transcoder on a separate endpoint needs hand-built ConnectionParams(..., grpc_path_prefix=...). Off Emscripten nothing changes: the helpers build exactly the params they always did (pinned by test_helper_params_off_emscripten_are_unchanged).

Key pieces

packages/web/ — companion distribution weaviate-client-web, defined for its one environment: it imports pyodide at module scope and is only importable under Emscripten (no CPython seams — no force= installs, no CPython sender). _shim.py (pure-Python grpc module shim), _channel.py (GrpcWebChannel: unary calls, metadata → headers with transport-owned protocol fields, grpc-web status → AioRpcError, grpc-timeout encoding, 404/405 diagnostics), _framing.py (single uncompressed trailer enforced), _httpx_fetch.py (REST over JS fetch), README. Versioned in lockstep with weaviate-client (setuptools_scm from the same git tag) and pinned to it with == at build time, so mismatched pairs cannot resolve.

Base client (no-ops off Emscripten):

• weaviate/__init__.py — platform-guarded bootstrap: a bare import weaviate soft-imports the companion; a missing companion raises an install hint.
• setup.cfg — grpcio ; sys_platform != "emscripten"; the grpc-web extra (weaviate-client-web ; sys_platform == "emscripten"), a no-op on CPython.
• connect/base.py — grpc_path_prefix on ConnectionParams (port collision allowed with a prefix); _check_grpc_web_usable runs once, at client construction (sync client or missing shim fails fast); grpc-web.path_prefix channel option for the shim.
• connect/helpers.py — _webify(): the async helpers route gRPC to the REST endpoint under Emscripten; Con006 when a caller's gRPC endpoint is discarded.
• connect/v4.py — sync client rejected at construction under Emscripten (a 3-line check in _ConnectionBase.__init__); _ping_grpc passes the gRPC error through so the diagnostics can say what went wrong; _check_package_version ignores OSError (fetch blocked by a page CSP).
• exceptions.py — WeaviateGRPCUnavailableError gains the gRPC code/details and a grpc-web branch (no firewall/port advice; a 404 names the two real causes: server < 1.38.3 or a wrong prefix).
• collections/batch/async_.py — batch.stream() fails fast under grpc-web, pointing to insert_many() (bidi streaming is impossible over fetch).
• embedded.py — explicit error under Emscripten. proto/v1/__init__.py — grpcio version fallback when dist metadata is absent (Emscripten only), drift-pinned by proto_test.

Dropped vs #2056 (and why)

• The _deadline / _Deadlines plumbing across v4.py and both batch modules: the web package already treats non-finite timeouts as "no deadline" in both transports (_encode_timeout, _abort_signal_ms); the one CPython effect (Thread.join(inf) in the sync batch-stream shutdown wait) is a separate one-line fix, not included here.
• The ConnectionSync.__init__ signature override (→ 3 lines in the base constructor) and the duplicate channel-level _check_grpc_web_usable call.
• Tests that pinned nothing or duplicated each other (test_no_helper_takes_a_grpc_path_prefix, the Emscripten __get_timeout test, the channel-level reject pair, the trivial platform-passes tests, …).

CI / tests

The package suite runs as pytest inside Pyodide (pyodide-e2e job: node --experimental-wasm-jspi ci/pyodide-e2e/units.mjs — async tests execute via JSPI stack switching; a missing flag fails loudly), followed by the e2e suite against core-native /v1/grpc-web (Weaviate 1.39.0). There is deliberately no CPython matrix for the package: it only ever executes under Pyodide, whose bundle pins the interpreter (currently 3.14). requires-python >= 3.10 stays: older Pyodide bundles ship 3.10–3.13 interpreters and the package is pure Python. The base client's import-hook branches run as subprocess tests in test/test_wasm_compat.py on the full 3.10–3.14 matrix. ruff / flake8 / pyright cover packages/web; build-and-publish builds, version-asserts and publishes both packages (the companion wheel-only). Locally: 139 pytest cases in Pyodide + bootstrap scenario; 36 in test_wasm_compat/test_connection_params; 4 in proto_test; pyright clean for the whole project.

Remaining before release

Lockstep versioning, the [grpc-web] extra and dual publishing are done. One manual check: PYPI_API_TOKEN must be able to create the new weaviate-client-web project on first publish (a project-scoped token cannot — use an account-scoped token or register a pending publisher on PyPI first).

🤖 Generated with Claude Code

https://claude.ai/code/session_01GUNU7QgDr9MmFZnjKY9zFN
https://claude.ai/code/session_01Caze9m6PBSfYkt77mMKj2b

[orca-security-eu]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Secrets	0   0   0   0	View in Orca

[Copilot]

Pull request overview

Adds Pyodide/WASM support to the async client through a companion grpc-web and browser-fetch transport.

Changes:

• Adds the weaviate-client-web grpc-web transport and shim.
• Integrates automatic WASM routing, diagnostics, and unsupported-feature guards.
• Adds unit and Pyodide end-to-end CI coverage.

Reviewed changes

Copilot reviewed 32 out of 34 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
weaviate/warnings.py	Adds endpoint-override warning.
weaviate/proto/v1/__init__.py	Adds Emscripten grpcio fallback.
weaviate/exceptions.py	Adds grpc-web diagnostics.
weaviate/embedded.py	Rejects embedded mode under WASM.
weaviate/connect/v4.py	Integrates grpc-web lifecycle and errors.
weaviate/connect/helpers.py	Routes async helpers through grpc-web.
weaviate/connect/base.py	Adds grpc-web connection parameters.
weaviate/collections/batch/async_.py	Rejects streaming batches over grpc-web.
weaviate/__init__.py	Bootstraps the companion package.
test/test_wasm_compat.py	Tests WASM-specific behavior.
test/test_connection_params.py	Tests grpc-web connection configuration.
setup.cfg	Excludes grpcio under Emscripten.
pyrightconfig.json	Type-checks the web package.
proto_test/test_proto.py	Tests grpcio fallback compatibility.
packages/web/tests/test_transport.py	Tests grpc-web transport behavior.
packages/web/tests/test_single_import.py	Tests automatic bootstrap.
packages/web/tests/test_shim_install.py	Tests shim installation.
packages/web/tests/test_httpx_fetch.py	Tests fetch-based REST transport.
packages/web/tests/test_framing.py	Tests grpc-web framing.
packages/web/tests/conftest.py	Configures package test imports.
packages/web/src/weaviate_client_web/py.typed	Marks the package as typed.
packages/web/src/weaviate_client_web/_shim.py	Implements the grpc API shim.
packages/web/src/weaviate_client_web/_sender.py	Implements HTTP senders.
packages/web/src/weaviate_client_web/_httpx_fetch.py	Implements fetch-backed HTTPX transport.
packages/web/src/weaviate_client_web/_framing.py	Implements grpc-web framing.
packages/web/src/weaviate_client_web/_channel.py	Implements the grpc-web channel.
packages/web/src/weaviate_client_web/__init__.py	Exposes and bootstraps the package.
packages/web/README.md	Documents usage and limitations.
packages/web/pyproject.toml	Defines companion package metadata.
ci/pyodide-e2e/run.mjs	Runs tests inside Pyodide.
ci/pyodide-e2e/package.json	Pins the Pyodide runtime.
ci/pyodide-e2e/e2e.py	Exercises WASM client workflows.
.gitignore	Ignores web-package build artifacts.
.github/workflows/main.yaml	Adds grpc-web and Pyodide CI jobs.

Suppressed comments (1)

weaviate/connect/helpers.py:538

• This value comparison cannot tell an omitted argument from an explicitly supplied grpc_port=50051 (which the helper's own examples use). Under WASM that explicitly requested endpoint is still replaced by the REST port, but Con006 is suppressed, contrary to the documented promise that caller-supplied ports are warned about. Preserve argument presence with a sentinel default, then resolve the omitted value to 50051.

            grpc_chosen_by_caller=grpc_port != _LOCAL_GRPC_PORT_DEFAULT,

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[tsmith023]

Why is this import within the function instead of at the top of the file?

[tsmith023]

Ah I see it's so that the import is lazy

This makes me wonder, should we instead provide grpc-web functionality using the extras syntax, i.e.

pip install weaviate-client[grpc-web]

so that pyodide and cpython users don't conflict?

The default installation pip install weaviate-client would ship as it is right now and the extras would bundle this extra weaviate_client_web dep into it. wdyt?

[g-despot]

Extras can only add dependencies, so [grpc-web] would just make the extra pull in weaviate-client-web as a dependency, it can't drop grpcio and it can't change behavior in code (that's the check sys.platform == "emscripten" inside the base package). The only new dep for the web packager is anyio, which is already gated with the emscripten check.

The emscripten check already keeps CPython and Pyodide installs from conflicting. Once published, users would install it with micropip.install('weaviate-client-web'). We can theoretically add the extras logic, but functionally it wouldn't add much benefit and the current separation logic in the code would remain the same.
The question is how we want users to install it and if we are managing two packages in the same repo, as I see it the options are:

1. Keep the separate package as is and install with micropip.install('weaviate-client-web')
2. Keep the separate package as is and install with micropip.install('weaviate-client[grpc-web]')
3. Move the web logic to the base package as one package so we have only one that installs with micropip.install('weaviate-client').

[tsmith023]

Agreed, my point here is that weaviate_client_web dep should not be added at all when calling pip install weaviate-client so it should be safe to import from pyodide.http import pyfetch from the top of the file here instead of inside the function body since these files won't be called at all when the dep is absent

Agreed that we need the emscripten check when running in grpc-web-mode but it should be conditional on the presence of the grpc-web dep. This point is in relation to the docstring comment:

"""
Imports ``pyodide.http`` lazily so this module stays importable on CPython (where ``pyodide`` does not exist)
"""

The logical implementation of the grpc-web functionality shouldn't have an import hack applied to it because it may be imported into an environment that doesn't support it. Instead, the weavate_client_web package should be defined coherently and consistently to be used in its respective environment

By requiring users do pip install weaviate-client[grpc-web] when they're running in emscripten, they guarantee that their client will have the necessary dependencies installed. Then, if they run pip install weaviate-client, the weaviate_client_web dep is never installed since it is not relevant thereby allowing the package itself to be defined coherently without worrying about whether it can run in cpython

wdyt?

[g-despot]

Agreed and it's implemented now. Just a notice, with the package importable only under Emscripten, its unit tests can't run as CPython pytest anymore.

[orca-security-eu]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	SAST	0   0   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca

[Copilot]

🟡 Changes recommended

Several transport edge cases can produce malformed requests, incorrect diagnostics, or disabled timeouts, and the stated CI matrix is absent.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 33/35 changed files
• Comments generated: 5
• Review effort level: Balanced

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.32432% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.77%. Comparing base (95b5d76) to head (1e9ad57).
⚠️ Report is 158 commits behind head on main.

Files with missing lines	Patch %	Lines
weaviate/exceptions.py	90.90%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2142      +/-   ##
==========================================
+ Coverage   86.64%   88.77%   +2.13%     
==========================================
  Files         300      306       +6     
  Lines       23172    24077     +905     
==========================================
+ Hits        20077    21374    +1297     
+ Misses       3095     2703     -392     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.