Support distributed processing by pickling expressions - #1543

Closed

timsaucer wants to merge 57 commits into

apache:mainfrom

timsaucer:feat/expr-pickle

Commits on May 15, 2026

feat: pickle support for Expr via PythonLogicalCodec inline encoding

timsaucer
and
claude
committed
cc1bff7
docs(pickle): user guide + Ray example + CI backstop + UDF.name

timsaucer
and
claude
committed
c7f95ac
refactor: drop dead state from PythonFunctionScalarUDF

timsaucer
and
claude
committed
0450713
refactor(codec): use arrow-rs native IPC for schema serialization

timsaucer
and
claude
committed
fd46c94
docs: strip implementation jargon from Expr pickle docstrings

timsaucer
and
claude
committed
0879309
docs: reframe distributed-expression docs around the user goal

timsaucer
and
claude
committed
cc5ce7e
feat: inline encoding for Python window UDFs

timsaucer
and
claude
committed
89d119f
feat: inline encoding for Python aggregate UDFs

timsaucer
and
claude
committed
4b51402
docs: extend the inline-UDF guarantee to aggregate + window UDFs

timsaucer
and
claude
committed
71760c5
docs(distributing-expressions): link to pickle docs, generalize UDF kinds

timsaucer
and
claude
committed
9b4bbd6
docs: drop manual pickle.dumps/loads from worker examples

timsaucer
and
claude
committed
ca77810
docs: restructure distribution page for the multiple-approach story

timsaucer
and
claude
committed
ba5edf2
chore: untrack internal design doc that was committed by accident

timsaucer
and
claude
committed
b99d973
refactor: rename MultiColumnWindowUDF -> PythonFunctionWindowUDF

timsaucer
and
claude
committed
9a4af41
docs(distributing-work): drop __main__ guard from example to match site style

timsaucer
and
claude
committed
44df444
feat: per-session toggle for Python UDF inline encoding

timsaucer
and
claude
committed
977e88c
docs: pickle module security warning link + move user-facing prose

timsaucer
and
claude
committed
d7a1ff4
docs(ray-example): drop unnecessary UDF registration

timsaucer
and
claude
committed
ad0f956
feat: sender-side session context for outbound pickles

timsaucer
and
claude
committed
8130ab2
fix(ipc): doctest collection failure on module docstring

timsaucer
and
claude
committed
1300076
docs(example): multiprocessing pool with distinct expressions per worker

timsaucer
and
claude
committed
d5055e1
fix(codec): preserve UDAF state field metadata across pickle round-trip

timsaucer
and
claude
committed
c8c9306
refactor(codec): pin Volatility wire format to explicit match

timsaucer
and
claude
committed
0b8672e
fix: drop Python callable from UDF Hash impls

timsaucer
and
claude
committed
d722bdc
perf(codec): cache cloudpickle module handle in a OnceLock

timsaucer
and
claude
committed
b5684bc
refactor(udf): scalar from_parts now takes Vec<DataType>, matching udaf/udwf

timsaucer
and
claude
committed
8af9c53
docs(pyproject): cloudpickle covers aggregate and window UDFs too

timsaucer
and
claude
committed
ddd76f8
refactor(codec): drop redundant buf.is_empty() guards

timsaucer
and
claude
committed
ca2f0ff
refactor(expr): drop redundant bytes(...) wrap in to_bytes

timsaucer
and
claude
committed
8a22df1
refactor(tests): drop redundant inline Expr reimports

timsaucer
and
claude
committed
d578c85
docs(codec): justify synthetic input field names and nullability

timsaucer
and
claude
committed
5ae44f2
fix(codec): use Execution error for inline-payload refusal

timsaucer
and
claude
committed
8a321d6
refactor(codec): cache cloudpickle via per-interpreter PyOnceLock

timsaucer
and
claude
committed
508f8f6
test(pickle): probe multiprocessing lazily via cached fixture

timsaucer
and
claude
committed
5762435
refactor(udwf): drop redundant PythonFunctionWindowUDF::from_parts

timsaucer
and
claude
committed
54bbdf8
refactor(udaf): name synthesized state fields state_{i}

timsaucer
and
claude
committed
cea9254
perf(udf): pointer-identity fast path in PartialEq for Python UDFs

timsaucer
and
claude
committed
2f241be
docs(example): note null→0 coercion in above-threshold UDF

timsaucer
and
claude
committed
4a8fadc
test(pickle): tighten refusal assertion from Exception to RuntimeError

timsaucer
and
claude
committed
9a265e4
docs(codec): clarify strict-mode security scope

timsaucer
and
claude
committed
0642160
build(uv): regenerate uv.lock with current uv

timsaucer
and
claude
committed
9d78485
test(pickle): rename strict-pickle inline-worker test for accuracy

timsaucer
and
claude
committed
eb762a0
docs(distributing): note fork-worker threading.local() inheritance

timsaucer
and
claude
committed
310080c
test(ffi): strict-mode Expr round-trip with an FFI-capsule scalar UDF

timsaucer
and
claude
committed
f25aa87
fix(udf): name property reports FFI capsule name, not the ignored ctor arg

timsaucer
and
claude
committed
69b0ddb
refactor(ctx): make with_python_udf_inlining `enabled` keyword-only

timsaucer
and
claude
committed
e39311c
refactor(codec): factor shared schema/signature helpers out of UDF encoders

timsaucer
and
claude
committed
9ef5d0a
feat(codec): split UDF wire-format magic into family prefix + version byte

timsaucer
and
claude
committed
9e15f72
fix(codec): point strict refusal at the sender, not the receiver

timsaucer
and
claude
committed
f432d6b
fix(udf): log Python __eq__ exceptions instead of silently mapping to unequal

timsaucer
and
claude
committed
fafefd0
docs(expr): restate to_bytes behavior when ctx is None

timsaucer
and
claude
committed
865d9df
docs(ipc): explain why _resolve_ctx imports SessionContext lazily

timsaucer
and
claude
committed
cbfa652
chore: replace blanket S301 allow with per-line noqa at pickle.loads

timsaucer
and
claude
committed
dfcf3bc
test(pickle): drop dead register_udf in strict refusal test

timsaucer
and
claude
committed
b9c6045
test(pickle): cover with_python_udf_inlining toggle off-then-on

timsaucer
and
claude
committed
1343057
docs(distributing): narrow Python 3.14 start-method note to Linux

timsaucer
and
claude
committed
8aad8da
docs: centralize pickle security caveat, link from codec/context docs

timsaucer
and
claude
committed
a843b37