Skip to content

pynumaflow-lite: Make mapper APIs more Pythonic. - #374

Merged
vigith merged 5 commits into
mainfrom
pynumaflow-lite-mapper
Sep 13, 2026
Merged

pynumaflow-lite: Make mapper APIs more Pythonic.#374
vigith merged 5 commits into
mainfrom
pynumaflow-lite-mapper

Conversation

@BulkBeing

@BulkBeing BulkBeing commented Sep 12, 2026

Copy link
Copy Markdown
Contributor
  • Removes the redundant keys parameter from handler method.
  • Installs shutdown handler by default, with an option to override

Syntax change

current:

async def mapper(self, datum: mapper.Datum) -> mapper.Messages:

New:

async def mapper(self, datum: mapper.Datum) -> list[Message]:

The keys parameter is removed, as there is already datum.keys.
Our list implementation (mapper.Messages) is removed, and uses builtin list.

Uncaught exceptions will be logged with ExceptionGroup instead of logging the first error.

Numaflow Map has shutdown...
  + Exception Group Traceback (most recent call last):
  |   File "/app/map_cat.py", line 27, in <module>
  |     asyncio.run(main())
  |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
  |     return runner.run(main)
  |            ^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
  |     return self._loop.run_until_complete(task)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
  |     return future.result()
  |            ^^^^^^^^^^^^^^^
  |   File "/app/map_cat.py", line 22, in main
  |     await MapAsyncServer(SimpleCat()).serve()
  |   File "/app/.venv/lib/python3.11/site-packages/pynumaflow_lite/_map_server.py", line 39, in serve
  |     await self._serve(install_signal_handlers=self._install_signal_handlers)
  |   File "/app/.venv/lib/python3.11/site-packages/pynumaflow_lite/_map_server.py", line 48, in _serve
  |     await self._core.start(self._handler)
  | ExceptionGroup: 227 map handler calls failed (227 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 3 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 4 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 5 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 6 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 7 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 8 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 9 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 10 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 11 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 12 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 13 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 14 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- 15 ----------------
    | Traceback (most recent call last):
    |   File "/app/map_cat.py", line 12, in handler
    |     raise RuntimeError("Simulated error")
    | RuntimeError: Simulated error
    +---------------- ... ----------------
    | and 212 more exceptions
    +------------------------------------

Similar changes will be made to other map variants in a follow up PR

- Removes the redundant keys parameter from handler method.
- Installs shutdown handler by default, with an option to override

Signed-off-by: Sreekanth <prsreekanth920@gmail.com>
@codecov

codecov Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.99%. Comparing base (82cd56f) to head (a4b5968).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #374   +/-   ##
=======================================
  Coverage   92.99%   92.99%           
=======================================
  Files          68       68           
  Lines        3656     3656           
  Branches      236      236           
=======================================
  Hits         3400     3400           
  Misses        190      190           
  Partials       66       66           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Sreekanth <prsreekanth920@gmail.com>
Signed-off-by: Sreekanth <prsreekanth920@gmail.com>
@BulkBeing
BulkBeing marked this pull request as ready for review September 12, 2026 13:47
Comment on lines +16 to +31
fn fail(&self, error: PyErr) -> Vec<map::Message> {
// Only the first error is reported; later requests may still be in flight
// while shutdown is underway, and their failures would be duplicates.
let mut error_slot = self.error_slot.lock().unwrap();
if error_slot.is_none() {
Python::attach(|py| error.print(py));
*error_slot = Some(error);
}
drop(error_slot);

if let Some(tx) = self.shutdown_tx.lock().unwrap().take() {
let _ = tx.send(());
}

Vec::new()
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does this fail work? can you please explain.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update this part. I was returning empty vec on failure since the mapper Rust trait needed a Vec to returned. Then convert it to exception on Python side. But this caused issues as only panic is propagated to numa sidecar for displaying in the UI.
This code is updated to use exception groups, updated the PR description with sample output.

Signed-off-by: Sreekanth <prsreekanth920@gmail.com>

@vigith vigith left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually how does other popular pyo3 projects do it?

Signed-off-by: Vigith Maurice <vigith@gmail.com>
@vigith
vigith enabled auto-merge (squash) September 13, 2026 02:13
@vigith
vigith merged commit dc5092d into main Sep 13, 2026
10 checks passed
@vigith
vigith deleted the pynumaflow-lite-mapper branch September 13, 2026 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants