Skip to content

T-17170 Flush after each RQ job and start forked children with a clean handler - #47

Merged
PetrHeinz merged 4 commits into
masterfrom
claude/t-17170-rq-fork-flush
Sep 24, 2026
Merged

PetrHeinz merged 4 commits into
masterfrom
claude/t-17170-rq-fork-flush

Conversation

@PetrHeinz

@PetrHeinz PetrHeinz commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

RQ runs every job in a forked work horse that ends with os._exit(), which skips the interpreter shutdown that normally lets the handler send what it has buffered. A job that finishes within the flush interval therefore loses all of its records. That is the pattern in #36: the long-lived worker's own rq.worker lines arrive, the job's own lines do not.

Two changes, tests first (the first commit is expected to fail CI):

  • logtail.rq.Worker flushes every LogtailHandler after each job, inside the work horse, before RQ exits it. Use it with rq worker --worker-class logtail.rq.Worker, or RQ = {"WORKER_CLASS": "logtail.rq.Worker"} with django-rq. rq stays out of the package's dependencies, the module only imports it when used.
  • A forked child now starts with a fresh queue, no flush thread and a new HTTP session. Until now it inherited the parent's queue, so once the child flushes it would re-send whatever the parent still had buffered at fork time, a thread object whose thread does not exist in the child, and a requests.Session whose socket it shared with the parent.

🤖 Generated with Claude Code

PetrHeinz and others added 3 commits September 23, 2026 16:30
… forked children inheriting the parent's handler state

RQ ends every work horse with os._exit(), so nothing flushes the handler; a logtail.rq.Worker that flushes after each job is expected. A forked child also inherits the parent's queue, flush thread object and HTTP session.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…TP session

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…very RQ job

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@PetrHeinz
PetrHeinz marked this pull request as ready for review September 24, 2026 14:13
…ork-flush

# Conflicts:
#	tests/test_handler.py
@PetrHeinz
PetrHeinz merged commit 24b099b into master Sep 24, 2026
13 checks passed
@PetrHeinz
PetrHeinz deleted the claude/t-17170-rq-fork-flush branch September 24, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant