refactor: unify and batch dataset pipeline workers

[joseph-isaacs]

Summary

• replace stage-specific download, conversion, and upload executors with one unified worker pool
• let every worker atomically claim download batches, upload batches, or conversion jobs
• prioritize upload high-water safety, download-buffer refill, ready uploads, then conversion
• perform true multi-file Hugging Face downloads with snapshot_download
• perform true multi-file Xet preuploads while preserving immutable planned commit boundaries
• adapt download and upload batch sizes independently from measured aggregate throughput
• preserve file/byte reservations, checkpoint failures for every claimed item, and resumability
• add a detailed plan/apply and scheduler guide
• apply a consistent Ruff baseline and pin Ruff as development tooling

Validation

• uv run --project scripts ruff format --check scripts
• uv run --project scripts ruff check scripts
• uv run --project scripts python -m unittest discover -s scripts/tests -v (26 tests, 1 skipped because the release vx binary is not built)
• end-to-end two-worker local download → Parquet conversion → upload scheduler test
• native multi-file download and preupload batching tests
• scheduler priority and adaptive batch-size tests
• python -m py_compile scripts/hf-sync.py
• git diff --check