Skip to content

examples: document the JAX/MPI conflict in the enlarge-subspace driver, unify GPU examples - #38

Merged
Sophia Wen (hfwen0502) merged 5 commits into
mainfrom
readme-gpu-examples-jax-note
Sep 18, 2026
Merged

Sophia Wen (hfwen0502) merged 5 commits into
mainfrom
readme-gpu-examples-jax-note

Conversation

@hfwen0502

Copy link
Copy Markdown
Member

Summary

Documents a real failure mode in run_sqd_enlarge_subspace_sbd.py, and makes
the GPU example commands consistent.

  • JAX/MPI conflict. The excitation-expansion step between rounds goes
    through qiskit-addon-sqd's own JAX-based
    enlarge_batch_from_transitions, which has no MPI awareness. Every rank
    runs it redundantly, and if JAX is set up for GPU, every rank tries to
    grab a GPU for it at once and the run dies. Documented in
    python/examples/README.md and in the driver's own module docstring, with
    the fix: run on a single rank, or set JAX_PLATFORMS=cpu (SBD's own
    --device gpu diagonalization is unaffected either way).
  • Consistent GPU examples. All --device gpu example commands now use
    8 MPI ranks with --adet_comm_size 4 --bdet_comm_size 2, and none set
    --task_comm_size.
  • AMD note. The Overview now spells out which --device value each
    backend maps to, and says to substitute --device gpu-omp for
    --device gpu on AMD GPUs.

Verification (on h100, 8x H100)

The JAX/MPI premise was confirmed directly rather than assumed — h100's JAX
is now GPU-enabled (jax.devices() reports 8 CudaDevices):

  • Without JAX_PLATFORMS=cpu, 8 ranks, --device gpu: fails with
    jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: CUDA_ERROR_OUT_OF_MEMORY.
  • With JAX_PLATFORMS=cpu, same command: converges to
    -76.2421767512, matching the documented value.
  • run_sqd_sbd.py's H2O example at the new 8-rank / 4x2 / --device gpu
    settings: -76.2359466308, matching its documented value.
  • The driver docstring's command, run verbatim: -76.2421767512.
  • JAX_PLATFORMS=cpu mpirun ... (prefix form, as documented) does propagate
    to all ranks under this MPICH build, so no -x/-genv is needed.
  • gpu-omp confirmed a valid --device choice in all three drivers'
    argparse, and the AMD guidance matches device_config.py's own docstrings
    (gpu() is "NVIDIA only ... on an AMD host use gpu_omp instead";
    gpu_omp() "Works on NVIDIA and AMD").

Not tested on AMD hardware — the AMD note restates this repo's own
documented backend support rather than a fresh measurement.

Test plan

  • Both documented energies reproduced at the new rank/grid settings.
  • Failure mode reproduced, and the documented workaround shown to fix it.
  • Driver docstring command runs verbatim.
  • Markdown fences balanced; no example command sets --task_comm_size.

…nify GPU example ranks

enlarge_batch_from_transitions (qiskit-addon-sqd, JAX-based) has no MPI
awareness -- every rank redundantly runs it, and if JAX is set up for GPU,
every rank tries to grab one at once, which doesn't work. Note it: use a
single MPI rank, or JAX_PLATFORMS=cpu to keep this step on CPU regardless
of rank count.

Also made the two --device gpu example commands consistent with each
other: 8 MPI ranks, --adet_comm_size 4 --bdet_comm_size 2, no
--task_comm_size (previously 2/2/2).
shorten the comments
The JAX_PLATFORMS=cpu example carried a stray backtick
(`JAX_PLATFORMS=cpu` mpirun ...`), which bash reads as an unterminated
command substitution -- copy-pasting it fails. Also a blank line before
the note so it reads as its own paragraph, 'an MPI job', and drop a
double blank line left behind by the earlier trim.

Verified all three documented commands on h100 (8 ranks, --device gpu,
4x2 grid): run_sqd_sbd.py H2O gives -76.2359466308 and
run_sqd_enlarge_subspace_sbd.py gives -76.2421767512, both matching the
documented values. Also confirmed the note's premise directly -- with
GPU-enabled JAX and 8 ranks, omitting JAX_PLATFORMS=cpu fails with
JaxRuntimeError RESOURCE_EXHAUSTED / CUDA_ERROR_OUT_OF_MEMORY.
…/MPI note

The module docstring still showed 'mpirun -np 4' with no mention of JAX,
which the README now documents as a setup that dies with
CUDA_ERROR_OUT_OF_MEMORY when JAX is GPU-enabled -- so anyone reading the
source or --help instead of the README got the broken recipe. Docstring
now carries the same JAX_PLATFORMS=cpu guidance and the same 8-rank / 4x2
GPU command the README uses.

Verified the docstring's command verbatim on h100: converges to
-76.2421767512, matching the documented value.
add notes about AMD GPUs
@hfwen0502
Sophia Wen (hfwen0502) merged commit 435c0fa into main Sep 18, 2026
11 checks passed
@hfwen0502
Sophia Wen (hfwen0502) deleted the readme-gpu-examples-jax-note branch September 18, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant