examples: document the JAX/MPI conflict in the enlarge-subspace driver, unify GPU examples - #38
Merged
Merged
Conversation
…nify GPU example ranks enlarge_batch_from_transitions (qiskit-addon-sqd, JAX-based) has no MPI awareness -- every rank redundantly runs it, and if JAX is set up for GPU, every rank tries to grab one at once, which doesn't work. Note it: use a single MPI rank, or JAX_PLATFORMS=cpu to keep this step on CPU regardless of rank count. Also made the two --device gpu example commands consistent with each other: 8 MPI ranks, --adet_comm_size 4 --bdet_comm_size 2, no --task_comm_size (previously 2/2/2).
shorten the comments
The JAX_PLATFORMS=cpu example carried a stray backtick (`JAX_PLATFORMS=cpu` mpirun ...`), which bash reads as an unterminated command substitution -- copy-pasting it fails. Also a blank line before the note so it reads as its own paragraph, 'an MPI job', and drop a double blank line left behind by the earlier trim. Verified all three documented commands on h100 (8 ranks, --device gpu, 4x2 grid): run_sqd_sbd.py H2O gives -76.2359466308 and run_sqd_enlarge_subspace_sbd.py gives -76.2421767512, both matching the documented values. Also confirmed the note's premise directly -- with GPU-enabled JAX and 8 ranks, omitting JAX_PLATFORMS=cpu fails with JaxRuntimeError RESOURCE_EXHAUSTED / CUDA_ERROR_OUT_OF_MEMORY.
…/MPI note The module docstring still showed 'mpirun -np 4' with no mention of JAX, which the README now documents as a setup that dies with CUDA_ERROR_OUT_OF_MEMORY when JAX is GPU-enabled -- so anyone reading the source or --help instead of the README got the broken recipe. Docstring now carries the same JAX_PLATFORMS=cpu guidance and the same 8-rank / 4x2 GPU command the README uses. Verified the docstring's command verbatim on h100: converges to -76.2421767512, matching the documented value.
add notes about AMD GPUs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents a real failure mode in
run_sqd_enlarge_subspace_sbd.py, and makesthe GPU example commands consistent.
through qiskit-addon-sqd's own JAX-based
enlarge_batch_from_transitions, which has no MPI awareness. Every rankruns it redundantly, and if JAX is set up for GPU, every rank tries to
grab a GPU for it at once and the run dies. Documented in
python/examples/README.mdand in the driver's own module docstring, withthe fix: run on a single rank, or set
JAX_PLATFORMS=cpu(SBD's own--device gpudiagonalization is unaffected either way).--device gpuexample commands now use8 MPI ranks with
--adet_comm_size 4 --bdet_comm_size 2, and none set--task_comm_size.--devicevalue eachbackend maps to, and says to substitute
--device gpu-ompfor--device gpuon AMD GPUs.Verification (on h100, 8x H100)
The JAX/MPI premise was confirmed directly rather than assumed — h100's JAX
is now GPU-enabled (
jax.devices()reports 8CudaDevices):JAX_PLATFORMS=cpu, 8 ranks,--device gpu: fails withjax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: CUDA_ERROR_OUT_OF_MEMORY.JAX_PLATFORMS=cpu, same command: converges to-76.2421767512, matching the documented value.
run_sqd_sbd.py's H2O example at the new 8-rank / 4x2 /--device gpusettings: -76.2359466308, matching its documented value.
JAX_PLATFORMS=cpu mpirun ...(prefix form, as documented) does propagateto all ranks under this MPICH build, so no
-x/-genvis needed.gpu-ompconfirmed a valid--devicechoice in all three drivers'argparse, and the AMD guidance matches
device_config.py's own docstrings(
gpu()is "NVIDIA only ... on an AMD host use gpu_omp instead";gpu_omp()"Works on NVIDIA and AMD").Not tested on AMD hardware — the AMD note restates this repo's own
documented backend support rather than a fresh measurement.
Test plan
--task_comm_size.