macOS: take OpenMP from the compiler's own tree when it has one - #34
Draft
Jim Garrison (garrison) wants to merge 6 commits into
Draft
Jim Garrison (garrison) wants to merge 6 commits into
Jim Garrison (garrison) wants to merge 6 commits into
Conversation
An LLVM that builds the openmp runtime -- Homebrew's llvm formula, and the conda-forge clang packages -- installs libomp.dylib and omp.h inside its own tree, and `-fopenmp` links THAT copy regardless of what else is on the link line. The Darwin block nonetheless always added a libomp of its own, from $CONDA_PREFIX or from $(brew --prefix)/opt/libomp, so building with such a compiler put two same-named runtimes into _core_cpu.so. The result is an abort at the first parallel region: OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized. [ 8] libomp.dylib __kmpc_fork_call + 52 [ 9] _core_cpu.cpython-314-darwin.so sbd::GenerateExcitation... Resolve the compiler before choosing OpenMP rather than merely reporting it afterwards, and prefer a runtime shipped beside that compiler. Only when there is none -- Apple clang, which has no OpenMP at all -- fall through to conda and then Homebrew as before. BLAS stays a separate question: the compiler tree carries no OpenBLAS, so that continues to come from conda or Homebrew. This was not reachable from this repo's own CI, whose macOS cells build with Apple clang and so take the third path. It reproduces in Qiskit/qiskit-addon-sqd#367, which pins CC/CXX to Homebrew LLVM in order to build another extension, and it is not fixable from that side: the paths were hardcoded here and ignored CPPFLAGS/LDFLAGS entirely. Verified on Linux (tox -e py) that the non-Darwin path is untouched, and unit-tested the new detection against a synthetic LLVM keg, Apple clang, and a tree with a header but no library. Assisted-by: Claude Opus 5
The previous commit looked for the compiler's omp.h at <prefix>/include, so detection returned nothing for the very compiler it was written for and the build fell through to Homebrew's standalone libomp -- the two-runtime abort it was meant to prevent. Confirmed in Qiskit/qiskit-addon-sqd#367, where the diagnostic reported "libomp and BLAS from Homebrew" while CXX was /opt/homebrew/opt/llvm/bin/clang++. An LLVM installs omp.h into its clang resource directory (lib/clang/<ver>/include), not <prefix>/include; Homebrew's llvm formula compiles `#include <omp.h>` in its own test with no -I at all. Only libomp.dylib lands in <prefix>/lib. So ask the driver via `clang++ -print-resource-dir` rather than guessing, which also avoids hardcoding an LLVM version into the path. Return the include and library directories separately, since they are no longer under one prefix. Unit-tested against synthetic trees for the Homebrew-LLVM layout, Apple clang, a tree with libomp but no omp.h, and a compiler that does not understand -print-resource-dir. Linux (tox -e py) still passes. Assisted-by: Claude Opus 5
Jim Garrison (garrison)
marked this pull request as ready for review
September 17, 2026 20:58
The preceding commits teach setup.py to take OpenMP from the compiler's own tree when it has one, but nothing here exercises that: every macOS cell uses Apple clang, which has no OpenMP of its own and so only ever takes the Homebrew-libomp branch. That gap is not hypothetical -- the first version of the detection looked for omp.h at <prefix>/include, found nothing, silently fell back to Homebrew, and still built and still passed. It took a downstream repo pinning CC/CXX to Homebrew LLVM to notice. Add one macOS cell that pins Homebrew LLVM, whose clang carries a libomp of its own. `cc` joins the matrix identity rather than being an extra key on an existing cell, because an include entry matching an existing os/python-version pair merges into it -- which would have replaced the Apple clang coverage for 3.14 instead of adding to it. Verified by expanding the matrix: 11 cells, with macos/3.14 present under both cc: system and cc: llvm. A passing suite alone would not protect this path, since the fallback compiles and tests clean and only aborts once another OpenMP consumer shares the process. So also assert, on every macOS cell, that exactly one OpenMP runtime is mapped after the backend loads -- read from dyld, and after get_backend(), since backends load lazily on first use. Assisted-by: Claude Opus 5
The new cc: llvm CI cell aborted with "OMP: Error #15" on its first run: the build took OpenMP from /opt/homebrew/opt/libomp while -fopenmp linked /opt/homebrew/Cellar/llvm/23.1.0/lib/libomp.dylib. So _compiler_openmp() returned None for the very compiler it exists to detect, and the fallback then picked a second runtime. CC/CXX did reach the build -- tox's passenv covers the .pkg env, verified locally -- so the compiler was right and the probe was wrong. The paths differ in form: CC/CXX is Homebrew's opt/ alias (/opt/homebrew/opt/llvm/bin/clang++) while -print-resource-dir answers with the real Cellar path. Resolve the compiler with realpath before deriving lib_dir or invoking it, so both forms agree. Also print what the probe found -- compiler, lib_dir, resource_dir, and whether each of libomp and omp.h is present -- rather than returning a silent None. That silence is what made this expensive to chase: the fallback compiles cleanly and passes this repo's own suite, and only aborts once a second OpenMP consumer shares the process. Tested against synthetic kegs reached both through an opt/ symlink and by real path. Linux unaffected (tox -e py). Whether the symlink was the whole cause is not yet established -- the runner's probe output will say, since it now reports each check. Assisted-by: Claude Opus 5
The cc: llvm cell still aborts, and the probe added in the previous commit printed nothing at all in CI -- so it said nothing about why. tox hides the build backend's stdout at default verbosity; the wheel is built through pyproject_api in the .pkg env, and only stderr comes through. Route the probe and the other Darwin build-decision messages (resolved compiler, which OpenMP was chosen, the -print-resource-dir failure notice) to stderr. No behaviour change; this is about being able to see which check returns false on the runner instead of inferring it from the flags that end up on the compile line. Note for anyone reading the previous run's log: the -L/opt/homebrew/opt/libomp/lib and -I/opt/homebrew/opt/libomp/include lines in it are Homebrew's `brew install libomp` caveat text, not flags this build used. Linux unaffected (tox -e py). Assisted-by: Claude Opus 5
The cc: llvm cell has failed three runs in a row while reporting nothing about why: not the OpenMP probe added for exactly this purpose, and not even the unconditional "Using MPI from", "RPATH will be set to" and "Configuring CPU backend" lines setup.py has always printed. tox runs the wheel build through pyproject_api and, at default verbosity, prints one "build_wheel>" line and discards the rest -- stdout and stderr alike, which is why routing the probe to stderr did not help either. Checked locally which level is needed: -v still yields nothing, -vv shows it. This is instrumentation, not a fix. The underlying question -- why _compiler_openmp() does not engage on that runner -- is still open, and the next run should finally say. Assisted-by: Claude Opus 5
Jim Garrison (garrison)
marked this pull request as draft
September 19, 2026 02:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On macOS the Darwin block always supplied a libomp of its own, from
$CONDA_PREFIXor from$(brew --prefix)/opt/libomp. That is wrong whenever the compiler already ships one: an LLVM that builds theopenmpruntime — Homebrew'sllvmformula, and the conda-forge clang packages — carrieslibomp.dylibin its own tree, and-fopenmplinks that copy regardless of what else is on the link line. Naming a second libomp then puts two same-named runtimes into_core_cpu.so, which aborts at the first parallel region:What changed
The compiler is now resolved before OpenMP is chosen rather than merely reported afterwards, and a runtime shipped beside that compiler wins. The order is:
libomp— the Apple clang case, which has no OpenMP at allBLAS stays a separate question, since the compiler tree carries no OpenBLAS.
Detection asks
clang++ -print-resource-dirrather than guessing paths. That matters:omp.his installed into the clang resource directory (lib/clang/<ver>/include), not<prefix>/include— Homebrew's own formula test compiles#include <omp.h>with no-Iat all — whilelibomp.dylibdoes land in<prefix>/lib. My first attempt assumed<prefix>/include, so detection silently returned nothing and the build fell straight back to path 3; the second commit fixes that. Asking the driver also avoids hardcoding an LLVM version.Why this repo's CI could not catch it
The macOS cells here build with Apple clang, which takes path 3 and is unaffected. Reproducing it needs a compiler that brings its own OpenMP. Qiskit/qiskit-addon-sqd#367 pins
CC/CXXto Homebrew LLVM in order to build another extension, and that is where it showed up — and it was not fixable from that side, because these paths were hardcoded here and ignoredCPPFLAGS/LDFLAGSentirely.Verified against that job: the build now reports
Scope
This does not close #27. With sbd-eigensolver now linking exactly one libomp, that abort still reproduces, so the second copy belongs to another package in the process —
pyscf, which the notebook imports beforesbd, is the leading candidate and ships its own OpenMP in its macOS wheels. A diagnostic is running downstream to attribute it directly. What this PR removes is sbd-eigensolver's own contribution to the problem.Testing
tox -e pyon Linux, to confirm the non-Darwin path is untouchedlibompbut noomp.h, and a compiler that does not understand-print-resource-dirI have no Mac locally, so the macOS verification is via that downstream job rather than direct.
This PR was generated by Claude Opus 5 under my guidance.