ansible: workaround broken Homebrew python@3.14 pyexpat on macOS Tahoe #221

New issue

Closed

opened 2026-06-04 10:11:58 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-06-04 10:11:58 +00:00

Owner

`coily ops *` broken on macOS Tahoe: Homebrew python@3.14 pyexpat fails to dlopen

On macOS Tahoe (Darwin 25.x), every coily ops verb that needs an SSM token fails with:

aws: [ERROR]: dlopen(/opt/homebrew/Cellar/python@3.14/.../pyexpat.cpython-314-darwin.so):
  Symbol not found: _XML_SetAllocTrackerActivationThreshold
  Expected in: /usr/lib/libexpat.1.dylib
coily: ops forgejo repo list: fetch /forgejo/api-token: exit status 255

Root cause

The Homebrew python@3.14 3.14.5 arm64_tahoe bottle ships pyexpat.cpython-314-darwin.so with an absolute install-name pointing at /usr/lib/libexpat.1.dylib. The shared object expects _XML_SetAllocTrackerActivationThreshold — a symbol the system libexpat in Tahoe does not export. brew reinstall python@3.14 reproduces the broken bottle. DYLD_FALLBACK_LIBRARY_PATH / DYLD_INSERT_LIBRARIES don't help because dyld binds the absolute path and the lookup itself succeeds — only the symbol resolution fails.

Homebrew's own expat 2.8.1 exports the symbol. The fix is to repoint pyexpat at it (install_name_tool -change) and re-sign the Mach-O ad-hoc (codesign -fs -).

Workaround landed

This issue ships ansible/playbooks/fix-pyexpat.yml, an idempotent local-connection playbook that:

Probes whether import pyexpat already works (no-op when fine).
install_name_tool -change /usr/lib/libexpat.1.dylib /opt/homebrew/opt/expat/lib/libexpat.1.dylib on the bottle's pyexpat.so.
codesign -fs - <pyexpat.so> so dyld accepts the modified binary.
Verifies import pyexpat, aws --version, and a real aws ssm get-parameter /forgejo/api-token call before reporting success.

Run:

cd ~/projects/coilyco-flight-deck/infrastructure
coily pkg uv run ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/fix-pyexpat.yml

Future work

Fold the play into freshen.yml (own role, gated on macOS) so coily ansible-freshen action=apply self-heals.
Track upstream: file a Homebrew core bug so the Tahoe bottle gets rebuilt against a libexpat with the required symbol. Once a fixed bottle lands, this workaround becomes a no-op via the probe step.

## `coily ops *` broken on macOS Tahoe: Homebrew python@3.14 pyexpat fails to dlopen On macOS Tahoe (Darwin 25.x), every `coily ops` verb that needs an SSM token fails with: ``` aws: [ERROR]: dlopen(/opt/homebrew/Cellar/python@3.14/.../pyexpat.cpython-314-darwin.so): Symbol not found: _XML_SetAllocTrackerActivationThreshold Expected in: /usr/lib/libexpat.1.dylib coily: ops forgejo repo list: fetch /forgejo/api-token: exit status 255 ``` ### Root cause The Homebrew `python@3.14` 3.14.5 `arm64_tahoe` bottle ships `pyexpat.cpython-314-darwin.so` with an absolute install-name pointing at `/usr/lib/libexpat.1.dylib`. The shared object expects `_XML_SetAllocTrackerActivationThreshold` — a symbol the system libexpat in Tahoe does not export. `brew reinstall python@3.14` reproduces the broken bottle. `DYLD_FALLBACK_LIBRARY_PATH` / `DYLD_INSERT_LIBRARIES` don't help because dyld binds the absolute path and the lookup itself succeeds — only the symbol resolution fails. Homebrew's own `expat` 2.8.1 exports the symbol. The fix is to repoint pyexpat at it (`install_name_tool -change`) and re-sign the Mach-O ad-hoc (`codesign -fs -`). ### Workaround landed This issue ships `ansible/playbooks/fix-pyexpat.yml`, an idempotent local-connection playbook that: 1. Probes whether `import pyexpat` already works (no-op when fine). 2. `install_name_tool -change /usr/lib/libexpat.1.dylib /opt/homebrew/opt/expat/lib/libexpat.1.dylib` on the bottle's pyexpat.so. 3. `codesign -fs - <pyexpat.so>` so dyld accepts the modified binary. 4. Verifies `import pyexpat`, `aws --version`, and a real `aws ssm get-parameter /forgejo/api-token` call before reporting success. Run: ``` cd ~/projects/coilyco-flight-deck/infrastructure coily pkg uv run ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/fix-pyexpat.yml ``` ### Future work - Fold the play into `freshen.yml` (own role, gated on macOS) so `coily ansible-freshen action=apply` self-heals. - Track upstream: file a Homebrew core bug so the Tahoe bottle gets rebuilt against a libexpat with the required symbol. Once a fixed bottle lands, this workaround becomes a no-op via the probe step.

coilysiren referenced this issue from a commit

2026-06-04 10:14:05 +00:00

feat: add ansible playbook to patch broken pyexpat on macOS Tahoe

coilysiren closed this issue

2026-06-04 10:14:05 +00:00