[ISSUE] Runtime pure Python fallback for NVX doesn't work consistently

[aurieh]

📝 Note for maintainers: Please triage this issue (review,
classify, and assign appropriate labels).

---

What is your issue about?

When AUTOBAHN_USE_NVX=0 is set during build time, leaving it unset at runtime does not allow the pure Python fallback for Utf8Validator and create_xor_masker to load:

$ AUTOBAHN_USE_NVX=false pip install -q --no-cache --no-binary=autobahn autobahn
$ python -c 'import autobahn.websocket as w; print(w.HAS_NVX, w.USES_NVX)'
True True
$ python -c 'import autobahn.websocket.utf8validator as u; u.Utf8Validator()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import autobahn.websocket.utf8validator as u; u.Utf8Validator()
                                                  ~~~~~~~~~~~~~~~^^
  File "/tmp/tmp.NtrSLmGHQz/venv/lib/python3.13/site-packages/autobahn/nvx/_utf8validator.py", line 100, in __init__
    from _nvx_utf8validator import lib
ModuleNotFoundError: No module named '_nvx_utf8validator'
$ python -c 'import autobahn.websocket.xormasker as x; x.create_xor_masker(b"\0" * 4)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import autobahn.websocket.xormasker as x; x.create_xor_masker(b"\0" * 4)
                                              ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/tmp/tmp.NtrSLmGHQz/venv/lib/python3.13/site-packages/autobahn/nvx/_xormasker.py", line 165, in create_xor_masker
    return XorMaskerSimple(mask)
  File "/tmp/tmp.NtrSLmGHQz/venv/lib/python3.13/site-packages/autobahn/nvx/_xormasker.py", line 144, in __init__
    super().__init__(mask, use_simd=False)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.NtrSLmGHQz/venv/lib/python3.13/site-packages/autobahn/nvx/_xormasker.py", line 101, in __init__
    from _nvx_xormasker import ffi as _ffi, lib as _lib
ModuleNotFoundError: No module named '_nvx_xormasker'

This seems to be because HAS_NVX is set based on whether the wrapper modules are importable:

autobahn-python/autobahn/websocket/__init__.py

Lines 43 to 53 in b063660

	# Step 1: Probe for NVX availability (was it built and can we import it?)
	_has_nvx = False
	try:
	# Try importing both NVX modules to verify they're available
	from autobahn.nvx._xormasker import create_xor_masker as _nvx_xor_test # noqa: F401
	from autobahn.nvx._utf8validator import Utf8Validator as _nvx_utf8_test # noqa: F401
	_has_nvx = True
	except ImportError:
	# NVX not available (not built or CFFI compilation failed)
	pass

which is not an indicator of whether NVX was built because the actual _nvx_* imports are lazy:

autobahn-python/autobahn/nvx/_utf8validator.py

Line 100 in b063660

from _nvx_utf8validator import lib

autobahn-python/autobahn/nvx/_xormasker.py

Line 101 in b063660

from _nvx_xormasker import ffi as _ffi, lib as _lib

---

Additional context

---

Triage Notes (for maintainers only)

---

AI Assistance Disclosure

• I did not use any AI-assistance tools to help create
  this issue.
• I did use AI-assistance tools to help create this
  issue.
• I have read, understood and followed the projects'
  AI Policy
  when creating code, documentation etc. for this pull
  request.

[oberstet]

first, installation from source-dist and without building NVX should work at run-time (without using NVX)!

next, our workflows already test a whole bunch of things related, including running all AutobahnTestsuite tests for both with-NVX and without-NVX at run-time

what we probably should add as well is install-from-source-dist and then test (without NVX only)

I need to look / investigate / add that.

oberstet@amd-ryzen5:~/work/wamp/autobahn-python$ find .github/workflows/ -name "*.yml" -exec grep -Hi "AUTOBAHN_USE_NVX" {} \;
.github/workflows/wheels-arm64.yml:              -e AUTOBAHN_USE_NVX=1 \
.github/workflows/wstest.yml:              export AUTOBAHN_USE_NVX=1
.github/workflows/wstest.yml:              echo "==> Testing WITH NVX acceleration (AUTOBAHN_USE_NVX=1)"
.github/workflows/wstest.yml:              export AUTOBAHN_USE_NVX=0
.github/workflows/wstest.yml:              echo "==> Testing WITHOUT NVX acceleration (AUTOBAHN_USE_NVX=0)"
.github/workflows/wstest.yml:              export AUTOBAHN_USE_NVX=1
.github/workflows/wstest.yml:              echo "==> Testing WITH NVX acceleration (AUTOBAHN_USE_NVX=1)"
.github/workflows/wstest.yml:              export AUTOBAHN_USE_NVX=0
.github/workflows/wstest.yml:              echo "==> Testing WITHOUT NVX acceleration (AUTOBAHN_USE_NVX=0)"
.github/workflows/wheels-docker.yml:          AUTOBAHN_USE_NVX: 1
.github/workflows/wheels-docker.yml:          echo "AUTOBAHN_USE_NVX=$AUTOBAHN_USE_NVX"
.github/workflows/wheels.yml:          export AUTOBAHN_USE_NVX=0
.github/workflows/wheels.yml:          export AUTOBAHN_USE_NVX=1
.github/workflows/wheels.yml:          $env:AUTOBAHN_USE_NVX = "1"
oberstet@amd-ryzen5:~/work/wamp/autobahn-python$ 

---

on a side note, but related, wanted to stress again: autobahn install from binary wheel does work including NVX, on x86-64 and ARM64 on RHEL9, and on CPython and PyPy - all combinations of the former!

fwiw, here is a snipped from a Docker image (one combo):

# Update all installed packages and install Python 3.11 (avail. since RHEL 9.2 AppStream), and minimal tools
RUN microdnf update -y && \
    microdnf install -y jq python3.11 python3.11-pip && \
    microdnf clean all && \
    rm -rf /var/cache/dnf /var/cache/yum

# Base path for typedef int GmbH software and user application
RUN mkdir -p /opt/typedefint /app

# Create virtual environments:
#  - for aux tools (e.g. /opt/typedefint/tools/bin/cyclonedx-py)
#  - for Autobahn|Python and user applications
RUN python3.11 -m venv /opt/typedefint/tools && \
    /opt/typedefint/tools/bin/pip install --no-cache-dir cyclonedx-bom && \
    python3.11 -m venv /opt/typedefint/python && \
    /opt/typedefint/python/bin/pip install --no-cache-dir autobahn[all]==25.10.2

[oberstet]

one more note: if / once we fix above, and generally, we'll aim to publish this as part of an upcoming v25.11.1 then.

this will then include NVX (already in v25.10.2) native code (optional), but it will also contain vendored Flatbuffers (also optional) - the latest stable 25.9.23! - pls see: #1761

that supports the upcoming support of WAMP-Flatbuffers for Flatbuffers for WAMP transport and for WAMP payload serializer (all combinations), here: #1765

[oberstet]

alright, I can reproduce this bug, and your analysis is spot-on: it's not enough for the wrappers to be importable during run-time detection because the actual CFFI extension modules are only loaded lazy.

oberstet@amd-ryzen5:~/work/wamp/autobahn-python$ just distclean 
==> Performing a deep clean (distclean)...
--> Removing venvs, cache, and build/dist directories...
--> Searching for and removing nested Python caches...
--> Searching for and removing compiled Python files...
--> Searching for and removing setuptools egg-info directories...
--> Searching for and removing coverage data...
==> Distclean complete. The project is now pristine.
oberstet@amd-ryzen5:~/work/wamp/autobahn-python$ just create
==> No venv name specified. Auto-detecting from system Python...
==> Defaulting to venv: 'cpy312'
==> Creating Python virtual environment 'cpy312' using cpython-3.12 in ./.venvs/cpy312...
Using CPython 3.12.11
Creating virtual environment with seed packages at: ./.venvs/cpy312
 + pip==25.3
Activate with: source .venvs/cpy312/bin/activate
==> Successfully created venv 'cpy312'.
Python 3.12.11
pip 25.3 from /home/oberstet/work/wamp/autobahn-python/.venvs/cpy312/lib/python3.12/site-packages/pip (python 3.12)
==> Activate Python virtual environment with: source ./.venvs/cpy312/bin/activate
oberstet@amd-ryzen5:~/work/wamp/autobahn-python$ source ./.venvs/cpy312/bin/activate
(cpy312) oberstet@amd-ryzen5:~/work/wamp/autobahn-python$ AUTOBAHN_USE_NVX=0 pip install -q --no-cache --no-binary=autobahn autobahn
(cpy312) oberstet@amd-ryzen5:~/work/wamp/autobahn-python$ pip show autobahn
Name: autobahn
Version: 25.10.2
Summary: WebSocket client & server library, WAMP real-time framework
Home-page: https://autobahn.readthedocs.io/
Author: 
Author-email: typedef int GmbH <contact@typedefint.eu>
License-Expression: MIT
Location: /home/oberstet/work/wamp/autobahn-python/.venvs/cpy312/lib/python3.12/site-packages
Requires: cryptography, hyperlink, txaio
Required-by: 
(cpy312) oberstet@amd-ryzen5:~/work/wamp/autobahn-python$ python -c 'import autobahn.websocket as w; print(w.HAS_NVX, w.USES_NVX)'
True True
(cpy312) oberstet@amd-ryzen5:~/work/wamp/autobahn-python$ python -c 'import autobahn.websocket.utf8validator as u; u.Utf8Validator()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/oberstet/work/wamp/autobahn-python/autobahn/nvx/_utf8validator.py", line 100, in __init__
    from _nvx_utf8validator import lib
ModuleNotFoundError: No module named '_nvx_utf8validator'
(cpy312) oberstet@amd-ryzen5:~/work/wamp/autobahn-python$ 

[oberstet]

merged, will be in released in 25.11.1 - you can preview/check on nightly #1770