fix(iast): wrong memory address in subprocess in mcp servers [backport 3.19] #15565
Conversation
IAST-enabled applications using Gunicorn/Uvicorn workers were experiencing segmentation faults (~33% crash rate on MCP streaming requests) due to memory corruption when processes fork. - C++ global singletons (`taint_engine_context`, `initializer`) initialized at module load - Taint maps storing PyObject pointers by memory address - Child processes after fork inherited stale pointers from parent process memory - Accessing these stale pointers → use-after-free → SIGSEGV crash - Implemented `pthread_atfork` handler that automatically resets C++ global state in child processes after every fork: - Added comprehensive null-check wrappers around all native functions to prevent crashes when native state is - Fixed test regression issues where context slots weren't being freed: **[AddressSanitizer (ASAN)](https://github.com/google/sanitizers/wiki/AddressSanitizer)** is a fast memory error detector that catches use-after-free, buffer overflows, and other memory corruption bugs at runtime. **1. Runtime Environment (No Recompilation Required)** The simplest way to test is using LD_PRELOAD with the system's libasan: ```bash ASAN_LIB=$(gcc -print-file-name=libasan.so) LD_PRELOAD=$ASAN_LIB \ ASAN_OPTIONS="detect_leaks=0:symbolize=1:abort_on_error=0" \ python3 -m pytest tests/appsec/iast/test_fork_handler_regression.py -v ``` **ASAN_OPTIONS explained:** - `detect_leaks=0` - Disable leak detection (Python has many false positives) - `symbolize=1` - Show human-readable stack traces - `abort_on_error=0` - Continue after first error (collect all errors) **2. Build with ASAN (Optional, for deeper analysis)** For more thorough testing, compile the native extension with ASAN: ```bash export CFLAGS="-fsanitize=address -fno-omit-frame-pointer -g" export CXXFLAGS="-fsanitize=address -fno-omit-frame-pointer -g" export LDFLAGS="-fsanitize=address" pip install --no-build-isolation --force-reinstall -e . ASAN_OPTIONS="detect_leaks=0:symbolize=1:abort_on_error=0" \ python3 -m pytest tests/appsec/iast/test_fork_handler_regression.py -v ``` This script demonstrates the fork safety fix and can be used to verify ASAN finds no errors: ```python """Minimal fork safety reproduction test for ASAN verification.""" import os from ddtrace.appsec._iast._taint_tracking import OriginType from ddtrace.appsec._iast._taint_tracking._native import initialize_native_state from ddtrace.appsec._iast._taint_tracking._taint_objects import taint_pyobject from ddtrace.appsec._iast._taint_tracking._context import ( start_request_context, debug_context_array_free_slots_number, debug_num_tainted_objects ) def main(): print(f"[Parent PID {os.getpid()}] Initializing IAST...") # Initialize native state initialize_native_state() # Create context and tainted objects in parent ctx_id = start_request_context() print(f"[Parent] Context created: {ctx_id}") # Create some tainted objects (populates native maps) for i in range(10): taint_pyobject(f"data_{i}", "source", f"value_{i}", OriginType.PARAMETER) tainted_count = debug_num_tainted_objects(ctx_id) print(f"[Parent] Tainted objects: {tainted_count}") # Fork pid = os.fork() if pid == 0: # Child process print(f"[Child PID {os.getpid()}] Started after fork") # Verify pthread_atfork reset worked free_slots = debug_context_array_free_slots_number() print(f"[Child] Free slots: {free_slots}") if free_slots > 0: print("[Child] ✅ Context slots were reset (pthread_atfork worked!)") os._exit(0) # Success else: print("[Child] ❌ Context slots NOT reset") os._exit(1) # Failure else: # Parent waits for child _, status = os.waitpid(pid, 0) exit_code = os.WEXITSTATUS(status) if exit_code == 0: print(f"[Parent] ✅ Child exited cleanly - Fork safety verified!") return 0 else: print(f"[Parent] ❌ Child failed with exit code {exit_code}") return 1 if __name__ == "__main__": exit(main()) ``` **Run with ASAN:** ```bash LD_PRELOAD=$(gcc -print-file-name=libasan.so) \ ASAN_OPTIONS="detect_leaks=0:symbolize=1:abort_on_error=0" \ python3 test_fork_asan.py ``` **Expected output (success):** ``` [Parent PID 12345] Initializing IAST... [Parent] Context created: 0 [Parent] Tainted objects: 10 [Child PID 12346] Started after fork [Child] Free slots: 2 [Child] ✅ Context slots were reset (pthread_atfork worked!) [Parent] ✅ Child exited cleanly - Fork safety verified! ``` **What ASAN would report WITHOUT this fix:** ``` ==12346==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000040 ==12346==The signal is caused by a READ memory access. #0 0x7f... in get_tainted_object_map_by_ctx_id #1 0x7f... in debug_context_array_free_slots_number ``` (cherry picked from commit d1b4fd8)
|
|
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 215 ± 3 ms. The average import time from base is: 213 ± 2 ms. The import time difference between this PR and base is: 2.2 ± 0.1 ms. Import time breakdownThe following import paths have grown:
|
Performance SLOsComparing candidate backport-15514-to-3.19 (0b742c2) with baseline 3.19 (dbbe2f1) 🟡 Near SLO Breach (1 suite)🟡 flasksimple - 18/18✅ appsec-getTime: ✅ 4.601ms (SLO: <4.750ms -3.1%) vs baseline: ~same Memory: ✅ 64.315MB (SLO: <66.500MB -3.3%) vs baseline: +5.0% ✅ appsec-postTime: ✅ 6.624ms (SLO: <6.750ms 🟡 -1.9%) vs baseline: -0.4% Memory: ✅ 64.244MB (SLO: <66.500MB -3.4%) vs baseline: +4.7% ✅ appsec-telemetryTime: ✅ 4.589ms (SLO: <4.750ms -3.4%) vs baseline: -0.2% Memory: ✅ 64.242MB (SLO: <66.500MB -3.4%) vs baseline: +4.9% ✅ debuggerTime: ✅ 1.856ms (SLO: <2.000ms -7.2%) vs baseline: ~same Memory: ✅ 47.828MB (SLO: <49.500MB -3.4%) vs baseline: +4.6% ✅ iast-getTime: ✅ 1.856ms (SLO: <2.000ms -7.2%) vs baseline: -0.2% Memory: ✅ 44.626MB (SLO: <49.000MB -8.9%) vs baseline: +4.7% ✅ profilerTime: ✅ 1.915ms (SLO: <2.100ms -8.8%) vs baseline: ~same Memory: ✅ 48.811MB (SLO: <50.000MB -2.4%) vs baseline: +4.9% ✅ resource-renamingTime: ✅ 3.366ms (SLO: <3.650ms -7.8%) vs baseline: ~same Memory: ✅ 54.600MB (SLO: <56.000MB -2.5%) vs baseline: +5.0% ✅ tracerTime: ✅ 3.355ms (SLO: <3.650ms -8.1%) vs baseline: ~same Memory: ✅ 54.471MB (SLO: <56.500MB -3.6%) vs baseline: +4.7% ✅ tracer-nativeTime: ✅ 3.357ms (SLO: <3.650ms -8.0%) vs baseline: -0.2% Memory: ✅ 54.451MB (SLO: <60.000MB -9.2%) vs baseline: +4.7% 📉 Performance Improvements (1 suite)📉 iastaspects - 118/118✅ add_aspectTime: ✅ 0.381µs (SLO: <10.000µs 📉 -96.2%) vs baseline: 📉 -15.6% Memory: ✅ 37.906MB (SLO: <41.500MB -8.7%) vs baseline: +0.3% ✅ add_inplace_aspectTime: ✅ 0.382µs (SLO: <10.000µs 📉 -96.2%) vs baseline: 📉 -14.5% Memory: ✅ 37.847MB (SLO: <41.500MB -8.8%) vs baseline: +0.1% ✅ add_inplace_noaspectTime: ✅ 0.287µs (SLO: <10.000µs 📉 -97.1%) vs baseline: +0.6% Memory: ✅ 37.749MB (SLO: <41.500MB -9.0%) vs baseline: ~same ✅ add_noaspectTime: ✅ 0.355µs (SLO: <10.000µs 📉 -96.4%) vs baseline: -1.4% Memory: ✅ 37.847MB (SLO: <41.500MB -8.8%) vs baseline: ~same ✅ bytearray_aspectTime: ✅ 1.309µs (SLO: <10.000µs 📉 -86.9%) vs baseline: +1.3% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: ~same ✅ bytearray_extend_aspectTime: ✅ 1.473µs (SLO: <10.000µs 📉 -85.3%) vs baseline: -5.2% Memory: ✅ 37.867MB (SLO: <41.500MB -8.8%) vs baseline: +0.1% ✅ bytearray_extend_noaspectTime: ✅ 0.615µs (SLO: <10.000µs 📉 -93.8%) vs baseline: ~same Memory: ✅ 37.729MB (SLO: <41.500MB -9.1%) vs baseline: ~same ✅ bytearray_noaspectTime: ✅ 0.489µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -0.2% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: ~same ✅ bytes_aspectTime: ✅ 1.290µs (SLO: <10.000µs 📉 -87.1%) vs baseline: -0.5% Memory: ✅ 37.827MB (SLO: <41.500MB -8.8%) vs baseline: -0.2% ✅ bytes_noaspectTime: ✅ 0.499µs (SLO: <10.000µs 📉 -95.0%) vs baseline: +0.4% Memory: ✅ 37.827MB (SLO: <41.500MB -8.8%) vs baseline: ~same ✅ bytesio_aspectTime: ✅ 1.290µs (SLO: <10.000µs 📉 -87.1%) vs baseline: -1.9% Memory: ✅ 37.808MB (SLO: <41.500MB -8.9%) vs baseline: ~same ✅ bytesio_noaspectTime: ✅ 0.501µs (SLO: <10.000µs 📉 -95.0%) vs baseline: -0.4% Memory: ✅ 37.729MB (SLO: <41.500MB -9.1%) vs baseline: -0.3% ✅ capitalize_aspectTime: ✅ 0.736µs (SLO: <10.000µs 📉 -92.6%) vs baseline: +0.5% Memory: ✅ 37.886MB (SLO: <41.500MB -8.7%) vs baseline: ~same ✅ capitalize_noaspectTime: ✅ 0.438µs (SLO: <10.000µs 📉 -95.6%) vs baseline: +0.6% Memory: ✅ 37.867MB (SLO: <41.500MB -8.8%) vs baseline: +0.3% ✅ casefold_aspectTime: ✅ 0.739µs (SLO: <10.000µs 📉 -92.6%) vs baseline: -0.4% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: -0.2% ✅ casefold_noaspectTime: ✅ 0.372µs (SLO: <10.000µs 📉 -96.3%) vs baseline: -0.6% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: -0.1% ✅ decode_aspectTime: ✅ 0.730µs (SLO: <10.000µs 📉 -92.7%) vs baseline: -0.1% Memory: ✅ 37.985MB (SLO: <41.500MB -8.5%) vs baseline: +0.2% ✅ decode_noaspectTime: ✅ 0.426µs (SLO: <10.000µs 📉 -95.7%) vs baseline: +0.9% Memory: ✅ 37.886MB (SLO: <41.500MB -8.7%) vs baseline: ~same ✅ encode_aspectTime: ✅ 0.710µs (SLO: <10.000µs 📉 -92.9%) vs baseline: -1.2% Memory: ✅ 37.886MB (SLO: <41.500MB -8.7%) vs baseline: +0.2% ✅ encode_noaspectTime: ✅ 0.402µs (SLO: <10.000µs 📉 -96.0%) vs baseline: -0.1% Memory: ✅ 37.768MB (SLO: <41.500MB -9.0%) vs baseline: -0.3% ✅ format_aspectTime: ✅ 3.447µs (SLO: <10.000µs 📉 -65.5%) vs baseline: +2.0% Memory: ✅ 37.906MB (SLO: <41.500MB -8.7%) vs baseline: ~same ✅ format_map_aspectTime: ✅ 3.724µs (SLO: <10.000µs 📉 -62.8%) vs baseline: +2.2% Memory: ✅ 37.906MB (SLO: <41.500MB -8.7%) vs baseline: +0.4% ✅ format_map_noaspectTime: ✅ 0.820µs (SLO: <10.000µs 📉 -91.8%) vs baseline: -0.8% Memory: ✅ 37.867MB (SLO: <41.500MB -8.8%) vs baseline: +0.3% ✅ format_noaspectTime: ✅ 0.591µs (SLO: <10.000µs 📉 -94.1%) vs baseline: -1.2% Memory: ✅ 37.768MB (SLO: <41.500MB -9.0%) vs baseline: -0.2% ✅ index_aspectTime: ✅ 0.343µs (SLO: <10.000µs 📉 -96.6%) vs baseline: -4.9% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: ~same ✅ index_noaspectTime: ✅ 0.313µs (SLO: <10.000µs 📉 -96.9%) vs baseline: -0.6% Memory: ✅ 37.749MB (SLO: <41.500MB -9.0%) vs baseline: -0.2% ✅ join_aspectTime: ✅ 1.284µs (SLO: <10.000µs 📉 -87.2%) vs baseline: 📉 -11.2% Memory: ✅ 37.808MB (SLO: <41.500MB -8.9%) vs baseline: -0.2% ✅ join_noaspectTime: ✅ 0.538µs (SLO: <10.000µs 📉 -94.6%) vs baseline: -0.1% Memory: ✅ 37.886MB (SLO: <41.500MB -8.7%) vs baseline: ~same ✅ ljust_aspectTime: ✅ 2.762µs (SLO: <20.000µs 📉 -86.2%) vs baseline: +7.3% Memory: ✅ 37.749MB (SLO: <41.500MB -9.0%) vs baseline: ~same ✅ ljust_noaspectTime: ✅ 0.409µs (SLO: <10.000µs 📉 -95.9%) vs baseline: -0.4% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: -0.2% ✅ lower_aspectTime: ✅ 2.278µs (SLO: <10.000µs 📉 -77.2%) vs baseline: +4.1% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: -0.2% ✅ lower_noaspectTime: ✅ 0.374µs (SLO: <10.000µs 📉 -96.3%) vs baseline: -0.7% Memory: ✅ 37.886MB (SLO: <41.500MB -8.7%) vs baseline: +0.1% ✅ lstrip_aspectTime: ✅ 2.205µs (SLO: <20.000µs 📉 -89.0%) vs baseline: +0.4% Memory: ✅ 37.808MB (SLO: <41.500MB -8.9%) vs baseline: -0.2% ✅ lstrip_noaspectTime: ✅ 0.386µs (SLO: <10.000µs 📉 -96.1%) vs baseline: -0.9% Memory: ✅ 37.847MB (SLO: <41.500MB -8.8%) vs baseline: -0.1% ✅ modulo_aspectTime: ✅ 0.978µs (SLO: <10.000µs 📉 -90.2%) vs baseline: -6.0% Memory: ✅ 37.847MB (SLO: <41.500MB -8.8%) vs baseline: +0.4% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 1.543µs (SLO: <10.000µs 📉 -84.6%) vs baseline: -2.9% Memory: ✅ 37.906MB (SLO: <41.500MB -8.7%) vs baseline: +0.3% ✅ modulo_aspect_for_bytesTime: ✅ 0.951µs (SLO: <10.000µs 📉 -90.5%) vs baseline: -7.8% Memory: ✅ 37.886MB (SLO: <41.500MB -8.7%) vs baseline: +0.2% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 1.171µs (SLO: <10.000µs 📉 -88.3%) vs baseline: -5.8% Memory: ✅ 37.867MB (SLO: <41.500MB -8.8%) vs baseline: ~same ✅ modulo_noaspectTime: ✅ 0.674µs (SLO: <10.000µs 📉 -93.3%) vs baseline: +0.9% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: -0.1% ✅ replace_aspectTime: ✅ 5.112µs (SLO: <10.000µs 📉 -48.9%) vs baseline: +3.6% Memory: ✅ 37.926MB (SLO: <41.500MB -8.6%) vs baseline: +0.3% ✅ replace_noaspectTime: ✅ 0.467µs (SLO: <10.000µs 📉 -95.3%) vs baseline: +1.4% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: -0.3% ✅ repr_aspectTime: ✅ 0.944µs (SLO: <10.000µs 📉 -90.6%) vs baseline: -0.9% Memory: ✅ 37.867MB (SLO: <41.500MB -8.8%) vs baseline: ~same ✅ repr_noaspectTime: ✅ 0.457µs (SLO: <10.000µs 📉 -95.4%) vs baseline: ~same Memory: ✅ 37.867MB (SLO: <41.500MB -8.8%) vs baseline: -0.2% ✅ rstrip_aspectTime: ✅ 1.873µs (SLO: <20.000µs 📉 -90.6%) vs baseline: +1.4% Memory: ✅ 37.886MB (SLO: <41.500MB -8.7%) vs baseline: +0.2% ✅ rstrip_noaspectTime: ✅ 0.382µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -0.8% Memory: ✅ 37.847MB (SLO: <41.500MB -8.8%) vs baseline: -0.1% ✅ slice_aspectTime: ✅ 0.481µs (SLO: <10.000µs 📉 -95.2%) vs baseline: -3.4% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: ~same ✅ slice_noaspectTime: ✅ 0.449µs (SLO: <10.000µs 📉 -95.5%) vs baseline: -1.5% Memory: ✅ 37.886MB (SLO: <41.500MB -8.7%) vs baseline: ~same ✅ stringio_aspectTime: ✅ 1.819µs (SLO: <10.000µs 📉 -81.8%) vs baseline: +8.0% Memory: ✅ 37.768MB (SLO: <41.500MB -9.0%) vs baseline: -0.2% ✅ stringio_noaspectTime: ✅ 0.919µs (SLO: <10.000µs 📉 -90.8%) vs baseline: -0.4% Memory: ✅ 37.906MB (SLO: <41.500MB -8.7%) vs baseline: ~same ✅ strip_aspectTime: ✅ 2.174µs (SLO: <20.000µs 📉 -89.1%) vs baseline: -1.5% Memory: ✅ 37.906MB (SLO: <41.500MB -8.7%) vs baseline: ~same ✅ strip_noaspectTime: ✅ 0.397µs (SLO: <10.000µs 📉 -96.0%) vs baseline: +2.6% Memory: ✅ 37.965MB (SLO: <41.500MB -8.5%) vs baseline: +0.4% ✅ swapcase_aspectTime: ✅ 2.557µs (SLO: <10.000µs 📉 -74.4%) vs baseline: +6.5% Memory: ✅ 37.808MB (SLO: <41.500MB -8.9%) vs baseline: ~same ✅ swapcase_noaspectTime: ✅ 0.538µs (SLO: <10.000µs 📉 -94.6%) vs baseline: -0.8% Memory: ✅ 37.847MB (SLO: <41.500MB -8.8%) vs baseline: -0.2% ✅ title_aspectTime: ✅ 2.400µs (SLO: <10.000µs 📉 -76.0%) vs baseline: +2.4% Memory: ✅ 37.867MB (SLO: <41.500MB -8.8%) vs baseline: +0.2% ✅ title_noaspectTime: ✅ 0.504µs (SLO: <10.000µs 📉 -95.0%) vs baseline: +0.1% Memory: ✅ 37.709MB (SLO: <41.500MB -9.1%) vs baseline: -0.3% ✅ translate_aspectTime: ✅ 3.466µs (SLO: <10.000µs 📉 -65.3%) vs baseline: +6.4% Memory: ✅ 37.847MB (SLO: <41.500MB -8.8%) vs baseline: ~same ✅ translate_noaspectTime: ✅ 1.054µs (SLO: <10.000µs 📉 -89.5%) vs baseline: +0.7% Memory: ✅ 37.847MB (SLO: <41.500MB -8.8%) vs baseline: +0.2% ✅ upper_aspectTime: ✅ 2.283µs (SLO: <10.000µs 📉 -77.2%) vs baseline: +4.6% Memory: ✅ 37.945MB (SLO: <41.500MB -8.6%) vs baseline: +0.1% ✅ upper_noaspectTime: ✅ 0.375µs (SLO: <10.000µs 📉 -96.2%) vs baseline: +1.2% Memory: ✅ 37.788MB (SLO: <41.500MB -8.9%) vs baseline: -0.2%
|
Backport d1b4fd8 from #15514 to 3.19.
IAST-enabled applications using Gunicorn/Uvicorn workers were experiencing segmentation faults (~33% crash rate on MCP streaming requests) due to memory corruption when processes fork.
C++ global singletons (
taint_engine_context,initializer) initialized at module loadTaint maps storing PyObject pointers by memory address
Child processes after fork inherited stale pointers from parent process memory
Accessing these stale pointers → use-after-free → SIGSEGV crash
Implemented
pthread_atforkhandler that automatically resets C++ global state in child processes after every fork:Added comprehensive null-check wrappers around all native functions to prevent crashes when native state is
Fixed test regression issues where context slots weren't being freed:
AddressSanitizer
(ASAN) is a fast memory error detector that catches use-after-free, buffer overflows, and other memory corruption bugs at runtime.
1. Runtime Environment (No Recompilation Required)
The simplest way to test is using LD_PRELOAD with the system's libasan:
ASAN_OPTIONS explained:
detect_leaks=0- Disable leak detection (Python has many false positives)symbolize=1- Show human-readable stack tracesabort_on_error=0- Continue after first error (collect all errors)2. Build with ASAN (Optional, for deeper analysis)
For more thorough testing, compile the native extension with ASAN:
This script demonstrates the fork safety fix and can be used to verify ASAN finds no errors:
Run with ASAN:
Expected output (success):
What ASAN would report WITHOUT this fix:
(cherry picked from commit d1b4fd8)
Description
Testing
Risks
Additional Notes