Data race: ctypes `PyCData_NewGetBuffer` reads `b_ptr` without the critical section `_ctypes_resize` holds

[Naserume]

Bug report

Bug description:

#131336 / #128182 made ctypes.resize and addressof/byref thread-safe by taking a critical section in _ctypes_resize, which reallocates obj->b_ptr:

cpython/Modules/_ctypes/callproc.c

Lines 1875 to 1935 in a2a8466

	/*[clinic input]
	@critical_section obj
	_ctypes.resize
	obj: object(subclass_of="clinic_state()->PyCData_Type", type="CDataObject *")
	size: Py_ssize_t
	/
	[clinic start generated code]*/
	static PyObject *
	_ctypes_resize_impl(PyObject *module, CDataObject *obj, Py_ssize_t size)
	/*[clinic end generated code: output=11c89c7dbdbcd53f input=bf5a6aaea8514261]*/
	{
	ctypes_state *st = get_module_state(module);
	StgInfo *info;
	int result = PyStgInfo_FromObject(st, (PyObject *)obj, &info);
	if (result < 0) {
	return NULL;
	}
	if (info == NULL) {
	PyErr_SetString(PyExc_TypeError,
	"expected ctypes instance");
	return NULL;
	}
	if (size < info->size) {
	PyErr_Format(PyExc_ValueError,
	"minimum size is %zd",
	info->size);
	return NULL;
	}
	if (obj->b_needsfree == 0) {
	PyErr_Format(PyExc_ValueError,
	"Memory cannot be resized because this object doesn't own it");
	return NULL;
	}
	if ((size_t)size <= sizeof(obj->b_value)) {
	/* internal default buffer is large enough */
	obj->b_size = size;
	goto done;
	}
	if (!_CDataObject_HasExternalBuffer(obj)) {
	/* We are currently using the objects default buffer, but it
	isn't large enough any more. */
	void *ptr = PyMem_Calloc(1, size);
	if (ptr == NULL)
	return PyErr_NoMemory();
	memmove(ptr, obj->b_ptr, obj->b_size);
	obj->b_ptr = ptr;
	obj->b_size = size;
	} else {
	void * ptr = PyMem_Realloc(obj->b_ptr, size);
	if (ptr == NULL)
	return PyErr_NoMemory();
	obj->b_ptr = ptr;
	obj->b_size = size;
	}
	done:
	Py_RETURN_NONE;
	}
	static PyObject *

But the buffer-protocol getbuffer, PyCData_NewGetBuffer (reached via memoryview(cdata)), reads self->b_ptr without that critical section:

cpython/Modules/_ctypes/_ctypes.c

Lines 3103 to 3130 in a2a8466

	static int
	PyCData_NewGetBuffer(PyObject *myself, Py_buffer *view, int flags)
	{
	CDataObject *self = _CDataObject_CAST(myself);
	ctypes_state *st = get_module_state_by_def(Py_TYPE(Py_TYPE(myself)));
	StgInfo *info;
	if (PyStgInfo_FromObject(st, myself, &info) < 0) {
	return -1;
	}
	assert(info);
	PyObject *item_type = PyCData_item_type(st, (PyObject*)Py_TYPE(myself));
	if (item_type == NULL) {
	return 0;
	}
	if (view == NULL) return 0;
	StgInfo *item_info;
	if (PyStgInfo_FromType(st, item_type, &item_info) < 0) {
	return -1;
	}
	assert(item_info);
	view->buf = self->b_ptr;
	view->obj = Py_NewRef(myself);
	view->len = self->b_size;

So memoryview(obj) racing ctypes.resize(obj) reads a b_ptr that resize concurrently reallocates.

Reproducer:

import ctypes
from threading import Thread

buf = (ctypes.c_char * 64)()

def viewer():
    for _ in range(20000):
        try:
            memoryview(buf)
        except Exception:
            pass

def resizer():
    for i in range(20000):
        try:
            ctypes.resize(buf, 128 if (i & 1) else 256)
        except Exception:
            pass

threads  = [Thread(target=viewer)  for _ in range(6)]
threads += [Thread(target=resizer) for _ in range(2)]
for t in threads: t.start()
for t in threads: t.join()

TSAN Report:

==================
WARNING: ThreadSanitizer: data race (pid=1524546)
  Read of size 8 at 0x7fffb6b50208 by thread T1:
    #0 PyCData_NewGetBuffer /cpython/./Modules/_ctypes/_ctypes.c:3129:23 
    #1 PyObject_GetBuffer /cpython/Objects/abstract.c:455:15
    #2 _PyManagedBuffer_FromObject /cpython/Objects/memoryobject.c:97:9 
    #3 PyMemoryView_FromObjectAndFlags /cpython/Objects/memoryobject.c:813:42 
    #4 PyMemoryView_FromObject /cpython/Objects/memoryobject.c:856:12
    #5 memoryview_impl /cpython/Objects/memoryobject.c:1017:12 
    #6 memoryview /cpython/Objects/clinic/memoryobject.c.h:63:20 
    #7 type_call /cpython/Objects/typeobject.c:2472:11 
    #8 _PyObject_MakeTpCall /cpython/Objects/call.c:242:18
    #9 _PyObject_VectorcallTstate /cpython/./Include/internal/pycore_call.h:142:16)
    #10 PyObject_Vectorcall /cpython/Objects/call.c:327:12
    #11 _Py_VectorCall_StackRefSteal /cpython/Python/ceval.c:726:11 
    #12 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:4559:35 

  Previous write of size 8 at 0x7fffb6b50208 by thread T7:
    #0 _ctypes_resize_impl /cpython/./Modules/_ctypes/callproc.c
    #1 _ctypes_resize /cpython/./Modules/_ctypes/clinic/callproc.c.h:139:20
    #2 _Py_BuiltinCallFast_StackRef /cpython/Python/ceval.c:817:11
    #3 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:2510:35
    #4 _PyEval_EvalFrame /cpython/./Include/internal/pycore_ceval.h:122:16

SUMMARY: ThreadSanitizer: data race /cpython/./Modules/_ctypes/_ctypes.c:3129:23 in PyCData_NewGetBuffer
==================

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

[devdanzin]

This race is memory-unsafe, not value-benign — it escalates to a crash. The exported b_ptr that PyCData_NewGetBuffer reads is the live buffer pointer; while a memoryview holds it, ctypes.resize() reallocates and frees it, so a write (or read) through the view hits freed heap and corrupts the allocator.

Reproducer (crashes ~every run on a free-threaded --with-address-sanitizer build of main, PYTHON_GIL=0):

import ctypes, threading
buf = (ctypes.c_char * 64)()
N = 60000
def viewer():
    for _ in range(N):
        try:
            mv = memoryview(buf).cast('B')
            mv[:] = b'\xff' * len(mv)          # write through the exported (maybe-freed) b_ptr
            mv.release()
        except Exception:
            pass
def resizer():
    for i in range(N):
        try:
            ctypes.resize(buf, (1 << 20) if (i & 1) else 64)   # realloc frees/moves b_ptr
        except Exception:
            pass
ts = [threading.Thread(target=viewer) for _ in range(6)] + \
     [threading.Thread(target=resizer) for _ in range(6)]
for t in ts: t.start()
for t in ts: t.join()

The SEGV lands inside _ctypes_resize itself — its realloc trips over the freelist that the racing write-through-the-stale-export corrupted:

AddressSanitizer: SEGV ... in mi_block_nextx            (mimalloc freelist traversal)
  ... _mi_heap_realloc_zero   Objects/mimalloc/alloc.c:732
  _ctypes_resize_impl         Modules/_ctypes/callproc.c:1925
  _ctypes_resize              Modules/_ctypes/clinic/callproc.c.h:139

On a TSan build the same script also reports the write side racing the buffer memory (__tsan_memcpy ← copy_base Objects/memoryobject.c:353 ← memory_ass_sub), complementing the read-side report (PyCData_NewGetBuffer:3129) in the issue.

A second, related gap (deterministic, no threads): ctypes.resize() doesn't refuse a cdata that has live buffer exports, so even single-threaded it frees a still-exported buffer (feel free to open an issue for that or tackle it here):

buf = (ctypes.c_char * 64)()
mv = memoryview(buf)          # live export
ctypes.resize(buf, 4096)      # succeeds — no BufferError
bytes(mv[:8])                 # reads freed memory (0xdd... DEADBYTE on a debug build)

Edit: On a GIL --with-address-sanitizer build with PYTHONMALLOC=malloc this aborts deterministically (every run) with a clean use-after-free naming the alloc / free / access:

AddressSanitizer: heap-use-after-free ... READ of size 8
    #1 PyBuffer_ToContiguous   Objects/memoryobject.c:1063        (bytes(memoryview(buf)))
    #2 _PyBytes_FromBuffer     Objects/bytesobject.c:2974
  freed by thread T0 here:
    #0 realloc
    #1 _ctypes_resize_impl     Modules/_ctypes/callproc.c:1925    (resize freed the exported buffer)
  previously allocated by thread T0 here:
    #1 PyCData_MallocBuffer    Modules/_ctypes/_ctypes.c:3293
    #2 generic_pycdata_new     Modules/_ctypes/_ctypes.c:3611

bytearray/array.array raise BufferError: Existing exports of data: object cannot be re-sized here; ctypes has no such guard. The critical-section fix on PyCData_NewGetBuffer closes the concurrent race but wouldn't address this — _ctypes_resize also needs an export/b_locked-count check before it reallocates. Might be worth handling both together.

Reproducer reduced and this comment drafted with AI assistance (Claude Code), then verified by hand on a free-threaded TSan/ASan build of main.