Repository navigation
cuda.core: a host copy function with the CUhostFn signature for host_launch #3256
Description
Problem
host_launch (#2058) accepts a ctypes function pointer declared as ctypes.CFUNCTYPE(None, ctypes.c_void_p). The driver calls such a pointer directly, with no Python frame and no GIL. That is the right path for a host copy that runs in stream order, such as the pipeline in #2058 (comment): each chunk is a device-to-pinned copy followed by a host function that copies the pinned slot into the destination array, while the main thread keeps enqueuing work.
A pure-Python package has nothing to point that pointer at. A ctypes callback built from a Python function always acquires the GIL. The C library's memcpy takes three arguments, not the one void * of CUhostFn. So the package needs a compiled extension of its own for one memcpy.
Proposal
Provide one C host function in cuda.core with the CUhostFn signature. It reads dst, src, and nbytes from the user_data it receives and copies the bytes. It never enters Python, so it never takes the GIL. Expose it in a form that host_launch and the graph callback methods accept as fn unchanged, together with a small helper that packs (dst, src, nbytes) into user_data bytes. cuda.core copies the packed arguments and owns the copy until the function has run, which is the existing user_data contract for bytes.
The pipeline above then needs no extension: per chunk, a copy_to into the pinned slot, then host_launch of the copy function with the packed arguments.
The same function serves #2989, where a stream-ordered release needs an in-package host function.
Design question
The public form of the function. Two candidates:
- A module attribute: a ctypes function pointer with the declared
CUhostFnprototype. It passes through the existing validation and works in the stream path and the graph path alike. Recommended: it needs no new entry point. - A free function
host_copy(stream, dst, src, nbytes)besidelaunchandhost_launch. It would duplicate the capture handling thathost_launchandGraphBuilder.callbackalready have.
Out of scope
A staged device-to-host copy utility with a pinned staging ring. Slot sizing, slot reuse, and backpressure are application policy and belong to the caller, not to the cuda.core foundation. The copy function is the piece that lets a package build such a ring in pure Python.
Activity
srimanachanta commentedon Oct 28, 2026
@Andy-Jost, I'll take this. I agree with option 1.
_resolve_host_callbackalready accepts aCUhostFnctypes pointer, so a single attribute works withhost_launch, all the graph callback methods and update(). A free function would cover only the stream path.Plan:
#2989 needs a different host function, one that enqueues deferred cleanup rather than copying memory, so I'd drop it from this issue.
On driver 615.78.08, every host function in a context runs on one driver thread, in order, regardless of stream. A 1 s callback on one stream delayed callbacks on 15 other streams by 1 s. A second context got its own thread. So a large copy delays every other host function in its context, including Python callbacks. I'll document this and recommend chunk sizes.
Questions:
Andy-Jost commentedon Oct 28, 2026
Thanks for taking it, @srimanachanta. The plan matches the proposal; answers below.
Name.
cuda.core.HOST_COPY_FNfits the package's precedent for module-level singleton objects:LEGACY_DEFAULT_STREAMandPER_THREAD_DEFAULT_STREAMare exported the same way, and the API guidelines want functions to be verbs, whichpack_host_copy_argsis. Keep both names.Question 1, the argument layout. Make it public. Document the record as three native-width fields in this order,
uintptr_t dst,uintptr_t src,size_t nbytes, with no padding beyond the natural alignment, so an application can pre-pack one record per ring slot and pass its address as an integeruser_data, and can read a pack back.pack_host_copy_argsthen is a convenience over a documented contract, not the only way in.Question 2, what
pack_host_copy_argsaccepts. Accept integers,Buffer, and buffer-protocol objects. From objects, take the address and the size and check bounds; a bounds error at pack time is the one check that costs nothing and catches the common mistake. Do not check host accessibility with a driver query insidepack_host_copy_args: a pack can happen on a callback thread, for example a Python host callback that enqueues the next chunk, and a driver call is not allowed there. Use only what is known without the driver: aBufferknows whether its resource is host-accessible, and a buffer-protocol object is host memory by construction. A raw integer is the caller's responsibility; say so in the docstring.#2989 needs a different host function, as you say; keep it out of this issue.
The per-context host-function thread is worth a paragraph in the docs with the chunk-size advice, since it also serializes the user's own Python callbacks behind a long copy.
srimanachanta commentedon Oct 28, 2026
@Andy-Jost I built #3269 on top of your #3258 rebased on
main. Lmk if you have any feedback, and I will rebase onto main it once #3258 is merged and if any changes are made to #3258.