Check this yourself

The first needs a machine with a CUDA GPU and a CUDA build of PyTorch; it fails on purpose without one, and I ran it on an RTX 5080. It counts the calls that make the host wait for the device. A plain copy back to the host waits every time. A non-blocking copy into pinned memory does not.

import warnings, torch
assert torch.cuda.is_available(), "this check needs a CUDA machine"
x = torch.ones(4, device="cuda")
pinned = torch.empty(4, pin_memory=True)

def stalls(fn):
    torch.cuda.set_sync_debug_mode("warn")
    with warnings.catch_warnings(record=True) as w:
        warnings.simplefilter("always")
        for _ in range(5):
            fn()
    torch.cuda.set_sync_debug_mode("default")
    return sum("synchronizing" in str(m.message) for m in w)

print("x.cpu():", stalls(lambda: x.cpu()))
print("copy_ into pinned, non_blocking:", stalls(lambda: pinned.copy_(x, non_blocking=True)))

x.cpu(): 5
copy_ into pinned, non_blocking: 0

PyTorch also prints a warning that this debug mode is a prototype and does not yet detect all synchronizing operations. Keep that in mind: a count of zero means none were detected, and I treat the profiler as the authority on where the time goes.

The second needs only the standard library. It times one function as “A” and the identical function as “B”, in fresh processes, in A B B A order, eight times. The true ratio is 1. Mine, on an ordinary workstation that was doing other work, was not close to 1, and yours will differ from these numbers every time you run it.

import statistics, subprocess, sys
CODE = "import time; t = time.perf_counter(); sum(i * i for i in range(2_000_000)); print(time.perf_counter() - t)"

def run():  # a fresh process every time
    return float(subprocess.check_output([sys.executable, "-c", CODE]))

ratios = []
for _ in range(8):  # eight ABBA groups; "A" and "B" are the very same code
    a1, b1, b2, a2 = run(), run(), run(), run()
    ratios.append(((a1 + a2) / 2) / ((b1 + b2) / 2))
print("A/B per group:", " ".join(f"{r:.3f}" for r in ratios))
print(f"median {statistics.median(ratios):.3f}, spread {min(ratios):.3f} to {max(ratios):.3f}")

A/B per group: 1.197 1.012 0.933 0.790 1.234 0.820 0.794 1.665
median 0.972, spread 0.790 to 1.665

This one runs on the CPU. It shows how large the noise can be, and nothing about a GPU.

The move

Profile before touching anything. A native-stack sampling profiler showed one thread pinned at a full core while the GPU was not fully busy, and most of that thread was waiting on copies from the device. Then I counted the synchronizing calls with the framework’s own debug mode, so the problem had a number I could drive down.

The fix followed from the count. Downloads go into a ring of pinned host slots, each with an event, so the host waits for a slot and not for the whole device. Uploads stage through pinned memory in one non-blocking copy. Both halves are needed, because a non-blocking copy to or from pageable memory still blocks. I judged the change by two things: the count dropping, and the bytes handed to the encoder being identical to before.

Every speed claim after that is timed as base and change pairs, run back to back in A B B A order, a fresh process for each run, on an idle GPU. The benchmark script refuses to start while the GPU is already busy beyond a fixed threshold unless I override it, because a busy GPU makes every reading wrong in a way that looks fine.

Why the obvious version fails

The obvious version is one reading before and one after. The second block above is why that fails: identical code produced readings that differ by far more than most optimizations are worth, so a single pair can show a gain or a loss that is not there. Interleaving cancels drift, such as thermal state or a neighbouring process, that would otherwise be charged to whichever version ran second.

The obvious profile fails the same way. Time spent in a Python line says the line is slow, not that it is waiting. Counting the synchronizing calls tells those apart, and it turns “the GPU feels underused” into something a change can be shown to fix or not to fix.

What this does not cover

• One GPU on one workstation. It is vision inference, not LLM serving, and not distributed.
• The kernel I wrote is in Triton’s Python DSL, not hand-written CUDA C++.
• The depth model and the notebook the project started from are upstream open-source work. The pipeline around them is mine.
• The sync-debug count is a prototype feature that does not see every wait, and the second snippet runs on the CPU. Neither is a GPU benchmark.
• The code is private, so nothing about the pipeline itself can be re-run from here.