cuda-solver-lab

Conjugate Gradient and Jacobi solvers for the 2D Poisson equation: serial C++, OpenMP, hand-written CUDA kernels, cuSPARSE + cuBLAS reference and Fortran OpenACC, profiled with Nsight Compute. By Pavan Yadav Annappa.

Code on GitHubDev log (bugs found) README and method

Plots

Results

All numbers: measured on NVIDIA L4, CUDA 12.9, 2026-10-03. Generated by scripts/make_report.py from results/raw/.

Hardware ceilings (bw_probe)

Best measured bandwidth: 261.0 GB/s (87.0 % of the 300 GB/s datasheet value; ECC is enabled on this GPU). FP64 FMA peak: 398 GFLOP/s (FP32/FP64 = 64). FP64 ridge point = 398 / 261.0 = 1.53 flop/byte.

CG: time per iteration (ms), fixed iteration count

CG at 4096^2 (16,777,216 unknowns): bandwidth, roofline, speedups

Roofline: the fused CG iteration does 0.25 flop/byte against a ridge point of 1.53 flop/byte, so the attainable rate is bandwidth x intensity = 65 GFLOP/s, 16.4 % of FP64 peak: CG is memory bound by a factor of 6.1, and the only lever is bytes moved per iteration.

Speedups at 4096^2 (best GPU variant = CUDA fused row-strip + CUDA Graph): 25.6x vs CPU serial, 8.2x vs best CPU OpenMP, 2.39x vs cuSPARSE + cuBLAS; CPU OpenMP vs serial: 3.1x.

Fortran at 4096^2: gfortran OpenACC 9.152 ms/iter; Fortran OpenACC nvfortran, GPU 7.014 ms/iter (1.30x faster than gfortran, 1.20x the time of the best CUDA variant, 6.8x faster than best CPU OpenMP); Fortran do concurrent nvfortran, GPU 7.791 ms/iter (1.17x faster than gfortran, 1.33x the time of the best CUDA variant, 6.1x faster than best CPU OpenMP).

Launch-bound regime (256^2, everything L2 resident): stencil 24.8 us/iter -> 20.7 us with a CUDA Graph (1.20x); fused 17.9 -> 16.7 us (1.08x). At 4096^2 the graph changes stencil by 1.001x (launch cost is hidden behind ms-long kernels).

CG time to solution, 1024^2, relative residual 1e-8

GPU solves check the residual every iteration (graph variants every 10), which adds a device-to-host copy per check; it is included in these times.

Jacobi sweep: effective bandwidth (GB/s, 24 B per point update)

Nsight Compute, one launch per kernel at 4096^2 (ncu --set full)

First fused kernel, model traffic 537 MB (read r, p_old; write p_new, Ap): the 1D grid-stride version moves 840 MB (reads 2.01x the model), the row-strip version 633 MB (reads 1.25x); kernel time 3.28 -> 2.46 ms under ncu.

ncu locks clocks to base during profiling, so durations differ slightly from the timed runs. Sectors/request is averaged over all global loads of the kernel: a fully coalesced warp load touches 8 sectors (32 B each) for 8-byte values and 4 for 4-byte values; higher means scattered accesses.

Unprivileged ncu on this host fails with:

==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM

(RmProfilingAdminOnly=1), so the profiles were taken with passwordless sudo.

Nsight Systems kernel census, cuSPARSE + cuBLAS path (n = 1024, 10 warm-up + 100 timed iterations)

13 GPU kernels per CG iteration for the library path vs 2 for the fused variant.

Nsight Systems, Fortran GPU builds (n = 1024, 100 fixed iterations)

Kernel names are compiler generated: gfortran numbers the offloaded regions (MAIN__$_omp_fn$N), nvfortran names them by source line, with a __red kernel finishing each reduction. -Minfo compiler feedback for the nvfortran builds is in results/minfo/.

compute-sanitizer

43 of 44 runs clean (memcheck, racecheck, synccheck, initcheck x every CG variant + Jacobi). Not clean: cg_cuda cusparse racecheck: RACECHECK SUMMARY: 100 hazards displayed (374 errors, 187 warnings).