ECE 408 · University of Illinois

GPT-2 inference, in CUDA

We took a sequential CPU reference of GPT-2's forward pass and built it up in CUDA. The goal was simple: make it fast. We started with naive kernels, profiled the bottlenecks, and threw everything at it—tensor cores, FlashAttention-2, KV-caching, CUTLASS. Here's how we got a 24.2× speedup.

Read the report Source code

Temperature 1.5 — Aryan Singh, Alen Daniel, Sathya Mitiran, Pranav Balasubramanian

01

Overview

We implemented the full forward pass for all 48 decoder layers of GPT-2 in CUDA, running on NCSA Delta (A40, sm_86).

6,750+
Lines of CUDA / C++
Across baseline + every optimization variant
7
GPU kernels
encoder · layernorm · matmul · attention · softmax · gelu · residual
14
Optimizations
4 required (M2) · 3 required (M3) · 7 we added ourselves
24.2×
End-to-end speedup
Full forward pass, M1 baseline → optimized
48
Transformer layers
GPT-2 1.5B decoder blocks
1.5B
Model params
GPT-2 XL configuration

What we did: Wrote all the GPU kernels, hit all three milestones, and implemented the optimizations below.

What we didn't do: Course staff set up the project and gave us the base scaffolding + sequential CPU reference (built on Andrej Karpathy's llm.c).

02

Architecture

Standard decoder-only transformer. 48 blocks. Each one does masked multi-head self-attention and a GeLU feed-forward network, wrapped in layer norm and residual connections. Pretty standard stuff.

  1. 01Encoder (embeddings)encoder
  2. 02LayerNorm 1layernorm
  3. 03Multi-Head Attentionattention
  4. 04Residual addresidual
  5. 05LayerNorm 2layernorm
  6. 06FFN + GELUgelu
  7. 07Residual addresidual
  8. 08Final LN → Logitslayernorm
Transformer decoder architecture
Decoder-only transformer: masked self-attention + FFN per block.
03

How it progressed

We let Nsight do the talking. Profile, find the bottleneck, fix it, repeat.

  1. 00

    Sequential CPU implementation

    reference

    The starting point. Single-threaded, slow, but correct. We used this to check our GPU work. (Provided by course staff, based on Karpathy's llm.c).

    ground truth

  2. 01

    Naive CUDA baseline

    M1

    One thread per output, all global memory. It worked, but matmul was eating 97% of the GPU time (243 ms). The profile told us exactly where to start.

    matmul 243 ms · 97.4% of GPU time

  3. 02

    Required optimizations

    M2

    We hit the GEMM hard. Tiling, shared memory, TF32 tensor cores. We also rewrote layer norm and softmax using warp reductions. Matmul was no longer the main bottleneck.

    matmul 10.7 ms · 22.8×

  4. 03

    Advanced + competition build

    M3

    This is where it got fun. FlashAttention-2 kept the attention matrix on-chip. KV-caching made generation actually viable. CUTLASS fused a bunch of FFN steps together. End result: 24.2× faster end-to-end.

    full forward 249.8 → 10.3 ms · 24.2×

04

Kernels

The forward pass is basically seven main operations. Here's what each one does and where the bottlenecks were.

    05

    Optimizations

    14 optimizations total. Some were required for the class, some we did just to squeeze out more performance. Expand any row for the details.

    06

    Results

    Numbers from our Nsight profiles and final report. Run on an NVIDIA A40 (batch 4, seq 64) unless otherwise noted.

    07

    Competition build

    For the class speed competition, we just cherry-picked the fastest version of each kernel and slapped them together. comp.sh handles the wiring.

    • Tensor-core TF32 GEMM — matmul / attentionreq_1
    • Warp-reduction kernels — layernorm / softmaxreq_3
    • CUTLASS fused matmul + bias + GeLU — FFN fc1op_16
    • 256-byte aligned host allocation — activationsop_13
    cp kernels_req_1/attention.cuh  kernels/
cp kernels_req_1/matmul.cuh     kernels/
cp kernels_req_3/layernorm.cuh  kernels/
cp kernels_req_3/softmax.cuh    kernels/

make test_gpt2      # + op_16 fused GeLU, op_13 aligned activations
sbatch verify.slurm # profile on a Delta A40

    comp.sh copies the tensor-core kernels (req_1) and reduction kernels (req_3) into kernels/, while gpt2.cuh folds in the CUTLASS fused-GeLU FFN (op_16) and 256-byte aligned activations (op_13). It builds test_gpt2 and runs verify.slurm on a Delta A40.

    Logits OK — max error 0.036, RMSE 0.0012. All tests passed.

    08

    Milestones

    The class broke the project into three graded milestones—baseline, required optimizations, then advanced work and the final report.

    09

    Setup

    Built with nvcc and profiled on the NCSA Delta cluster.

    Hardware
    NVIDIA A40 (Ampere, sm_86) · NCSA Delta · gpuA40x4
    Build
    nvcc -O3 -std=c++17 -arch=sm_86 -rdc=true · Makefile + test.sh
    Libraries
    cuBLAS / cuBLASLt · CUTLASS · wmma tensor-core API
    Profiling
    Nsight Systems · Nsight Compute · Compute Sanitizer · SLURM
    make test_gpt2              # forward pass + correctness vs CPU
make test_gpt2_kernels     # per-kernel verification
make next_token_generation # autoregressive demo (KV-cache)
sbatch verify_nsys.slurm   # profile on Delta