Inference engineering has quickly become one of the most valuable skill sets in tech.

i'm going to give you a project-only path from zero systems experience to a model service you can deploy, measure, optimize, and defend... you will serve your first model in month one and write a graphics processing unit (GPU) kernel in month two

because inference engineering is the work of making a model meet a service contract under real limits for latency, memory, quality, reliability, and cost

the mistake is studying GPU topics as a playlist instead of carrying one model recipe, one workload, and one set of answer checks through every change

LinkedIn's September 2025 labor-market update said AI engineering postings made up nearly 7% of technical postings on its platform, up 63% year over year...

that is a platform-specific signal about AI engineering overall, not a separate count of inference roles or a promise of a job report

the plan spans 26 weeks at 35 hours per week, or 910 hours... this is a demanding planning estimate, and it has not been validated with a group of beginners

the field guide includes the projects, copyable model recipe, weekly checks, worked calculations, selected resources, and a final independent assessment, so you can read it over several sessions

here's what you'll build, in order:

• the recipe: define the task and the evidence
• the setup: prepare a safe workspace and budget
• the six-month build: progress from a first model to a verified release
• the 26-week route: build and pass each week
• the project release contracts: define each system boundary and proof
• the finish line: handle an unfamiliar problem
• your AI helper: use assistance without losing ownership
• the build sheet: keep settings, checks, and results together

the route starts on your Mac, adds real GPU measurements, and carries the same model recipe through each project

---

the recipe: understand the job

inference means using a trained model to produce an output, such as an answer, a summary, or a tool call

an inference engineer builds the service around that computation and takes responsibility for how it behaves

the questions become concrete:

• correctness: did the model receive the right input and return an acceptable answer?
• speed: how long did users wait as traffic increased?
• capacity: how many requests fit on the available hardware?
• recovery: what happens when a worker disappears halfway through an answer?
• cost: how much did each useful answer cost, including failed work?

your recipe is where those answers live...

start with this folder and fill it as the projects grow:

recipes/model-hardware-workload/
  recipe.yaml
  launch.sh
  smoke_test.py
  benchmark.sh
  prompts/
  evals/
  results/
  README.md

recipe.yaml records the model revision, software versions, hardware, and settings, so a changed dependency cannot quietly become a different experiment

the launch file starts the service, the smoke test checks that it answers, and the application tests check whether its answers are useful

the results folder keeps the raw measurements, while the README explains how to repeat the run and where the setup fails

by the end, each important setting should point to a test that explains why you chose it...

write the task contract before choosing an optimization

for the support-ticket example, define the input as an identifier and a message, and the required output as the same identifier plus a product and a priority

define allowed product and priority values, the rule for missing fields, and whether uncertain cases must be labeled instead of guessed

write a small set of expected answers by hand, including a ticket with no product, one with two plausible priorities, and one with a misleading identifier

use simple exact checks for identifiers and allowed values, then review the fields where language interpretation is required

keep some tickets out of prompt and setting changes until the final quality check

this contract tells the reviewer what the system must preserve when the model, template, quantization, cache policy, or serving engine changes...

---

the setup: make room for the work

this plan assumes 26 weeks at 35 hours a week, which gives you 910 hours for building, experiments, reading, review, and repairs

i'm setting the target at entry-level inference work: deploying a model, diagnosing a bottleneck, making a justified change, and recovering from a failure

the schedule is a planning estimate, and it has not been validated with a group of beginners

keep the pass conditions and give yourself more time if a stage needs it...

35 hours each week is a demanding, near full-time schedule

at 20 hours each week, the same 910 hours take about 46 weeks before holidays, blocked cloud access, or extra repairs

keep the sequence if you have less time, but extend the calendar and keep the pass conditions

do not turn a six-month title into pressure to skip the work that proves a skill

use a normal week like this:

• 18 hours: build and improve the code
• 6 hours: run experiments and inspect results
• 4 hours: learn the concepts needed for the next change
• 3 hours: review the work and explain your decisions
• 4 hours: repair failed checks

your Mac handles the early Python programs, request clients, test data, and reports...

a GPU (graphics processing unit) is a processor built for parallel work, and later projects need real GPU measurements

CUDA, NVIDIA's system for writing code that runs on compatible GPUs, needs a supported device, so rent cloud hardware when you reach that part

plan for a single GPU first, then two GPU workers, plus one experiment across two separate computers

prepare tests before renting machines, set spending and device limits, then check that machines and paid storage stop when the session ends

the full 26-week schedule, recipe template, cost worksheet, project checks, resource path, and assessment are included below

---

choose hardware for the question

• Mac CPU (central processing unit): build references, parsers, simulators, tests, and reports
• one supported NVIDIA GPU: run CUDA and Triton, a Python-based language for writing GPU kernels, profile, serve a model, and inspect cache pressure
• two real GPU workers: test replicas, sharding, experts, scaling, and cache handoff
• two actual computers: measure one real network-bound communication or serving case

a Mac result and a CUDA result need separate recipes because the hardware and execution paths differ...

a simulation can test a decision rule, but it cannot establish GPU speed, physical memory use, or live cache eviction...

---

estimate cloud spending before a session

use the provider's current written rates to estimate your session cost

GPU type, region, minimum billing time, storage, and transfer charges change the total

estimated session cost =
  single-GPU hours × single-GPU hourly rate
  + two-GPU wall-hours × 2 × per-GPU hourly rate
  + two-host wall-hours × 2 × per-host hourly rate
  + storage + transfer + other charged resources

an example booking budget is 80 single-GPU hours, 20 two-GPU wall-hours, and 8 two-host wall-hours, before storage or transfer

this example is a planning allowance, not a price quote or a guarantee that the course fits those hours

record currency, region, device type, minimum billing period, and whether the host price already includes its GPU

if the host rate already includes the GPU, count the host price once and do not add a second GPU charge

prepare inputs locally, run a short smoke experiment, set a shutdown deadline, export your results, and check for persistent disks or endpoints

start with one model and one serving engine, the software that runs the model and handles incoming requests

you can add more choices after you can explain the first one

prepare the inputs on your Mac, measure the hardware path you intend to claim, then export the evidence and check every paid resource

---

the six-month build

---

month 1: run a model correctly

weeks 1 to 4 · project 1: your model recipe and change reviewer

give the service a small job: read a synthetic support ticket and return its identifier, product, and priority

write the expected answers yourself, including missing fields and confusing identifiers, then keep some cases aside for the final test

this gives every later optimization something to protect... the service still has to complete the original job

1. build the first pieces in Python

Python is the programming language you'll use for request handling, tests, and experiment scripts

in your first week, write a program that reads a ticket, extracts its fields, handles a missing value, and saves the result

learn strings, lists, dictionaries, loops, functions, and file handling through that program

use CS50's Python course when you need an explanation or practice for the next step

put the work in Git, the version-history tool that lets you save changes, compare them, and return to a working version

then build a small test server that sends an answer in chunks, pauses, or fails, and write a client that records what arrives

an API is the interface another program uses to talk to your service, and this client is your first way to test it...

2. learn the system around the program

Linux is the operating system you'll use on the cloud machines, so learn it through the service you are preparing to run

find its process and port, read its logs, inspect memory use, repair a file permission, and stop it cleanly

use The Missing Semester's shell lesson to learn the commands as those tasks appear

record dependencies, recreate the environment, and automate the parser tests so a code change can reveal a broken assumption

package the working service in a container, which bundles its software and runtime, then check that another machine can start it

keep the learning endpoint private and credentials outside the repository

use synthetic tickets for the learning workload, and check a model's license before redistributing its weights or a hosted service

do not copy private customer conversations into a prompt, issue, benchmark, or assistant session unless you have permission and an approved data path

keep keys in a local secret store or cloud secret manager, remove them from logs, and set request and spending limits before testing

model output is untrusted input: parse it, validate its fields, and keep tool execution behind explicit application checks...

3. turn the test service into a model service

compare vLLM, an open-source library for model inference and serving, and SGLang, another system for serving models, on model and GPU support, template handling, streaming, and available measurements

Text Generation Inference (TGI), Hugging Face's model-serving toolkit, is a useful third comparison

the repository was archived on March 21, 2026, and its maintainers describe the project as in maintenance mode, so use it as a dated comparator instead of the main learning engine...

Qwen3-0.6B is one small starting candidate, with a model card that describes its input requirements and generation modes

choose vLLM or SGLang after checking the model against current support documentation, then record the exact model revision...

a tokenizer converts text into pieces called tokens, while a chat template adds the markers that identify each speaker

inspect the actual tokens using the chat-template guide, because the same visible conversation can be formatted differently for different models...

record the template, stopping rules, output parser, and any reasoning-mode setting in your recipe

use vLLM's recipes as references, credit what you adapt, and explain your changes for this model and task

MLX-LM, a library for running models on Apple silicon, gives you a local path with its own recipe and comparison results

the pass condition: another person can start the service from your instructions, run the answer checks, and identify the exact model and settings

now you have a working service... the next step is finding out what happens when more people use it

follow one request from formatted input to checked output... prefill builds the prompt state, then decode extends the answer using that state

---

month 2: measure performance and write a kernel

weeks 5 to 8 · finish project 1 and build project 2: the GPU performance investigator

your change reviewer reads the recipe, measurements, and answer checks, then returns one of three decisions: supported, hold, or insufficient evidence

it should catch a misleading improvement before that improvement reaches the deployment

1. make the measurements trustworthy

use AIPerf, a tool that sends requests and records performance, to generate load for your service

your own code checks the records and decides whether the proposed change helps the application

keep these measurements separate:

• time to first token, or TTFT: the wait before the first generated content, with the timer's start point stated
• inter-token latency, or ITL: the gaps between generated tokens, which require token-level observations
• throughput: the amount of work completed per unit of time
• useful goodput: completed requests that meet your answer-quality and delay requirements, per unit of time

a network chunk can contain several tokens, so label client chunk gaps as chunk gaps when individual token times are unavailable...

send short chats, long documents, and structured-output requests, then increase the number of active requests and plot how the service changes

also schedule arrivals independently of completion, so a slow service cannot quietly reduce the offered load

count every scheduled request, including failures and missed sends, and record when the load generator itself falls behind

plant a false win by shortening the candidate's answers or dropping failed requests... your reviewer must reject the comparison

2. understand the computation you're timing

build a tiny decoder with fixed synthetic weights on the CPU, the general-purpose processor in your computer

this small model checks the math behind token generation, while the downloaded model handles the language task...

implement matrix multiplication, normalization, attention, a feed-forward block, and output scores, checking small examples by hand

normalization rescales values, while softmax turns a set of scores into probabilities

learn array dimensions, dot products, probabilities, and numerical error through those operations

attention lets each token use information from other positions, while a causal mask prevents it from reading future tokens...

prefill processes the input, decode generates the next tokens, and the key-value cache, or KV cache, saves attention data for reuse

compare cached generation with a full-sequence reference, using an explicit tolerance for small numerical differences

3. write and investigate one GPU operation

a kernel is a program that runs on the GPU, and softmax or normalization makes a useful first operation...

write a reference in C++, a lower-level programming language, then a CUDA version, learning pointers, indexing, parallel threads, and synchronization

then use Triton, a language for writing GPU kernels, to implement and compare another version

test awkward dimensions, partially filled blocks, zeros, and extreme values, with an independent reference and declared numerical tolerances

use nvidia-smi for device identity and basic activity, then Nsight Compute, NVIDIA's kernel profiler, to inspect the expensive operation

connect each hardware term to something you measure:

• bandwidth: how quickly data moves through a memory path
• arithmetic intensity: how much computation you perform for the data moved
• occupancy: resident thread groups, called warps, compared with the maximum each streaming multiprocessor supports
• tensor cores: specialized hardware for supported matrix operations

study a small tiled matrix multiplication, then compare separate operations with fusion, which combines operations to reduce intermediate work...

try a CUDA graph, which replays a prepared sequence of GPU operations, and count its setup time and retained memory

the pass condition: your reviewer catches the false benchmark, your kernel passes new inputs, and a profile supports your explanation

compare against a suitable library implementation, time final runs outside the profiler, and keep a fallback wherever your version loses

an isolated kernel result only supports a service-speed claim after you integrate and measure it in the service...

next, make the memory behind that service predictable

---

month 3: manage memory and test faster generation

weeks 9 to 12 · projects 3 and 4: the context memory manager and model setting selector

longer conversations leave more state to store, and that state competes with model weights and temporary computation for GPU memory...

1. predict what fits

for a decoder using ordinary full attention, start with this logical key-value (KV) cache estimate:

KV bytes = 2 × layers × cached tokens × KV heads
           × head dimension × bytes per element

the leading 2 counts the key and value tensors, while the other terms describe the model, sequence length, and storage format...

sum across requests before accounting for shared data, and use a different formula when the model stores a different kind of state

compare the estimate with live measurements, including weights, temporary work, reserved space, and partly filled cache blocks...

the block IDs show logical order mapped to physical storage... shared prefixes occupy one set of blocks, while each request keeps its own tail

PagedAttention describes how an engine manages cache in blocks, making it a useful reference for this experiment

build a rule that admits requests only when the measured capacity and declared input/output limits allow them

reduce the cache budget, add pressure, and record which requests wait, lose saved state, or repeat earlier work

2. make reuse safe

prefix caching reuses computation for matching input prefixes, which means reuse depends on compatible tokens, model revisions, and execution settings...

check the model and adapter revisions, token IDs, token positions, attention mask, cache layout, and any attention setting that changes the stored state

do not route to a cached prefix just because the visible text looks the same...

use the vLLM prefix-cache design to build your compatibility checks

make a small simulator for allocation, reuse, cancellation, and release, including messages that arrive after a request has ended

then test the real engine separately, checking memory recovery and output agreement against fresh-state runs

state which internals the engine exposes, because simulator checks cannot establish what unobserved engine code actually did...

if your engine supports it, add a bounded CPU-memory cache with LMCache, a system for storing and moving reusable model state...

compare restoring state with recomputing it, including transfer, reconstruction, and queue time

3. select faster settings without losing the task

quantization uses fewer bits to represent numbers, so begin by writing a tiny quantizer that packs values and reconstructs them

then compare supported model configurations, recording the formats used for weights, intermediate values, accumulated results, and cached state separately

FP16, FP8, and INT8 are number formats, while AWQ is a method for choosing how to quantize weights

keep data used to choose quantization settings separate from tuning inputs and the final test set...

speculative decoding adds a draft model that proposes tokens for the target model to check...

implement a tiny acceptance-and-rejection example from the speculative decoding paper, then run one compatible draft/target pair

the exact method preserves the target distribution under its assumptions, and your implementation still needs checks for rejection handling and state rollback

measure the whole request, including rejected draft work, and keep a case where ordinary decoding is preferable...

use XGrammar, a tool for constraining output format, to test structured responses

check the actual field values after checking the format, then use a local tool emulator to verify arguments and final state

the pass condition: explain the memory gap, catch invalid reuse, and accept or reject each generation change using both quality and performance evidence

your service now has limits it can explain... the next problem is sharing those limits between users

---

month 4: share capacity and account for cost

weeks 13 to 16 · projects 5 and 6: the shared scheduler and ready model pool

a scheduler chooses which work runs next, while a pool controller decides how much model capacity should be ready

1. keep long requests from taking over

mix short conversations with long documents, then measure what happens to answers already being generated

continuous batching lets requests enter and leave a running batch, while chunked prefill divides a long input into smaller pieces of work

use Sarathi-Serve, a serving-system paper, to study how those scheduling choices affect delay and throughput...

write your own admission or request-selection rule, then connect it to a real gateway queue or supported engine hook

measure waiting time and completed work per tenant, meaning each customer or isolated user group sharing the service...

run two real replicas, which are separate copies of the model service, and route matching prefixes toward reusable cache

then add queue delay to that decision... the worker with the useful cache may already have the longest wait

2. prepare capacity before the queue wins

give workers explicit states: absent, staging, loading, ready, draining, and failed

use a new generation identifier for each start attempt, so a late message from an old worker cannot make its replacement ready...

measure download, loading, compilation, and warmup separately, then require a successful inference check before a worker accepts traffic

compare fixed capacity with scaling based on queued work, request age, and measured startup time...

test a burst that ends before the extra worker becomes ready, because that delay changes the value of the scaling decision

use Kubernetes, a system for managing containers, for the two-worker deployment

KEDA, the Kubernetes Event-driven Autoscaler, can scale replicas from external signals, while acquiring new GPU machines needs its own capacity mechanism

ServerlessLLM gives you a model-loading and placement design to compare with your measurements

continuous batching changes which requests occupy a batch... chunked prefill changes how much new-input work enters an iteration

3. make the bill reconcile

count startup, warmup, idle time, failed work, retries, storage, and transfer charges where they apply, using dated rates or actual bills

report cost per million output tokens and cost per useful completed answer, keeping the definition of useful tied to quality and delay

allocate shared costs with a stated method, label estimates, and check that tenant allocations plus shared overhead equal the resource total

NVIDIA Management Library (NVML) is NVIDIA's library for querying GPU status, and its GPU-utilization measure reports time with kernels active

model-FLOPs utilization, or MFU, compares estimated model computation with the hardware's applicable peak, so report its assumptions and precision

the pass condition: explain the cache-versus-queue tradeoff, reject stale worker messages, and reconcile the cost report

now make the same service behave sensibly when a worker fails during a request

---

month 5: recover from failure and use several devices

weeks 17 to 20 · projects 7 and 8: the request gateway and device placement planner

the gateway sits between clients and model workers, making it responsible for limits, routing, and what users see when an attempt fails

1. define the rules for an interrupted answer

give each user request an identifier and each execution attempt its own identifier, then save their events in a durable journal

make journal updates atomic with database transactions, and test a restart between two updates

before output reaches the client, allow a compatible retry within the original deadline and retry budget

once output becomes visible, report an interruption or start an explicit new attempt, because silently joining two generations can change the answer

check a fallback's task quality, input limits, and output format before treating it as a substitute

add authentication and request limits, separating request rate, total usage allowance, and simultaneous active requests

send cancellation to the worker, account for work that continues, and keep uncertain outcomes visible until you can resolve them

SGLang's gateway, a request router from another model-serving system, provides components you can study and reuse

2. break the service on purpose

kill a replica, delay replies, abandon a stream, and deploy a worker that fails its readiness check

record affected requests, recovery time, resource cleanup, and the return to the previous working recipe

a service-level objective, or SLO, defines the behavior you aim to provide, such as acceptable response delay...

use Google's site reliability engineering (SRE) workbook to connect failed requests with the service's allowed error budget

3. measure the cost of splitting work

return to your tiny decoder and implement two ways to share its work across devices

tensor parallelism splits operations between devices, while pipeline parallelism puts different model stages on different devices...

compare both with the single-device reference, then compare two replicas with a split model using the same total hardware allocation

use NCCL tests, benchmarks for the NVIDIA Collective Communications Library (NCCL), to measure the messages your design actually needs

add a small mixture-of-experts model, which routes tokens to selected submodels called experts

use synthetic weights to check routing, reordering, expert execution, and output combination, including empty experts and one overloaded expert

run the path across two real devices, then run one communication or serving experiment across two separate computers

record the actual connection, because results from ordinary Ethernet cannot establish the performance of a different interconnect...

the pass condition: recover without breaking the client rules, match the distributed reference, and explain a case where communication removes the speed benefit

you can now split computation... next, test whether splitting the input and output phases is worth the state transfer

---

month 6: move model state and publish the evidence

weeks 21 to 26 · project 9: the prefill/decode controller, followed by the final release

disaggregated inference assigns input processing and output generation to separate workers, which requires moving the right saved state between them...

use NVIDIA Dynamo, a system for coordinating inference workers, with a supported engine integration to build the experiment

NIXL, a data-transfer library for inference systems, supplies components for moving cache state between workers

1. make the handoff correct

write the controller around this sequence:

1. reserve space on the destination
2. transfer the cache state
3. verify the request, model, format, and worker generation
4. commit the destination state for use
5. confirm the handoff
6. release resources that are no longer needed

cancel during transfer, lose a confirmation, and deliver a completion from an old worker, checking eventual cleanup each time...

then transfer real cache data between GPU workers and compare with the same model running without the split...

compare next-token scores within declared tolerances where available, or deterministic continuations where the engine only exposes output

state the weaker coverage of output-only checks, vary prefixes and context lengths, and include an invalid-state case your checks must reject

if transfer compresses or transforms cache values, define a separate numerical-error and task-quality contract for that transformation...

2. decide whether the split helps

compare split workers with colocated serving, where both phases share workers, using the same total GPU allocation

count transfer time, queue time, first-token delay, the next-token gap, useful goodput, and cost

your result may favor keeping the phases together... that is a useful deployment decision when the evidence supports it

this is the project's simplified handoff contract... compare output with an unsplit reference under the recipe's correctness rules and numerical tolerance

Prime Inference is a production case study connecting scheduling, cache layout, compression, and tool-call parsing

choose one reported bottleneck and reproduce a small version, while treating Prime's reported gains as results from its own setup

3. prepare the release someone else can run

in weeks 23 and 24, write a second recipe for another model or hardware constraint, then repeat the correctness and performance checks

a small image-and-text model is one option, with separate measurements for image preparation, encoding, waiting, and generation

freeze the workload and compare three configurations:

1. the engine's standard setup
2. the same engine with tuned settings
3. the tuned setup with your own change

keep other conditions comparable for the question under test, repeat runs in different orders, and retain failures and normal run variation

use AIPerf's repeated-run reporting to help present uncertainty instead of selecting the best run

include workloads where your change loses, and keep final test inputs separate from the inputs that guided tuning

have another person reproduce the release from a clean environment, then repair the instructions wherever they needed private help

the pass condition: two executable recipes, a reproducible comparison, a failure report, and a deployment recommendation supported by the raw results

reserve week 25 for repairs and a practice assessment, then use week 26 to find out how much of the work you can carry independently

---

the 26-week route: what to build and prove each week

each week ends with a working change, a test that could reveal a mistake, and a decision you can explain

an ordinary week uses 18 hours to build, 6 to run experiments, 4 to learn a needed concept, 3 to review, and 4 to repair

arrange a peer or mentor review before you start

review hours include your preparation and response, not free reviewer labor

every second week, ask the reviewer to change an input or operating condition

then find and explain the cause yourself

---

month 1: foundations and the first recipe

week 1 · own the experiment record

Build: write a Python program that reads synthetic tickets and a settings file, checks required fields, compares two configurations, and saves a run record in Git

Pass: run from a clean directory, reject a broken configuration with a useful error, and repair a reviewer-selected parser bug

Keep: the program, fixtures, tests, and first release · CS50 Python · Mac CPU

week 2 · understand a streaming request

Build: a local HTTP (Hypertext Transfer Protocol) service that sends chunks, pauses, fails, or disconnects, plus a client with request IDs, deadlines, cancellation, and raw event times

Pass: account for every scheduled request, and make sure one stalled request cannot erase another request's outcome

Keep: client, failure examples, event format, and timing checks · Python asyncio · Mac

week 3 · operate a Linux process

Build: package the service and run it on a Linux CPU machine

inspect its process, port, permissions, memory, logs, and shutdown

Pass: diagnose a port conflict or missing file, stop the process cleanly, and repeat tests from a clean checkout

Keep: launch and stop procedures, container definition, and troubleshooting notes · The Missing Semester · Linux CPU is enough

week 4 · serve the first model

Build: serve Qwen3-0.6B with supported vLLM or SGLang settings on a compatible GPU, using synthetic ticket inputs with expected answers

Pass: record the exact model revision, runtime, tokenizer, chat template, generation mode, stopping rules, and answer quality

confirm continuous batching is active in the supported runtime and record the scheduler settings

then send staggered concurrent requests and save server-side scheduler evidence showing a new request admitted while another is still decoding, and finished work releasing capacity while peers remain active

client-side overlap alone does not prove continuous batching

if the engine exposes no supported scheduler trace or metric, mark the claim unverified until you add instrumentation

Keep: executable recipe, smoke test, quality baseline, and independent review · Qwen3-0.6B · one GPU

---

month 2: measurement and GPU execution

week 5 · build the deployment reviewer

Build: use AIPerf, which sends requests and records performance, or GuideLLM, a tool for generating controlled workloads against model endpoints, then import the runs into your reviewer

compare model settings and answer quality, then plot rising concurrency and scheduled arrivals

Pass: reject false wins caused by shorter answers, missing timeouts, changed checkpoints, different timing boundaries, or a saturated load generator

Keep: short-chat, long-input, and structured-output load curves with raw events · one GPU

these curves explain the measurement, not a measured result... the shared vertical line marks the first binding latency limit, and your operating point needs headroom

week 6 · explain the computation

Build: a tiny CPU decoder with fixed weights, attention, normalization, a feed-forward block, and cached and uncached generation

Pass: check small calculations by hand, explain prefill and decode, and match both paths within a declared numerical tolerance

Keep: decoder reference, shape notes, tests, and arithmetic worksheet · The Scaling Book · Mac CPU

week 7 · write a CUDA operation

Build: implement softmax or normalization in CUDA against an independent CPU reference, then try a small tiled matrix multiplication

Pass: check uneven shapes and boundary values, diagnose one indexing or synchronization error, and use Compute Sanitizer for memory faults

Keep: kernel, independent reference, sanitizer output, and baseline times · NVIDIA CUDA notebooks · one GPU

study how warps, blocks, registers, shared memory, and global memory work together before trying to tune occupancy or tensor-core use

week 8 · choose when your kernel runs

Build: implement the operation in Triton and add a dispatch rule for supported input shapes

Pass: new shapes either meet the correctness contract or take a fallback

a profile must support your explanation of the bottleneck

Keep: execution investigator, profile, and a losing or inconclusive case · Nsight Compute · one GPU

---

month 3: memory and answer quality

week 9 · compare predicted and physical memory

Build: calculate logical KV size, simulate cache pages, and inspect live engine reports for allocation, pressure, and eviction

Pass: record physical page use under pressure and explain rounding, shared blocks, weights, workspace, and any unobserved state

Keep: calculator, monitor, discrepancy report, and pressure trace · PagedAttention · one GPU

week 10 · govern reuse and admission

Build: set an admission limit and identify reusable prefixes by model revision, adapter, tokens, positions, and cache layout

Pass: a simulator rejects stale reuse after cancellation or a late message

the live engine must separately pass tests for memory release, changed prompts, and answers against fresh runs

Keep: context governor and failure replay · vLLM prefix caching · GPU and host memory

week 11 · measure a precision change

Build: pack and restore a small quantized tensor, then compare the model paths your hardware and engine actually support

Pass: compare answer quality, delay, throughput, memory, and cost with separate tuning and final data

mark formats unsupported when your hardware or engine cannot run them

Keep: precision decision and protected quality report · study AWQ, SmoothQuant, and KIVI, a paper on two-bit key-value cache quantization · compatible GPU

week 12 · test speculative and constrained generation

Build: calculate token acceptance and rejection in a small example

then run a compatible draft and target model and test structured output

Pass: check state rollback, keep a workload where speculation loses, and score tool choice and arguments in addition to valid JSON

Keep: generation policy, cost record, and local tool tests · speculative decoding and XGrammar · compatible GPU

---

month 4: scheduling and economics

week 13 · schedule requests with different needs

Build: mix short chats with long inputs and outputs

connect your admission or tenant-selection rule to a real queue or engine hook

Pass: measure chunked-prefill effects on active decodes, test a noisy tenant and a wrong cost estimate, and count every offered request

Keep: scheduler and tenant-level curves · Sarathi-Serve · one GPU

week 14 · compare cache locality and queue delay

Build: route requests over two real replicas, comparing prefix-only, queue-only, and bounded routing under bursty traffic

Pass: show when waiting for a warm prefix costs more than using a free replica

use engine reuse evidence where available

Keep: routing policy and a case where locality loses · SGLang gateway and ServeGen · two GPUs

week 15 · manage the model lifecycle

Build: implement absent, staging, loading, ready, draining, and failed states with a new generation ID for each start

Pass: require an inference probe before readiness, reject old acknowledgements, and explain what happens when new capacity starts too late

Keep: pool controller and cold/warm traces · ServerlessLLM · one, then two GPUs

week 16 · scale and reconcile cost

Build: compare fixed capacity with queue-based scaling and record cost by tenant and model

Pass: reconcile startup, idle time, retries, failures, storage, and transfer with the bill

report cost per million tokens and cost per useful answer

Keep: scaling timeline, cost ledger, and MFU assumptions · KEDA · two GPUs

---

month 5: reliability and distributed systems

week 17 · define what a retry means

Build: give each logical request and execution attempt its own ID

record their state changes in a transactional journal

Pass: before output reaches the client, retry only within the original deadline and retry budget

after output appears, report an interruption or start a separate attempt

Keep: gateway contract, journal, and cancellation tests · SQLite transactions · local fixtures and a live endpoint

one user request can have several attempts... the gateway must record when output becomes visible and preserve that boundary during recovery

week 18 · prove the service can recover

Build: kill a replica, delay replies, disconnect a client, fail readiness, and test a deployment rollback

Pass: preserve every request outcome, clean resources, and explain recovery time and SLO error-budget use

label simulated GPU delays as simulations

Keep: repeatable chaos suite, incident timeline, and rollback runbook · Google's SRE workbook · two workers

week 19 · compare replicas with sharding

Build: implement small tensor-parallel and pipeline-parallel decoder paths, then measure real device communication

Pass: match a single-device reference, compare equal GPU budgets, and run one communication experiment across two real hosts

Keep: placement recommendation, topology, transport, and correctness tests · Megatron-LM and NCCL tests · two GPUs plus two hosts

week 20 · move expert work

Build: create a tiny mixture-of-experts path with routing, token reordering, execution, and weighted combination

Pass: test empty and overloaded experts across two devices

state what synthetic weights cannot prove about a trained model

Keep: topology planner and workload-skew fixture · EPLB, an expert-parallel load-balancing project, and DeepEP, a communication library for moving token work between experts · two GPUs

---

month 6: handoff, transfer, and assessment

week 21 · make cache handoff recoverable

Build: reserve, transfer, verify, commit, acknowledge, and release cache state using request, model, layout, and worker-generation IDs

Pass: cancellation, lost acknowledgements, and stale messages cannot attach wrong state or leak reserved memory

Keep: handoff controller and live transfer trace · NIXL and Dynamo's design · two GPUs

week 22 · decide whether to split prefill and decode

Build: compare split and colocated workers with equal GPU resources, the same model, and the same workload

Pass: compare transferred state with an unsplit reference, and test mismatched state and worker loss

report TTFT, next-token delay, useful goodput, and cost

Keep: recommendation and raw results

a sound decision may keep workers colocated · DistServe · two GPUs

week 23 · transfer the recipe to another model

Build: make a second recipe for a different model or hardware constraint

a small image/text model is one option

Pass: retest template, parser, precision, and memory assumptions

if you choose a vision model, separately measure image upload, processing, encoder, queue, and generation

Keep: second recipe and transfer analysis · vLLM model recipes and the selected model card · compatible hardware

week 24 · publish a reproducible comparison

Build: compare the stock engine, tuned settings, and your own change, saving raw results and failures

Pass: repeat runs in a changed order, report losing cases, keep final inputs out of tuning, and have another person repeat the method

Keep: three-configuration report, benchmark package, and two changes you can defend · AIPerf repeated-run reporting

week 25 · repair and rehearse

Build: close failed gates, rehearse a changed model or condition, then add a regression test for a repaired fault

Pass: follow the clean-run instructions and explain each important recipe setting

Keep: repaired release and assessment environment · spend cloud time only where a required check still needs it

week 26 · demonstrate independence

Build: launch the recipe against a new deployment constraint and two faults chosen by the reviewer

Pass: explain the diagnosis and repair, pass every correctness gate, and retest any repaired skill with a fresh case

Keep: two recipes, benchmark release, reviewer record, and technical defense

documentation is allowed

do not use a generated finished solution during assessed work

the final week uses the same 35 hours: 4 for setup, 18 for assessed work, 3 for defense, 4 for repairs, and 6 for fresh retests

book an independent reviewer early, share the system requirements and known assessment boundary, and let them choose the fault locations and changed constraints

---

the graduation rubric

software and deployment: start from a clean environment, diagnose a fault, add a regression check, and stop the service cleanly

model and memory: handle the task correctly after a changed template, context, position, or cache identity

measurement: account for all requests, catch a confound, rerun fairly, and limit your claim to what the measurements show

GPU execution: meet the numeric contract on a new input shape, profile the actual path, and name a valid fallback

reliability and distribution: preserve request behavior through cancellation, restart, stale messages, or lost transfer acknowledgment

transfer and judgment: adapt the recipe to a new supported model or constraint, include an optimization that loses, and defend the recommendation

all six domains must pass

a high score on one area cannot compensate for incorrect output, misleading measurements, or simulator-only GPU claims

keep the initial diagnosis, experiment trace, failed gates, repairs, fresh retests, and the result that would change your recommendation

---

the project release contracts

the weekly route tells you when to build each piece

these contracts tell you which responsibility belongs to each system and what evidence lets another engineer trust it

---

use three levels of evidence

level 1 · prove the rule: test a small component with known inputs, including boundary cases and invalid states

a simulator is useful here because you can explore many cancellations, queue shapes, or allocation patterns cheaply

level 2 · prove the component: connect the component to the real engine or GPU path and check the behavior it exposes

this catches differences between your model of an engine and the behavior that its API actually makes observable

level 3 · prove the service: run the full system with real requests, realistic load, quality checks, and the target hardware

only this level supports a live serving claim, and a simulated network or device cannot substitute for a real performance measurement

label evidence with the level it reached, the date, the exact versions, and what you could not observe

each level answers a different question... a rule that passes in a simulator still needs evidence from the real component and the complete service

---

project 1: the model recipe and change reviewer

the recipe owns the identity of one model, task, runtime, hardware setup, and workload

the reviewer compares a candidate with a named baseline and returns supported, hold, or insufficient evidence

it must catch a changed model or tokenizer revision, a different chat template, missing requests, malformed answers, and measurements taken from different timer boundaries

its decision record links to the recipe, tests, raw events, quality checks, hardware trace, and run configuration

the release gate is a clean run from a fresh checkout by another person who can explain how the model input was formed and where the result came from...

---

project 2: the GPU performance investigator

the investigator owns the path from scheduled request to completed answer, including the load generator's own health

record offered load, active requests, queue delay, time to first content, token gaps when truly observable, output tokens, failures, and useful completions

keep client chunk gaps separate from inter-token latency, because one network chunk can contain several tokens

include a traffic pattern that exposes long prompts competing with short generations, and one that creates a sudden burst

the release gate is a repeated load curve with raw request events and enough detail to explain whether the server or client became the bottleneck...

---

project 3: the context memory manager

the memory manager owns admission decisions when model weights, temporary work, and request state compete for limited memory

its input includes attention layout, context and output limits, cache format, active requests, and measured memory headroom

its output says whether to admit, wait, reject, evict, or reuse state, with a reason that can be inspected

test a full cache, partly filled blocks, cancellation during work, an edited prefix, and a late completion from an old request

only reuse state when model and adapter revisions, actual prefix tokens, positions, masks, cache layout, and relevant attention settings are compatible

the release gate pairs arithmetic and simulator tests with live engine evidence, and labels which internals the engine does not expose

---

project 4: the model setting selector

the selector owns a controlled comparison of settings such as quantization and speculative decoding

each candidate uses the same task, model identity, arrival pattern, answer limits, and quality contract as its baseline

keep prompt development and threshold selection separate from a protected evaluation set that you open only for the final decision

measure quality, first-token delay, decode speed, useful goodput, memory, and cost together, including rejected draft tokens and failed requests

keep one case where each optimization loses, such as a weaker answer, extra draft work, a longer prompt, or a slower full request

the release gate is a supported recommendation only when quality rules pass and the performance result repeats within the stated uncertainty

---

project 5: the shared scheduler

the scheduler owns which request gets the next unit of GPU work...

record arrival time, queue age, prompt length, remaining output budget, tenant, deadline, cache locality, and work already completed

compare a simple first-in-first-out policy with chunked prefill and a policy that considers request age or tenant fairness

test whether a long input delays requests already decoding, whether short jobs starve, and whether cache locality creates an unfair queue

publish per-tenant results beside the overall rate so a fast average cannot hide one group's delay

the release gate shows both an improvement and a losing case, with a rule for when the scheduler should fall back to the simpler policy

---

project 6: the ready model pool

the pool controller owns the transition from no capacity to a worker that passed an inference readiness check

track each worker through absent, staging, loading, warming, ready, draining, and failed states

assign every launch a new generation identifier, so an old worker cannot report ready after it has been replaced

measure image startup, weight loading, compilation, warmup, queue delay, and shutdown separately

compare a fixed pool with queue-based scaling under a burst that ends before new capacity is ready

the release gate includes cold-start effects and confirms machines, disks, and endpoints stop after the run

capacity is usable only after the current worker passes an inference probe... a late ready message from an earlier launch must be rejected

---

project 7: the request gateway

the gateway owns the client-visible contract across routing, retries, fallbacks, limits, and cancellation

give a user request one stable identifier and every backend execution a distinct attempt identifier

record when generated content becomes visible, because retrying before that point differs from replacing a partial answer

test a worker loss before output, a loss after output starts, a timed-out cancellation, a duplicate client retry, and a fallback with a different output format

never join partial answers from separate attempts without an explicit protocol that makes this safe

the release gate proves each request ends as completed, rejected, cancelled, timed out, or unresolved rather than silently disappearing

---

project 8: the device placement planner

the planner owns a measured choice between keeping a model together, splitting it across devices, or running separate replicas

describe the model size, parallelism strategy, GPU memory, device topology, network, workload, and request limits for each candidate

compare tensor parallelism, pipeline parallelism, and replicas before adding a small mixture-of-experts path

use the CPU decoder to test partition and combine logic, then measure communication and serving behavior on real devices

include a network-bound case and an awkward split that adds transfers without reducing the critical computation

the release gate gives the full hardware allocation and explains whether a speed change came from parallel execution, extra capacity, or the interconnect

---

project 9: the prefill/decode controller

the controller owns the handoff of request state from input processing to output generation

reserve destination memory before transfer, identify the request and model generation, verify the received state, and only then make it available to decode

test cancellation during transfer, a repeated message, a lost confirmation, a full destination, and a late response from a replaced worker

compare the split with colocated serving using the same total GPU capacity and workload

report transfer time, queue time, first-token delay, token gaps, useful goodput, quality, and cost

the release gate may recommend against the split when state movement costs more than separating the phases saves...

---

use one experiment contract for every optimization

before changing code, write down the question you want the run to answer

then record:

1. the prediction: what should improve, what could get worse, and why
2. the baseline: the exact working recipe and source revision you will compare against
3. the workload: prompt mix, context lengths, output limits, request arrivals, and quality checks
4. the change: one primary setting or code path, with other differences listed
5. the measurements: timers, token observations, resources, errors, cost, and useful-completion rules
6. the decision rule: the quality floor, performance target, and uncertainty you will accept
7. the reversal case: evidence that would make you reject or undo the change

choose the decision rule before seeing the final evaluation results

otherwise, it is easy to lower the quality bar or redefine a win after a promising graph appears

keep a protected set of task examples outside the data you use to tune prompts, thresholds, or quantization settings

use short development examples to debug, a separate tuning set to choose settings, and the protected set once to support the final claim

if a protected example leaks into tuning, replace it and record the change rather than pretending the original test stayed independent

repeat each important run enough times to show normal variation, and keep failed attempts in the report

do not choose the fastest run from a noisy batch and present it as the normal result

show the distribution, the number of runs, the traffic conditions, and what changed between the baseline and candidate

if a service gets faster by returning shorter or incomplete answers, it has not improved under the same task contract

if requests fail or time out, include them in the scheduled request count and report how many useful answers remain

choose the comparison and decision rule before the final run... preserve failed requests and measure whether the original task still passes

---

make the handoff reproducible

another person should be able to start from a clean directory and reach the same decision without your shell history

include the model and tokenizer revision, software versions, container or environment, hardware identity, launch command, data revision, and expected outputs

pin a dependency where a newer version could change the result, and write down a known reason to update that pin

provide one small smoke test that proves the setup works before any expensive load test begins

document where the service logs errors, where raw events go, how to stop the process, and how to shut down cloud resources

ask the reviewer to follow the instructions without a private explanation from you

each question they ask reveals a missing instruction, hidden assumption, or undocumented dependency

after fixing the runbook, ask them to repeat the test from the beginning and keep the fresh result

---

publish a benchmark people can reproduce

write the benchmark report for an engineer who wants to challenge your result, not for someone who only wants a winning chart

start with the question and the workload the system is meant to serve

name the model and tokenizer revisions, engine version, container, GPU model, number of devices, memory, driver, and network

show the request mix, input and output lengths, concurrency, arrival pattern, warmup method, run duration, repetitions, and failed requests

define every timer boundary, especially whether TTFT starts at scheduling, client send, server receipt, or another event

label token-level measurements, output-chunk gaps, and end-to-end request time separately

publish answer checks and quality thresholds beside latency, throughput, goodput, utilization, and cost

include the number of useful completions, timeouts, rejections, cancellations, and runs that failed to start

show the spread across repetitions and state which conditions changed between the baseline and candidate

link the runnable code, pinned recipe, data description, raw result format, analysis script, and a clean reproduction command

use synthetic or properly licensed inputs, and remove credentials, private prompts, and user data from the release

finish with the limits of the result and one workload where your change loses

if another person cannot reproduce the experiment, publish it as an unverified draft and explain what blocked the rerun

---

the finish line: handle an unfamiliar problem

the six-month promise needs an assessment that changes the problem after you've prepared your answer

ask a reviewer to choose a new compatible deployment constraint and two faults you have not practiced

you must launch the recipe, check quality, measure the service, diagnose the faults, and defend a repair or deployment recommendation

defend two changes you wrote: one in numerical or GPU code, and one in request handling or worker state

the reviewer checks six areas:

• software: can you change, test, and recover the service?
• model behavior: can you explain inputs, generation settings, and answer checks?
• measurement: can you catch an invalid comparison and state uncertainty?
• GPU work: can you defend one numerical or kernel change?
• reliability and distribution: can you preserve request and state correctness through a failure?
• judgment: can you adapt the recipe and reject a change the evidence does not support?

documentation is allowed, while the independent assessment excludes generated finished solutions and another person's completed patch

all required checks must pass, and a failed skill gets a fresh test after repair

this gives you evidence for entry-level work, while a job offer and independent ownership of a large production fleet remain separate outcomes

if you only run the assessment yourself, label it that way

---

your AI helper: keep the learning yours

Codex is an AI coding assistant you can use to explain errors, prepare small examples, review changes, and challenge unsupported conclusions

give it a bounded role for the current project:

1. explain only the concepts needed for this step
2. show a related example, then let me write the assessed code
3. ask for my prediction before running the experiment
4. preserve raw results and keep trusted tests separate from proposed optimizations
5. give debugging hints before a finished fix
6. ask me to explain why the change worked and where it might fail

for example, ask Codex to inspect one slow benchmark and suggest two measurements that could distinguish queue delay from decode time

predict the result first, run those measurements yourself, then ask it to challenge your interpretation using the raw events

when you need code help, give it one failing test and ask for a minimal explanation before requesting a patch

review every line it proposes, run the test suite, and compare the behavior with the current engine documentation

do not let it turn an unsupported engine flag into a recipe or a plausible speed estimate into measured evidence

before a review or exam, close the assistant and reproduce the important result from your own instructions

if you cannot explain the change or repair a small variation without generated code, return to that part of the project

record which code you wrote, adapted, or generated, and check suggested settings against the actual model and software documentation

the explanation belongs to you... if you cannot give it without the assistant, return to the experiment

use Codex to challenge the next step, then make the prediction, inspect the result, and repeat the important work without a generated solution

---

the build sheet: keep the whole plan together

---

the model inference recipe

this is the main artifact you keep updating, not a blank template to complete at the end

it records how to run one model for one task on specific hardware, how to tell whether its answer is correct, and what the run cost

schema_version: 1
identity:
  model_id: "model name from its card"
  model_revision: "immutable model revision"
  tokenizer_revision: "immutable tokenizer revision"
  adapter_revision: null
  artifact_checksum: "record when available"
  license_reference: "model license URL"
runtime:
  engine: "vLLM or SGLang"
  version_or_commit: "pinned version"
  container_digest: "pinned image digest"
  driver: "observed driver version"
  command_file: "launch.sh"
hardware:
  device_model: "observed device"
  device_count: 1
  memory_bytes_per_device: null
  topology_file: "topology.txt"
behavior:
  template_hash: "rendered chat-template implementation"
  generation_config: "generation.json"
  parser_version: "pinned parser"
  reasoning_mode: "documented mode or unsupported"
  stop_config: "stops.json"
  modalities: [text]
execution:
  weight_precision: "observed path"
  activation_precision: "observed path"
  accumulation_precision: "observed path"
  kv_precision: "observed path"
  context_limit: null
  concurrency_limit: null
  scheduling_config: "scheduling.json"
  speculative_config: null
workload:
  dataset_revision: "immutable test-data identity"
  arrival_schedule: "arrivals.jsonl"
  application_contract: "contract.json"
  protected_eval_manifest: "evaluation-manifest.json"
evidence:
  code_commit: "commit ID"
  run_manifest: "run.json"
  raw_events: "events.jsonl"
  quality_report: "quality.json"
  hardware_trace: "trace path"
  resource_ledger: "cost.csv"
  decision_record: "decision.md"
operations:
  readiness_command: "tested inference probe"
  rollback_recipe: "previous recipe ID"
  shutdown_command: "tested stop procedure"

the schema is a checklist, not a working launch configuration

fill every value from the actual model and engine documentation

each request event records the run, tenant, request, attempt, model recipe, observation point, monotonic timestamp, event type, and outcome

preserve network-chunk counts separately from generated-token counts, and keep errors and missing-data flags in the record

each change note records what stayed the same, what changed, the baseline, prediction, measured result, quality and cost checks, unsupported cases, and the test that could reverse the decision

the three valid verdicts are supported, hold, and insufficient evidence

---

four calculations to check before trusting the dashboard

request timing: a request is scheduled at 0.000 seconds, sent at 0.020 seconds, and returns its first content at 0.340 seconds

client-send TTFT is 320 ms, schedule-to-content is 340 ms, and a response completing at 0.840 seconds took the client 820 ms from send to completion

three chunks arrive at 0.340, 0.480, and 0.800 seconds and contain 2, 1, and 3 tokens

the observed chunk gaps are 140 ms and 320 ms

they cannot establish the timing of each individual token

request accounting: ten scheduled requests yield six completions, two rejections, one timeout, and one cancellation

if four completed requests meet the quality and delay requirements in ten seconds, useful goodput is 0.4 requests per second, with the other six outcomes shown beside it

KV memory: for a standard full-attention decoder, estimate logical cache bytes as 2 × layers × cached tokens × KV heads × head dimension × bytes per element

for an illustrative model with 24 layers, 4,096 tokens, 8 KV heads, 128 values per head, and two-byte values, one sequence uses 384 MiB of logical KV

four independent sequences use 1.5 GiB, before sharing, page rounding, weights, workspaces, and reserved memory

grouped-query attention (GQA) shares key-value heads across query heads, so use the model's KV-head count

multi-head latent attention (MLA) stores a compressed representation, while sliding-window attention and other architectures need their own accounting

measure each GPU using the engine's real sharding and memory layout instead of dividing total memory by the number of devices

speculative sampling: let the target probabilities be p = [0.10, 0.60, 0.30] and the draft probabilities be q = [0.50, 0.25, 0.25]

the token acceptance probabilities are min(1, p/q) = [0.20, 1, 1]

the accepted draft mass is [0.10, 0.25, 0.25], totaling 0.60

the remaining target mass, normalized for rejection, is [0, 0.875, 0.125]

combining acceptance and correction recovers the target distribution

enumerate this small case before implementing multi-token state rollback

cost: divide the resource costs in your declared measurement window by its useful completed answers

include failed work and warm capacity inside that window, and report that no finite cost per useful answer exists when zero answers meet the contract

model-FLOPs utilization: estimate the model operations done in an interval, then divide by the participating GPUs' precision-matched peak operations for that interval

state the operation estimate, precision, device count, and dense or sparse peak assumption

report GPU activity, bandwidth, queues, and goodput beside MFU

this arithmetic example uses one declared window... all resource cost stays in the numerator, even when some work produces no useful answer

the next two tables are indexes into the weekly route, not extra projects or prerequisites

---

the 12-stage crosswalk

1. systems foundations: Python, Git, Linux, processes, tests, arrays, and CPU/GPU differences · weeks 1 to 8
2. GPU architecture: warps, streaming multiprocessors, memory bandwidth, occupancy, tensor cores, and profiling · weeks 7 to 8
3. transformer inference: attention, prefill, decode, numerical behavior, and KV growth · weeks 6 and 9 to 12
4. serving engines: vLLM or SGLang, endpoints, batching, client load, and readiness · weeks 4 to 5 and 13 to 16
5. KV optimization: page management, admission, prefix reuse, and live memory evidence · weeks 9 to 10 and 14
6. quantization: supported FP16, BF16, FP8, INT8, AWQ, and GPTQ choices compared with held-out answers · week 11
7. speculation and parallelism: draft/target verification, rollback, tensor parallelism, pipeline stages, and experts · weeks 12 and 19 to 20
8. kernel optimization: C++, CUDA, Triton, profiling, fusion, graphs, and fallbacks · weeks 7 to 8
9. distributed inference: real device and host communication, placement, KV transfer, and disaggregation · weeks 19 to 22
10. autoscaling and economics: cold starts, queue-based scaling, idle capacity, cost allocation, and MFU · weeks 15 to 16
11. gateway and reliability: request contracts, fallback, rate and usage limits, SLOs, failures, and rollback · weeks 17 to 18
12. portfolio and benchmarks: second-model recipe, repeatable three-way comparison, public-quality evidence, and defense · weeks 23 to 26

---

the 15-build crosswalk

---

choose one serving engine and learn it well

in week 4, compare vLLM, SGLang, and TGI on model support, hardware, chat-template behavior, streaming, and measurement hooks

as of October 6, 2026, the TGI repository is archived and read-only, so use it as a dated reference rather than the main build

write the date and model revision beside each support claim, then choose vLLM or SGLang for the main build

the weekly plan links technical material beside the task that needs it, so the tutorial has an immediate test to solve

---

after week 26: advanced releases

• kernels and attention: add an owned bounded attention path, then compare it with FlashAttention, an algorithm designed to reduce attention memory traffic, and FlashInfer, a library for attention and cache operations, including layout and dispatch cost
• memory: test cache quantization or host offload separately, and account for reconstruction, active GPU storage, cancellation, and re-prefill
• generation: study Medusa or EAGLE-3, methods that propose multiple candidate tokens for verification, only after the draft/target path passes
• other hardware: port one reference operation to ROCm AITER, an AMD GPU kernel library, Pallas, a JAX tool for writing accelerator kernels, or Neuron NKI, AWS's kernel programming interface, and separate compilation evidence from GPU performance evidence
• MoE and transfer: expand synthetic experts and study DeepEP or Mooncake on compatible hardware
• generated kernels: evaluate KernelBench, a benchmark suite for GPU kernels, with protected tests and a trusted reference outside the generator's writable files

each advanced release has its own supported hardware, correctness check, baseline, and full-request measurement

choose one deeper project from a bottleneck you can measure... each branch returns to correctness checks and the full-request comparison

---

the visual guide

the fourteen diagrams sit beside the concepts they explain, from the first cloud session to the next advanced release

• the first service: the roadmap, local-to-cloud lab, and request path
• shared capacity: KV blocks, scheduling, load curves, and worker readiness
• trustworthy decisions: state handoff, retries, evidence levels, and controlled experiments
• continued progress: learning with Codex, cost accounting, and deeper project choices

here's the complete sequence to keep beside your repository:

1. month 1: one working service and an executable model recipe
2. month 2: trustworthy load curves, a change reviewer, and a checked GPU kernel
3. month 3: a memory manager and evidence for each generation setting
4. month 4: shared scheduling, ready capacity, and a cost report that adds up
5. month 5: a gateway that survives faults and a measured device layout
6. month 6: a verified state handoff, two recipes, and an independently assessed release

every week ends with a working increment, a failure you understand, and a decision you can explain

skip the answer checks and a speed gain can hide a broken task, while missing raw results leaves another person unable to inspect your claim

that's your first test set... and the thing every later improvement has to protect

open a terminal tonight, create the first recipe folder, and write ten synthetic support tickets as a starter set with their expected answers

---

I hope you enjoyed reading this

Please, this has taken me 202 days of mistakes and 40+ hours to compile all of this. ** I appreciate you reading till the end. T**

his has been written by me, my entire Codex and Claude history, and my AI agent Barry, so thank him as well.

It's been edited using DeepSeek V4 Flash.