Tracing a Request Through Mini-SGLang

Code Reading Notes #01 — From an HTTP Request to GPU Execution

As I've been learning about LLM inference, I've come across concepts such as prefill, decode, continuous batching, KV caching, and scheduling.

Understanding each concept in isolation is one thing. Understanding how they work together inside a real inference engine is another.

That led me to a simple question:

What actually happens after a user sends a prompt to an LLM serving engine, and how does the first generated token eventually reach the user?

Rather than starting with the full SGLang codebase, I began with the more approachable Mini-SGLang. Mini-SGLang is a relatively compact LLM serving engine that preserves important runtime mechanisms, including scheduling, KV cache management, attention backends, and model execution. Its smaller codebase makes it a useful place to build a mental model of an inference system.

In this post, I'll follow a text-only request through Mini-SGLang, from the HTTP API to the tokenizer, scheduler, GPU engine, and finally the streaming response.

I'll focus on a single request running with tensor parallelism set to one (TP=1). The discussion is based on source code inspection rather than an instrumented runtime trace or performance benchmark.

Architecture: Following a Request Across Processes

Let's start with the overall data flow:

User
  │ HTTP POST /generate
  ▼
API Server / FrontendManager
  │ TokenizeMsg
  ▼
Tokenizer Worker
  │ UserMsg(input_ids)
  ▼
Scheduler
  │ Batch Preparation / KV Allocation
  ▼
Engine / GPU
  │ Model Forward → Logits → Sampling
  ▼
Scheduler
  │ DetokenizeMsg(next_token)
  ▼
Detokenizer Worker
  │ UserReply(incremental_output)
  ▼
FrontendManager
  │ HTTP Streaming
  ▼
User

The first thing to notice is that LLM serving is not simply an HTTP handler calling model.forward() and returning the result. Mini-SGLang separates request handling, tokenization, scheduling, model execution, and detokenization. These components exchange messages through ZeroMQ (ZMQ).

Figure 1

End-to-End Request Lifecycle

API server processtokenizer workerscheduler process · per TP rankdetokenizer worker--num-tokenizer=0 (default):tokenizer + detokenizer share one processHTTPTokenizeMsgUserMsgDetokenizeMsgUserReplySSEUserclientAPI ServerPOST /generateTokenizerTokenizeManagerScheduleroverlap_loop()Engine / GPUforward_batch()DetokenizerDetokenizeManagerFrontendFrontendManager
The client sends POST /generate with {"prompt": "Hello", "max_tokens": 8}.

Conceptual animation based on mini-sglang @ 9a91cfa source code, not measured data.

Looking at launch.py, Mini-SGLang creates one scheduler worker per tensor-parallel rank (TP size). Each scheduler worker is a separate OS process that contains its own Scheduler and Engine.

A few terms are worth separating here:

  • Process: an independent program instance managed by the operating system, with its own memory space.
  • Worker: a unit responsible for a specific job; here it usually corresponds to one process.
  • Scheduler: handles request scheduling, batch construction, and decides when to run prefill or decode.
  • Engine: handles GPU work such as the model forward pass, the KV cache, and the attention backend.

For example, with --tp-size=2, Mini-SGLang creates two scheduler processes, each with its own Engine on a different GPU. This does not mean each GPU handles a separate request. Through tensor parallelism, the model weights and computation are split across the two GPUs, which work together on the same model forward pass.

So although the Scheduler and the Engine do different jobs, they live in the same process. They are not two separate processes talking over IPC.

Meanwhile, with the default --num-tokenizer=0, Mini-SGLang does not create a separate tokenizer process. Tokenization (text to token IDs) and detokenization (token IDs to text) share one worker process. Only when --num-tokenizer is greater than 0 does it create that many additional tokenizer workers, spreading CPU-side tokenization across multiple processes.

In other words, the Scheduler and the Engine are different responsibilities, but not necessarily different processes; the tokenizer and the detokenizer are different stages, but not necessarily different processes either.

This distinction matters for understanding serving architecture. When analyzing a performance bottleneck, we first need to know which layer the work happens in: tokenization may be limited by CPU throughput, the scheduler may be affected by batch scheduling or KV cache management overhead, and the Engine's decode may be limited by GPU HBM bandwidth.

Understanding process boundaries is what lets us judge which work can run in parallel, which resources are shared, and whether the real optimization target is CPU scheduling or GPU execution.

Source: Architecture Documentation, launch.py.

The Frontend: Turning an HTTP Request into a Message

Consider a request to Mini-SGLang's /generate endpoint:

POST /generate
Content-Type: application/json

{
  "prompt": "Hello",
  "max_tokens": 8
}

The FastAPI endpoint in server/api_server.py receives the request and delegates its lifecycle management to FrontendManager.

One of the first things FrontendManager does is assign a unique request identifier, or uid.

It also creates the internal state needed to associate future responses with this request.

The frontend then constructs a message similar to:

TokenizeMsg(
    uid=uid,
    text=req.prompt,
    sampling_params=SamplingParams(
        max_tokens=req.max_tokens,
        ignore_eos=req.ignore_eos,
    ),
)

This message is sent to the tokenizer worker through ZMQ.

The first mental model here is that the frontend manages the request lifecycle and the response channel. It also sets up a waiting state for the uid, so the HTTP stream can later receive the results the backend produces step by step.

Source: api_server.py.

Tokenization: From Text to Input IDs

The tokenizer worker receives TokenizeMsg and converts the input text into token IDs.

Conceptually:

"Hello"
    ↓
Tokenizer
    ↓
input_ids = [...]

The exact token IDs depend on the model's tokenizer, so I'm intentionally leaving them unspecified.

In Mini-SGLang, the relevant path is:

tokenize_worker()
    ↓
TokenizeManager.tokenize()
    ↓
tokenizer.encode()
    ↓
UserMsg(uid, input_ids, sampling_params)

The worker uses a Hugging Face tokenizer. When the input consists of chat messages, it also applies the model's chat template before encoding the prompt.

The resulting input_ids are represented as an integer tensor and sent to the scheduler in a UserMsg.

At this point, we have transformed human-readable text into model input IDs, but we haven't executed the transformer yet.

This helped me separate two responsibilities that are easy to mentally combine:

Tokenization prepares the input. Scheduling and model execution determine how that input is processed.

Source: tokenizer/server.py, tokenizer/tokenize.py.

Inside the Scheduler: Deciding What the GPU Runs Next

Once a request reaches the scheduler, the scheduler must decide which requests are runnable, how to group their work, and whether the next batch should perform prefill or decode.

A simplified view of Mini-SGLang's normal_loop() looks like this:

Receive Messages
      ↓
Process Requests
      ↓
Schedule Next Batch
      ↓
Prepare Batch
      ↓
GPU Forward
      ↓
Process Results

When a new UserMsg arrives, the scheduler passes it to PrefillManager.

The key decision happens in _schedule_next_batch():

batch = (
    self.prefill_manager.schedule_next_batch(self.prefill_budget)
    or self.decode_manager.schedule_next_batch()
)

In this version, the scheduler first tries to construct a prefill batch. If no prefill batch can be scheduled, it tries to schedule decode work.

These phases have different purposes.

Prefill processes the prompt tokens that do not yet have computed KV states. It builds the required KV cache and produces the prediction for the first output token.

Decode processes newly generated tokens while reusing the KV cache from previous positions.

Even with only one request, the scheduler needs to track its state and determine how much work must be performed in each iteration.

With multiple concurrent requests, the same mechanisms become the foundation for batching, admission control, and GPU resource management.

Source: scheduler.py, prefill.py.

Request State: Connecting Prefill and Decode

In LLM inference, prefill and decode are two consecutive phases of the same request. Prefill processes the prompt tokens, builds the KV cache, and predicts the first output token. Decode then processes one newly generated token per step, using the KV cache built so far to keep generating.

From a GPU hardware perspective, prefill processes many tokens at once, so GEMMs can reuse each weight across many tokens, which tends to make it compute-bound. Decode with a small batch does little computation per step but still has to read the model weights and the historical KV cache every step, so it tends to be limited by memory bandwidth.

Since prefill and decode process different tokens at each step, how does the runtime know which tokens already have KV cache and which still need to be computed? That is exactly what cached_len, device_len, and extend_len are for.

In core.py, a Req contains several important fields:

cached_len
device_len
extend_len = device_len - cached_len

I think about them this way:

  • [0, cached_len) represents token positions whose KV states are already available.
  • [cached_len, device_len) represents token positions that still need to be processed in the current forward pass.
  • extend_len tells us how many tokens need to be computed.

Let's walk through an example.

Suppose our prompt consists of three tokens, A B C, with no prefix cache hit.

First forward pass: Prefill

Before execution:

Tokens:       A   B   C
cached_len:   0
device_len:   3
extend_len:   3

The model processes A, B, and C, computes their KV states, and samples the next token from the resulting logits.

Let's call that token D.

The request now has the state:

KV already computed    Next token to process
 A   B   C         |          D

cached_len = 3
device_len = 4
extend_len = 1

This reveals a subtle but important distinction:

Sampling token D does not mean that D's own KV state has already been computed.

The prefill pass processed A, B, and C. Token D is the output selected from that computation.

D's KV state will be calculated in the next forward pass.

Second forward pass: Decode

The next iteration processes token D, using the cached K/V from A, B, and C as context.

The model then samples another token, E.

Conceptually, the state becomes:

KV already computed        Next token
 A   B   C   D         |       E

cached_len = 4
device_len = 5
extend_len = 1

Mini-SGLang advances this state using Req.complete_one():

self.cached_len = self.device_len
self.device_len += 1

This small function helped connect several ideas for me.

Prefill and decode are not unrelated operations. They are successive stages of the same autoregressive generation process, with request state and KV cache determining which token positions need computation.

Source: core.py, engine.py.

KV Cache: Who Allocates the Slot, and Who Actually Writes K/V?

Once request state made sense, the next question was: where do the K/V tensors computed by the transformer actually live, and how does the scheduler know where to write them?

I originally assumed the scheduler asked the GPU for new KV cache memory on every forward pass. That is not how it works. The Engine creates a KV cache pool on the GPU at startup; after that, the scheduler only hands out free positions from this pool instead of creating new tensors for each forward pass.

From Logical Position to Physical KV Slot

Suppose a request's input is [A, B, C], at logical positions 0, 1, 2. The GPU KV cache is shared by many requests, so these tokens are not necessarily stored in contiguous physical slots.

Mini-SGLang uses a page table to map one to the other:

Request (table_idx = 1)
         │
         ▼
Logical Position      0    1    2
                      │    │    │
Page Table Row 1      2    5    6
                      │    │    │
Physical KV Slots     2    5    6

For example, page_table[1, 2] = 6 means the token at logical position 2 of this request has its K/V in slot 6 of the KV pool.

Three components handle different parts of this:

  • TableManager manages the table indices available to requests. When a new request is admitted, allocate() gives it a row, table_idx.
  • CacheManager manages the free physical KV slots. During _prepare_batch(), allocate_paged() assigns slots to the tokens computed this round and updates the page table.
  • MHAKVCache is the GPU memory pool that actually stores the K/V tensors, with a separate K/V region for each transformer layer.

The page table stores physical token slot indices, not the K/V data itself and not raw GPU addresses. With the default page_size=1, each page is one token slot, and the same slot index has a K/V storage location in every layer.

Allocating KV Space Is Not Writing KV

This was the most important thing I found in the code. In _prepare_batch(), the scheduler first prepares space for the tokens to be computed this round:

self.cache_manager.allocate_paged(batch.reqs)
batch.positions = _make_positions(batch, self.device)
input_mapping = _make_input_tuple(batch, self.device)
batch.out_loc = self.engine.page_table[input_mapping]

batch.out_loc records which physical KV slots the newly computed tokens of this round will be written to, not every KV location of the request. Continuing the example:

cached_len = 0, device_len = 3, extend_len = 3
batch.out_loc = [2, 5, 6]

A, B, and C all need their K/V computed in this prefill and written to slots 2, 5, and 6. At this point only allocation has happened: the destination is decided, but no new K/V has been written yet.

Only when the Engine runs the transformer forward pass and computes each layer's Q, K, and V does the FlashInfer backend call:

self.kvcache.store_kv(k, v, batch.out_loc, layer_id)

MHAKVCache uses out_loc and layer_id to write the new K/V into that layer's KV buffer. FlashInfer then reads the historical and newly written K/V according to the attention metadata and computes the attention output:

Allocate KV Slots → Page Table → batch.out_loc
  → Compute Q, K, V → store_kv() (write)
  → FlashInfer (read) → Attention Output

If the request has a reusable prefix cache, the cached part reuses existing KV, and batch.out_loc contains only the tokens computed this round.

Watching the KV Lifecycle Through Request State

Figure 2

KV Cache + Request State

Req · table_idx = 1 · state 1cached_len0device_len3extend_len3ALLOCATEget positionsWRITEstore K/VREADrun attentionSAMPLEnext tokenLogical positionsreq.input_idsPage tablepage_table[1, pos]Physical KVkv_buffer slotApos 0Bpos 1Cpos 2pos 3pos 4pos 5––––––·0·12·3·456·7·891011page_size = 1 · one slot index maps to a K/V location in every layer (not a single K/V copy)
State 1 · prefill. Nothing is cached yet, so all three prompt tokens must be computed: extend_len = device_len − cached_len = 3.
KV already cachedComputed this forwardNewly sampled tokenAllocated, not writtenOther requests' slots

Conceptual animation based on mini-sglang @ 9a91cfa source code, not measured data.

This animation ties the ideas together. With [A, B, C], the initial state is cached_len=0, device_len=3, extend_len=3:

  • ALLOCATE: get positions. The page table maps logical positions to physical slots, but no KV is written yet.
  • WRITE: store K/V. The transformer computes new K/V and writes it into the assigned slots.
  • READ: run attention, reading the needed KV through the page table and metadata.
  • SAMPLE: produce the next token. The model samples D from the logits at the last position.

The easy-to-miss detail: D being generated does not mean D's K/V exists. D has a token ID and the next logical position, but it has not gone through its own transformer forward pass. Its KV slot and K/V are allocated and written in the next decode step.

This made the purpose of cached_len, device_len, and extend_len clearer to me: they do not just record the request's length; they decide which tokens the next forward pass must compute.

How Does KV Cache Affect Request Admission?

With the read/write flow in place, there is a question closer to serving optimization: if there is not enough KV cache space, can the scheduler still admit a new request?

In PrefillAdder._try_allocate_one(), Mini-SGLang first estimates the KV capacity a new request needs:

extend_len = req.input_len - cached_len
estimated_len = extend_len + req.output_len

if estimated_len + self.reserved_size > self.cache_manager.available_size:
    return None

estimated_len covers the input tokens not yet computed plus the expected output tokens; reserved_size holds back the estimated needs of other in-flight requests or requests already admitted in the same round. This is the admission capacity check, a separate stage from the actual slot allocation in _prepare_batch().

Figure 3

KV Capacity & Admission

GPU memory (schematic)

Model Weights
R1R1R1R1R1R1R1R1R2R2R2R2R2R214151617181920212223

KV pool · page_size = 1

Admission Decision · estimate future demand

estimated_len
4 + 8 = 12
reserved_size
2
Required
14
Available
10

Waiting for admission check

Allocation · slots for this round

_prepare_batch() → allocate_paged()

out_loc = —

Not in a prefill batch yet; no slots allocated

reserved_size: estimated needs of other in-flight or same-round requests · all numbers illustrative

R1 and R2 are already running and hold KV slots. New request R3 arrives with a 4-token prompt and max_tokens = 8.
Other requests (R1, R2)Allocated, not writtenK/V written

Conceptual animation based on mini-sglang @ 9a91cfa source code, not measured data.

Take R3 in the figure: 4 input tokens, up to 8 output tokens, and no prefix cache, so it is estimated to need 12 KV slots. With 2 more slots reserved, the admission check asks whether available capacity can hold 14 slots.

Once R3 is admitted, though, this prefill does not write KV for all 12 tokens. Only the 4 input tokens run forward this round, so out_loc contains just those 4 positions, for example [14, 15, 16, 17]. KV for generated tokens is allocated and written step by step in later decode passes.

From a GPU hardware perspective, this is why KV cache limits inference capacity. Model weights, KV cache, and other runtime buffers all need GPU memory; a model that loads successfully may still lack the KV cache capacity for many concurrent requests or long contexts. KV cache size depends on context length, concurrent requests, number of transformer layers, KV heads, head dimension, and data type.

So serving capacity is not just whether the model fits on the GPU; it depends on how much KV cache, in use or reserved, the remaining HBM can hold.

For me, this part of the code split KV cache into three separate questions:

  • Where to store? The scheduler decides which slots K/V go to through the page table.
  • When to write? The attention backend writes only after the transformer computes K/V.
  • How much can we serve? The scheduler decides whether to admit more requests based on available KV capacity and estimated demand.

With these three questions answered, KV cache is no longer just a buffer that avoids recomputation. It is a core resource that directly shapes scheduling, concurrency, and GPU memory efficiency.

Source: cache.py, table.py, mha_pool.py, fi.py, prefill.py.

GPU Forward: From Input IDs to the Next Token

We can now connect scheduling to actual model execution.

Mini-SGLang's scheduler contains a relatively small _forward() method:

batch.input_ids = self.token_pool[input_mapping]
forward_output = self.engine.forward_batch(batch, sample_args)
self.token_pool[output_mapping] = forward_output.next_tokens_gpu

These lines describe a crucial boundary.

First, the scheduler retrieves the token IDs required for the current batch.

Next, Engine.forward_batch() runs the model. Depending on the batch and the available execution path, the engine uses a regular model forward pass or CUDA Graph replay.

Inside the transformer, the attention layer processes Q, K, and V, applies the relevant positional transformations, and invokes the configured attention backend.

The attention implementation reads the required cached states and writes new K/V values.

The model then produces logits, which are used by the sampler to select the next token ID.

The distinction between these representations matters:

Input Token IDs
      ↓
Model Forward
      ↓
Attention + KV Cache
      ↓
Logits
      ↓
Sampler
      ↓
Next Token ID

Logits are not tokens, and token IDs are not text.

Logits represent scores over the vocabulary. Sampling converts those scores into a selected token ID. Detokenization will later convert the generated token sequence into readable text.

The newly selected token is also written into the token pool so the next decode iteration can continue.

Figure 4

Scheduler + GPU Forward

CPU · SchedulerGPU · Engine streamdata1stelsecomplete_one() · filter_reqs()UserMsg→ pending_listSchedule_schedule_next_batch()prefill ‖ decodePrefillPrefillManagerpending_listDecodeDecodeManagerrunning_reqs_prepare_batch()allocate_paged()positions · out_locScheduler._forward()token_pool → input_idsEngine.forward_batch()CUDA graph | model.forward()Model ForwardAttentionLayer → attn_backendLogitsfloat · [batch, vocab]Samplerargmax | top-k / top-pNext Tokenint32 id → token_poolphase = prefillextend_len = 3ABCD
_process_one_msg() checks the prompt length, clamps max_tokens, then add_one_req() appends the request to pending_list.
Prefill pathDecode path

Conceptual animation based on mini-sglang @ 9a91cfa source code, not measured data.

To me, _forward() is especially useful because it marks the boundary between the scheduler's view of the world and GPU model execution.

The scheduler decides what needs to run. The engine performs the computation.

Source: scheduler.py — _forward, engine.py — forward_batch, AttentionLayer.

Streaming the Output Back to the User

After GPU execution, the scheduler obtains the generated token ID and checks whether the request should finish.

Termination may depend on conditions such as reaching the maximum number of generated tokens or producing an EOS token.

The scheduler sends a message containing the request ID, generated token, and completion status:

DetokenizeMsg(
    uid,
    next_token,
    finished
)

The detokenizer worker converts token IDs into incremental text.

An important detail is that one token does not necessarily correspond to one complete character or word.

For that reason, the detokenizer maintains decoding state for each request rather than independently converting every token into a final printable string.

It may hold back incomplete or unstable fragments until enough information is available to produce readable output.

The worker then sends a UserReply back to the frontend, including the uid, incremental text, and completion status.

FrontendManager uses the request ID to wake up the corresponding waiting HTTP response and stream the new text to the client.

For the /generate endpoint, this is implemented through a StreamingResponse using the text/event-stream media type. The stream eventually emits a [DONE] marker.

This helped me recognize that generating the next token and displaying the next piece of text are different events.

The entire streaming experience depends on multiple components cooperating across process boundaries.

Source: detokenize.py, scheduler/io.py, api_server.py — Streaming.

A Brief Note on Overlap Scheduling

So far, I've used the simpler normal_loop() to explain the control flow.

Mini-SGLang also implements overlap_loop(), which is used when overlap scheduling is enabled—the default configuration unless explicitly disabled.

The basic idea is to overlap GPU work for the current batch with CPU-side processing of a previous batch's results.

Instead of treating scheduling, GPU execution, and output processing as entirely sequential steps, the runtime attempts to reduce idle time by allowing appropriate work to overlap.

This is an interesting reminder that inference optimization does not stop at the GPU kernel.

CPU scheduling overhead, memory transfers, synchronization, and the coordination of different execution stages can all affect serving performance.

I won't go deeper into CUDA streams or synchronization here. Those mechanisms deserve their own investigation, ideally supported by profiling measurements.

Source: scheduler.py — normal_loop and overlap_loop.

What I Learned

Reading through Mini-SGLang helped me connect several concepts that previously felt disconnected.

The frontend owns request handling and response delivery. The tokenizer prepares the model input. The scheduler manages request state, batching, and resources. The cache manager prepares KV storage. The engine and attention backend perform GPU computation. The sampler selects the next token, and the detokenizer turns generated token IDs into readable text.

Three ideas stood out to me.

First, prefill and decode are stages of one continuous autoregressive process. Sampling a new token and computing that token's KV state are not the same operation.

Second, KV caching is more than storing a tensor. It also involves request-to-memory mapping, allocation, reuse, and resource lifecycles.

Third, LLM serving is a systems problem. GPU kernels are important, but scheduling, memory management, process communication, and CPU/GPU coordination are part of the same performance picture.

I haven't instrumented the runtime or measured the performance of these stages yet. Everything discussed here is based on reading and reasoning about the implementation.

My next step is to follow an actual request ID in a GPU environment, record the transitions between tokenization, scheduling, prefill, decode, and detokenization, and eventually investigate metrics such as TTFT and TPOT.

For now, I want to document what I've learned instead of waiting until I've mastered every part of the inference stack.

Read → Trace → Understand → Write → Measure → Improve.


References

This article is based on source code reading, not an instrumented runtime trace or performance benchmark. Implementation details refer to the linked Git commit.