Serving LLMs Efficiently with vLLM: From Core Concepts to Real-World Optimization
This tutorial is part of the RIVF 2026 programme. It runs as a half-day session of four hours, including a break, with hands-on labs. See Tutorials for the full list of accepted tutorials.
| Tutorial Date | To be announced |
|---|---|
| Conference Dates | December 18–20, 2026 |
Abstract
vLLM has become the default open-source engine for serving LLMs, but most users treat it as a black box. This tutorial opens the box. The first half explains how vLLM actually works — the inference pipeline, KV cache, scheduling, and why these design choices matter. The second half shows how a team optimizes a serving engine in practice: how to find bottlenecks, tune kernels, manage memory and precision, and measure results honestly. Examples are drawn from Moreh-vLLM, Moreh's in-house vLLM development effort, but every technique is general and applies to any vLLM deployment.
Audience and Prerequisites
The tutorial is intended for people who deploy or study LLM serving systems: the closing discussion is built around attendees' own deployment problems and the research opportunities around them. The labs run in a browser, so attendees need only a laptop with a browser; no local GPU is required.
Learning Objectives
- Explain how vLLM works: the inference pipeline, the KV cache and PagedAttention, continuous batching, and the engine, scheduler, model runner and kernels.
- Configure vLLM well with prefix caching, chunked prefill, speculative decoding, quantized models and tensor or pipeline parallelism, and read the metrics it reports.
- Locate a bottleneck across compute, memory bandwidth, scheduling and CPU overhead using profiling tools and benchmark metrics such as TTFT, TPOT and goodput.
- Change one serving setting, such as the attention backend, the block size or a quantized model, and measure its effect on throughput and output quality.
Schedule
Lectures run for 130 minutes (Parts I–II 65 minutes, Parts III–IV 65 minutes) and labs for 60 minutes. The main content is hardware-neutral; AMD is the recurring example.
| Time | Session | What we cover | Format |
|---|---|---|---|
| 8:30–8:40 | Opening | Why serving cost matters; agenda; intro to Moreh and Moreh-vLLM; lab environment login | Framing |
| 8:40–9:15 | Part I — LLM inference and how vLLM works | Prefill vs. decode; why decode is memory-bound; KV cache and PagedAttention; continuous batching; vLLM architecture (engine, scheduler, model runner, kernels) | Lecture |
| 9:15–9:45 | Part II — Using vLLM well | Prefix caching, chunked prefill, speculative decoding, quantized models, tensor/pipeline parallelism; reading vLLM's metrics; common configuration mistakes | Lecture |
| 9:45–10:05 | Lab 1 — KV cache and batching in action | Serve a small model on MI250; vary concurrency; watch KV cache usage, batch size, TTFT/TPOT; toggle prefix caching and chunked prefill | Hands-on |
| 10:05–10:20 | Coffee break | ||
| 10:20–10:50 | Part III — Finding the bottleneck | Compute vs. memory bandwidth vs. scheduling vs. CPU overhead; tools (torch profiler, rocprof, vLLM tracing); benchmark metrics (TTFT, TPOT, goodput); misleading-number pitfalls. Case: profiling Moreh-vLLM releases | Lecture and case study |
| 10:50–11:10 | Lab 2 — Profile two workloads | Long-prompt/short-output vs. short-prompt/long-output; profile both; identify which is compute-bound and which is memory-bound | Hands-on |
| 11:10–11:45 | Part IV — Optimizing inside vLLM | (1) Kernels: attention, GEMM, MoE; Triton vs. vendor libraries vs. hand-tuned; NVIDIA/AMD portability. (2) Scheduler and KV cache: batching policy, block size, fragmentation. (3) Precision: INT8/INT4/FP8 weights and KV cache; what to quantize. Case: porting and tuning vLLM for AMD Instinct | Lecture and case studies |
| 11:45–12:05 | Lab 3 — Change one thing, measure it | Swap attention backend, change block size, some other performance factors or load an INT8/INT4 (GPTQ/AWQ) version of the model; compare throughput and output quality | Hands-on |
| 12:05–12:25 | Discussion and Q&A | Attendees' deployment problems; research opportunities | Discussion |
| 12:25–12:30 | Closing | Takeaways; reading list; getting involved with vLLM and Moreh-vLLM | Closing |
Hands-on Component
Labs run on AMD Instinct MI250 GPUs provided by Moreh, accessed through a browser (JupyterHub); attendees need only a laptop with a browser. Each GCD is shared by four to six attendees. Notebooks are guided: the environment is pre-started and attendees change one setting and observe. A recorded run of each lab serves as fallback. A teaching assistant from Moreh Vietnam supports the labs.
Instructors
Dr. Thanh Tuan Dao — Head of Moreh Vietnam
Thanh Tuan Dao is Head of Moreh Vietnam, where he leads the engineering team behind Moreh-vLLM, Moreh's internal vLLM engine for heterogeneous AI accelerators, as well as a range of LLM inference optimization projects built on vLLM and SGLang. Since joining Moreh in 2021, he has contributed across the company's AI infrastructure stack, from the MoAI platform and Moreh's PyTorch-compatible deep learning framework for large-scale AMD GPU clusters to quantization and LLM serving systems, and now leads performance engineering for Moreh's LLM inference deployments with enterprise and telecom customers. His research interests include GPU performance modeling and auto-tuning, compiler optimization for deep learning, and efficient LLM serving; his work has appeared at PLDI, SC, ASPLOS, and MASCOTS, and in IEEE Transactions on Parallel and Distributed Systems and IEEE Access. He received his Ph.D. and M.S. in Computer Science and Engineering from Seoul National University and his B.S. in Computer Science from VNU University of Engineering and Technology.
Loc Tran — Engineering Manager, Moreh Vietnam
Loc Tran is an ML systems engineer and Engineering Manager at Moreh Vietnam, where he leads the AI System team. He is the lead developer of Moreh-vLLM, Moreh's internal vLLM engine for heterogeneous AI accelerators, spanning AMD and NVIDIA GPUs as well as NPUs. His technical interests span deep learning frameworks and the programming languages, compilers and runtimes that support them. He works most closely on quantization, KV cache management, parallelism strategies for LLM serving, and performance portability across accelerator vendors. He graduated from the Talented Program in Computer Science at Hanoi University of Science and Technology.
Anh Cao — GPU Team Lead, Moreh Vietnam
Anh Cao is a GPU engineer at Moreh Vietnam, where he leads the GPU team and optimizes LLM inference on Moreh-vLLM, Moreh's internal vLLM engine for heterogeneous AI accelerators. His technical interests lie in GPU architecture and high-performance kernel programming. He works most closely on performance profiling and analysis for LLM serving on AMD and NVIDIA GPUs, writing custom GPU kernels for performance-critical operators such as attention, GEMM and MoE, and tuning vendor-library kernels. He graduated in Computer Science from Hanoi University of Science and Technology.
Contact
For inquiries, please contact: rivf2026@vinuni.edu.vn