Adapting Vision-Language Models for Low-Resource Document Intelligence: A Vietnamese Case Study
This tutorial is part of the RIVF 2026 programme. It runs as a half-day session of three and a half to four hours, combining lectures with guided practical exercises, delivered by CMC OpenAI. See Tutorials for the full list of accepted tutorials.
| Tutorial Date | To be announced |
|---|---|
| Conference Dates | December 18–20, 2026 |
Abstract
This tutorial presents a practical methodology for adapting vision-language models (VLMs) to document understanding when high-quality task-specific data is scarce. Vietnamese documents provide the running case study, while the methods apply to other low-resource languages and specialized document collections. The central question is how to establish whether adaptation improves document accuracy enough to justify its data, training, and deployment costs.
Vietnamese forms, invoices, and administrative documents expose challenges that generic benchmarks can obscure: diacritics and Unicode normalization, mixed Vietnamese–English content, degraded scans, dense tables, and changing layouts. The tutorial connects dataset engineering, supervised adaptation, and controlled evaluation so that participants can distinguish useful improvements from template memorization or superficial gains in output formatting.
The technical core covers supervised fine-tuning, implemented through full-parameter updates or parameter-efficient methods such as LoRA and QLoRA. Participants compare adapted models with prompting and OCR-based baselines, inspect errors on unseen layouts, and consider how validation and human review fit into an operational system. The focus is a reusable experimental workflow, illustrated through Vietnamese key information extraction, rather than a survey of model releases.
Audience and Prerequisites
The tutorial is intended for graduate students and researchers in NLP, computer vision, multimodal learning, and document analysis, as well as engineers building OCR, information extraction, and enterprise document systems. Participants should understand basic deep learning and Transformers and be comfortable reading Python. Previous document AI or multimodal training experience is helpful but not required.
Learning Objectives
- Compare OCR pipelines and end-to-end VLMs, and decide when prompting or model adaptation is appropriate.
- Construct multimodal instruction data with consistent schemas, explicit missing-value rules, and leakage-resistant document splits.
- Configure and interpret a small LoRA or QLoRA adaptation experiment, including image resolution, target modules, and memory trade-offs.
- Evaluate recognition, field extraction, document question answering, and structured outputs with task-appropriate metrics.
- Diagnose overfitting, hallucinated fields, and failures on unseen templates, then select validation and human-review mechanisms.
Schedule
The standard format is 240 minutes including a 10-minute break. A 210-minute option retains all six modules and the final discussion, with shorter introductory, data-engineering, and practical segments. Times below are elapsed minutes from the start.
| Elapsed | Session | 4 hours | 3.5 hours |
|---|---|---|---|
| 00–30 | I — From OCR to document intelligence | 30 | 25 |
| 30–70 | II — Dataset engineering | 40 | 30 |
| 70–125 | III — VLM adaptation | 55 | 50 |
| 125–135 | Break | 10 | 10 |
| 135–180 | IV — Guided practical exercise | 45 | 35 |
| 180–210 | V — Evaluation and failure analysis | 30 | 30 |
| 210–230 | VI — System integration | 20 | 20 |
| 230–240 | Discussion and Q&A | 10 | 10 |
| Total minutes | 240 | 210 |
Outline
- Part I — From OCR to document intelligence. Compare OCR–layout–extraction pipelines, layout-aware models, and OCR-free VLMs, using Vietnamese examples to examine text fidelity and layout sensitivity, and identify an appropriate baseline.
- Part II — Dataset engineering for low-resource documents. Define transcription, key information extraction, document VQA and table tasks; cover OCR-assisted annotation, human verification, synthetic examples, schema design, normalization, and splits that reduce leakage.
- Part III — Adapting vision-language models. Select a base model on language and OCR capability, resolution, context limits, licensing and compute; compare prompting with supervised fine-tuning using full updates, LoRA or QLoRA, and relate each decision to memory, stability and generalization.
- Part IV — Guided Vietnamese document adaptation exercise. Follow a reproducible key information extraction example on a small, legally shareable Vietnamese invoice or administrative-document collection, comparing base and adapted models on the same held-out documents.
- Part V — Evaluation and failure analysis. Evaluate transcription, field extraction and document VQA with task-appropriate metrics; separate schema compliance from semantic correctness; measure generalization on unseen templates, degraded scans and out-of-domain documents.
- Part VI — From adapted model to document system. Connect extraction to deterministic checks, databases, retrieval and human review; discuss hybrid OCR/VLM routing, schema validation, confidence calibration and exception handling.
Materials and Delivery
Materials comprise slides, an annotated reading list, a reproducible notebook, preprocessing and evaluation utilities, prompt and schema templates, training configurations, and a versioned code repository. Setup instructions will specify the chosen open model, dependency versions, license conditions, and tested hardware requirements before the tutorial.
Participants may follow the notebook on a laptop. Model training uses a presenter demonstration or an optional compatible GPU environment; no participant GPU is required to inspect the dataset, evaluate saved predictions, or complete the error-analysis exercise. Prepared checkpoints, cached outputs, and a recorded demonstration provide continuity if network access or live training fails.
Only public, synthetic, or appropriately permissioned and de-identified documents will be distributed, and the materials will document dataset provenance, permitted use, preprocessing, splits, and evaluation conventions.
Instructor
Nguyen Vu Son Lam — CMC OpenAI
The presenter's research portfolio spans document layout analysis, OCR enrichment, and Vietnamese information extraction, connecting classical document-processing methods with the practical questions that arise when adapting multimodal models.
Selected publications
- Son Lam Nguyen Vu, Tien Dong Nguyen, and Van Hai Pham. "A Robust Component-Based Template Matching Approach Using Document Layout Graph for Extracting Information." IEA/AIE 2024, Lecture Notes in Computer Science, vol. 14748, Springer, 2024, pp. 10–22.
- Dinh Nguyen Vu, Tien Dong Nguyen, Minh Tuan Dang, Thi Anh Ngoc Ninh, Vu Son Lam Nguyen, Viet Anh Nguyen, and Hoang Dang Nguyen. "Extending OCR Model for Font and Style Classification." CITA 2023, Lecture Notes in Networks and Systems, vol. 734, Springer, 2023, pp. 193–204.
- Lam Nguyen, Tien-Dong Nguyen, Minh-Tuan Dang, Dinh-Nguyen Vu, Viet-Anh Nguyen, and Hoang-Dang Nguyen. "An Empirical Study of Information Extraction from Vietnamese Documents." RIVF 2023, pp. 539–544.
The tutorial also discusses CATI-VLM, a document understanding model developed by CMC Applied Technology Institute, as a dated, source-attributed industry case study of benchmark evidence and the transition from models to document workflows. The hands-on component uses an independently reproducible open-model workflow and does not depend on proprietary CATI-VLM weights or internal CMC datasets.
Contact
For inquiries, please contact: rivf2026@vinuni.edu.vn