openT2L-MiniCPM-V-4.7-1B-docs - Zero-training LoRA adapter generator for documents
2 min readupdated
This hypernetwork (a neural network that generates weights for another model) by author ASHu2 solves the problem of slow fine-tuning for new document layouts. You provide just 4 examples of unfamiliar page types with the required JSON output, and in a single forward pass it creates a ready-to-use rank 8 LoRA adapter (a small plugin) for the MiniCPM-V-4.7-1B multimodal model. Unlike classical fine-tuning, this triggers no training loop, and the resulting adapter is saved in the standard PEFT format - it can be immediately merged into the base or converted for llama.cpp. The tool will be useful for developers of recognition and data extraction systems for invoices, tables, and forms who need to quickly adapt a VLM model to new English and Chinese formats.
Key features
- Instant adaptation: LoRA weight generation happens in a single pass based on 4 labeled pages without gradient fine-tuning.
- Standard PEFT format: the output is a regular adapter directory (rank 8, alpha 16) compatible with Hugging Face pipelines and
llama.cpp. - Text-to-LoRA architecture: a 72.2M parameter hypernetwork (3x512 MLP) modifies 48 language module layers without touching the visual encoder.
- Noticeable accuracy boost: across 78 test tasks the model had never seen during training, the average F1 score rose from 0.728 to 0.881.
What you can do
- Parsing new forms and receipts: quickly assemble specialized adapters to extract fields from invoices, receipts, payslips, and waybills using 4 samples.
- Extracting complex tables: adapt MiniCPM-V-4.7-1B to convert financial tables (for example, FinTabNet level) into structured JSON.
- Local deployment: generate an adapter on the server, merge it with the base 1B model, or convert it to the
llama.cppformat to run on devices.
Example
from huggingface_hub import snapshot_download
from peft import PeftModel
from t2l import load_hypernet, emit_adapter
gen, cfg = load_hypernet(snapshot_download("ASHu2/openT2L-MiniCPM-V-4.7-1B-docs"), device="cuda")
# support — тензор эмбеддингов из 4 размеченных примеров страниц
emit_adapter(gen, support, "my_adapter/", base_model="openbmb/MiniCPM-V-4.7-1B")
model = PeftModel.from_pretrained(model, "my_adapter/")Read more
DetailsBenchmarks (Gate 1, 78 new tasks, field-level F1 metric): overall result grew from 0.728 to 0.881 (+0.154), on synthetic forms from 0.813 to 0.957, on public tasks (FinTabNet, ChartQA) from 0.339 to 0.533.
Training: 333 tasks, 6000 steps on a single A10G VRAM (~5 hours), base languages are English and Chinese.
Limitations: weak boost on charts (+0.01 to +0.27); when working, you must strictly follow the prompt format and processor settings (
downsample_mode="16x", max_slice_nums=9).License: CC-BY-NC-4.0 (non-commercial use only due to training dataset licenses).










Comments