RAG-CPT-Qwen2.5-Coder-3B
a fine-tuned model for RAG architectures
2 min readupdated
This model is the result of continued pre-training of the base Qwen2.5-Coder-3B model, adapted for creating and working with production Retrieval-Augmented Generation (RAG) architectures. It was trained on a special corpus of 8.19 million tokens collected from popular open-source frameworks such as LangChain, LlamaIndex, Qdrant, and others. The model better understands the internal structure and code of RAG systems, helping engineers and developers design and implement search pipelines more efficiently. The output is a ready-to-use language model in bfloat16 and GGUF formats for local deployment.
Key features
- Base architecture Qwen2.5-Coder-3B with 3.09 billion parameters.
- Trained on an 8.19 million token corpus from leading open-source RAG frameworks.
- Optimized using 4-bit QLoRA with paged AdamW 8-bit.
- Cross-entropy loss reduced from 1.54 to 1.09 over 500 training steps.
What you can do
- Apply the model as an improved foundation for generating code and answers in local RAG systems.
- Use it for document analysis and integration with vector databases like Qdrant or Chroma.
- Deploy locally on consumer hardware thanks to available GGUF binaries for edge devices.
Example
The model is used as a base layer for creating RAG application development assistants. You can connect it via local servers (for example, llama.cpp) to generate indexing and search pipeline code.
Read more
DetailsMetrics: cross-entropy loss dropped from 1.54 to 1.09
Optimization: 4-bit QLoRA, 500 steps, paged AdamW 8-bit
Formats: Merged bfloat16 and high-precision GGUF for edge servers
Corpus size: 8.19M tokens (LlamaIndex, LangChain, Haystack, Qdrant, etc.)










Comments