Ocean Blue: Blue3
Qwen2.5 fine-tune focused on coding and self-distillation
2 min read
The Ocean Blue: Blue3 model is a language model (LLM, a computer program for generating text) based on Qwen2.5-7B-Instruct, fine-tuned using a special cycle with the larger Qwen2.5-72B-Instruct model acting as a teacher. During training, the model solved tasks with program verification: answers were verified, after which the model attempted more complex tasks, including those the teacher failed to solve, and trained on its own successful solutions. A small personality layer was also added to shape a unique voice.
Key features
- Base model: Built on
Qwen2.5-72B-Instructusing the Ocean Blue training cycle. - Self-distillation: Trained on its own successful and more complex solutions (stage 2).
- Personality layer: 304 responses written by the teacher in the model's signature style and selected by a judge.
- License: Weights are distributed under the Apache-2.0 license.
What you can do
- Code generation: Apply the model to solve complex programming tasks and debug code.
- Bot integration: Use as a foundation for chatbots with a custom communication style.
- Research: Test the capabilities of language model self-distillation and fine-tuning.
Example
The model accepts text prompts and programming tasks, outputting detailed and verified solutions based on its training.
Read more
Details• Results on held-out test: 53.3% after self-distillation (base is 12.4%, 72B teacher is 36.9%).
• Results on complex tasks (Deep water): 49.0% (base is 7.0%, teacher is 38.0%).
• Architecture: 7.62 billion parameters (7.62B),
transformers library.• Training: Used 803 verified teacher responses and 731 successful model solutions.










Comments