Llama-3-8B LoRA Collection
a collection of 10 task-specific adapters
2 min readupdated
A collection of 10 ready-to-use LoRA adapters (lightweight adjustments on top of a base model) trained on meta-llama/Meta-Llama-3-8B using popular open datasets. Each adapter targets a specific task, from general instruction following and multi-turn chat to math, reasoning, and code generation. This set is ideal for researchers and developers who want to quickly test the model across specific scenarios without training from scratch.
Key features
- 10 adapters covering different domains: Alpaca, Dolly, OpenAssistant, UltraChat, WizardLM, OpenOrca, GSM8K, MetaMath, CodeAlpaca, and Magicoder.
- Validated training on popular SFT datasets with a fixed number of epochs and tracked eval loss.
- Broad coverage ranging from general chat to specialized coding and grade-school math.
- Unified prompt format across all adapters, since the base model lacks a built-in chat template.
What you can do
- Generate code: use the CodeAlpaca or Magicoder adapters to write programs in various programming languages.
- Solve math problems: load the GSM8K or MetaMath adapters for accurate, step-by-step calculations.
- Build a chatbot: apply UltraChat or Guanaco for coherent multi-turn conversations.
- Test multi-LoRA setups: serve multiple modules simultaneously in inference engines like vLLM.
Example
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
repo = "Jianshu001/Llama-3-8B-LoRA-Collection"
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
base = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, repo, subfolder="llama3-8b-lora-gsm8k")
prompt = "<|begin_of_text|>### User:\nWhat is 15% of 240?\n\n### Assistant:\n"
ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
Read more
Details• Architecture: LoRA (r=16, alpha=32, ~42M trainable parameters)
• Training: AdamW, lr 2e-4, bf16 on NVIDIA H200
• Metrics: 8-72% eval loss improvement depending on the dataset
• License: Meta Llama 3 Community License plus the respective dataset license










Comments