mistral-7b-pubmedqa-loraplus
LoRA+ adapter for answering biomedical questions
2 min read
This repository provides a LoRA+ adapter (a small add-on for a base model for targeted fine-tuning) for the mistralai/Mistral-7B-Instruct-v0.3 model, trained on the PubMedQA dataset. The model analyzes an abstract from PubMed and a research question, then writes a short conclusion and gives a final verdict: yes, no, or maybe. The project was created by author iamaber for research purposes and to ensure reproducibility in the field of biomedical question answering.
Key features
- Improved answer accuracy on the official PubMedQA test set compared to the base model.
- Use of the LoRA+ training method with fixed hyperparameters and reproducible training code.
- Thoroughly documented evaluation metrics, including accuracy and macro-F1.
- Open-source code for training, evaluation, and inference, available in the linked GitHub repository.
What you can do
- Run model inference using the
peftlibrary on top of the base Mistral-7B-Instruct-v0.3 model. - Use the adapter as a starting point for researching language model fine-tuning methods on biomedical texts.
- Test the generation of concise logical conclusions based on scientific abstracts from PubMed.
- Study the training configuration and evaluation results on a subset of Medical MMLU tasks.
Example
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model_id = "mistralai/Mistral-7B-Instruct-v0.3"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
base_model_id,
dtype=torch.bfloat16,
device_map="auto",
);
model = PeftModel.from_pretrained(base_model, "iamaber/mistral-7b-pubmedqa-loraplus")
model.eval()
messages = [{"role": "user", "content": "Context: \n\nQuestion: \n\nWrite a short conclusion supported by the abstract, then give the final answer on its own line as 'Answer: yes', 'Answer: no', or 'Answer: maybe'."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
with torch.no_grad():
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))Read more
DetailsAccuracy on PubMedQA: 0.7280 (base model: 0.6600).
Macro-F1 on PubMedQA: 0.5831 (base model: 0.5300).
Training dataset: qiaojin/PubMedQA (pqa_labeled), 744 training examples.
Limitations: for research purposes only, not a model suitable for clinical practice.
License: Apache-2.0.










Comments