lora-roberta-base-glue
LoRA adapters for RoBERTa-base on GLUE tasks
2 min readupdated
A set of weights and code to reproduce LoRA (a lightweight add-on on top of a base model for efficient fine-tuning) training across five GLUE benchmark tasks (SST-2, MRPC, CoLA, RTE, STS-B). Built on a frozen roberta-base (FacebookAI/roberta-base) with rank r = 8 and alpha = 8, the adapters target the query and value projections across all 12 attention layers alongside a trained classification head. This repository is useful for researchers and developers exploring practical LoRA applications on classic language models.
Key features
- Five pre-trained weight sets for SST-2, MRPC, CoLA, RTE, and STS-B.
- From-scratch LoRA implementation built on top of a frozen
roberta-basemodel. - Each adapter includes a trained classification head.
- Detailed evaluation results across 5 seeds for every task.
What you can do
- Benchmark the pre-trained adapters on GLUE validation splits.
- Use the repository code as a base for your own LoRA adaptation experiments.
- Study how skipping the intermediate MNLI step affects adapter performance (for example, on RTE).
Example
import json
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoModelForSequenceClassification, AutoTokenizer
task = "rte"
root = snapshot_download("Bartu0nder/lora-roberta-base-glue")
cfg = json.load(open(f"{root}/{task}/config.json"))
tok = AutoTokenizer.from_pretrained(cfg["base_model"])
model = AutoModelForSequenceClassification.from_pretrained(
cfg["base_model"], num_labels=cfg["num_labels"], attn_implementation="sdpa")
state = load_file(f"{root}/{task}/adapter.safetensors")
print(f"Loaded adapter for {task} successfully")
Read more
DetailsValidation metrics (median across 5 seeds): SST-2 (Accuracy) — 94.8, MRPC (Accuracy) — 89.2, CoLA (Matthews corr.) — 63.1, RTE (Accuracy) — 78.7, STS-B (Pearson corr.) — 91.0.
Parameters: 294,912 trainable LoRA parameters and 592,130 classification head parameters.
Limitations: Adapters for MRPC, RTE, and STS-B were trained directly from
roberta-base without an intermediate MNLI step, which accounts for score variations on RTE.License: MIT.










Comments