MedSAE for bge-large-en-v1.5
interpretable medical retrieval via sparse autoencoders
2 min readupdated
A collection of sparse autoencoders (SAEs — models that decompose dense vectors into a set of distinct, human-understandable features) from the doctolib-lab team, accompanying their ML4H 2026 paper. The model takes 1024-dimensional embeddings from the popular BAAI/bge-large-en-v1.5 encoder and maps them into a sparse code of 3072 latent features, with only k features active at a time (64 or 256). Unlike standard dense vectors, each active feature corresponds to a specific medical concept, and the k256_m3072 configuration even outperforms the base model in retrieval accuracy on the target domain. Useful for researchers and developers building search systems over English biomedical literature who need to see why a document matched a query.
Key features
- Two ready-to-use checkpoints tailored for different goals:
k64_m3072for maximum feature interpretability andk256_m3072for higher retrieval accuracy. - Retrieval runs via standard cosine similarity between sparse query and document codes, requiring no complex extra layers.
- Trained exclusively on the CURE dataset (1,400 medical literature queries) using a combined MSE and InfoNCE loss function.
- Weights are released in
safetensorsformat alongsidecfg.jsonunder the non-commercial CC BY-NC 4.0 license.
What you can do
- Build explainable search over English medical research papers by surfacing the query's active latent concepts alongside each retrieved document.
- Use the
k64_m3072checkpoint in interpretability research to analyze which medical topics the BGE model encodes. - Drop in
k256_m3072sparse codes to replace standard BGE embeddings in biomedical and bioinformatics retrieval pipelines.
Example
import json, torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from sentence_transformers import SentenceTransformer
w = load_file(hf_hub_download("doctolib-lab/medsae-bge-large-en-v1.5", "k64_m3072/sae.safetensors"))
cfg = json.load(open(hf_hub_download("doctolib-lab/medsae-bge-large-en-v1.5", "k64_m3072/cfg.json")))
# После кодирования через BGE (с префиксом для query) применяем TopK + ReLU к (x - b_pre) @ encoder.T + bias
Read more
DetailsNDCG@10 (averaged across 3 seeds): on the CURE dataset,
k256_m3072 scores 60.50% compared to 53.06% for the original BGE; on R2MED Medical Sciences, 29.05% vs. 27.78%; and on R2MED Bioinformatics, 28.10% vs. 27.04%.On several out-of-domain datasets (NFCorpus, MIRIAD, R2MED Biology), the sparse codes trail the baseline BGE.
Limitations: English-only and restricted to scientific literature; the model has not been evaluated on clinical notes and is not intended for clinical decision-making.










Comments