LexSPLADE (XLM-RoBERTa-XL)
a contextual word encoder readable as a list of terms
2 min readupdated
This model tackles the word-in-context representation task. You wrap a target word in ... tags, and it outputs a 250,004-dimensional sparse vector (a vector where almost all values are zero) mapped to the XLM-R vocabulary. Because of this, the vector coordinates can be read directly as a list of semantically related words—distinguishing between a river bank and a bank account, for example. Built by jadermcs on top of facebook/xlm-roberta-xl (3.48B parameters), the checkpoint was fine-tuned on the balanced multilingual XL-LEXEME dataset, yielding a 5.9 AP gain on cross-lingual pairs over the English-only version. It is well suited for search engine developers, NLP researchers, and specialists working on word sense disambiguation (WSD).
Key features
- Multilingual and cross-lingual retrieval: supports over 20 languages—including Russian, English, Chinese, Arabic, and German—matching word meanings across languages.
- Interpretable vectors: on average, only 219 out of 250,004 vocabulary dimensions are active, making it easy to decode a word's representation as a list of semantically close tokens.
- Efficient target-span pooling: a custom module applies the MLM head exclusively to the tagged word, avoiding heavy logit tensors for the entire sequence.
- Two-stage training: first trained with LoRA (a parameter-efficient fine-tuning method) on English pairs, then fine-tuned in a second stage on 15,244 cleaned examples from XL-LEXEME before merging the adapter weights.
What you can do
- Semantic and lexical search: build hybrid search indexes where the exact meaning of a polysemous word in a specific sentence matters more than an averaged vector of the whole text.
- Cross-lingual term matching: find context-aware equivalents for words and concepts across Russian, English, and other languages.
- Semantic analysis and sense clustering: group word occurrences by meaning and visually inspect which associations the model picked up.
Example
from sentence_transformers import SparseEncoder
import torch
model = SparseEncoder("jadermcs/lexsplade-xlm-roberta-xl-silver-xllexeme", trust_remote_code=True, model_kwargs={"dtype": torch.bfloat16})
sents = ["He sat on the bank of the river.", "She withdrew money from her bank."]
v = model.encode(sents, convert_to_tensor=True).to_dense().float()
print(torch.nn.functional.cosine_similarity(v[0:1], v[1:2], dim=1)) # ~0.06
Read more
DetailsBenchmarks (AP): MCL-WiC test (en-en) — 97.63, MCL-WiC test (ru-ru) — 93.58, cross-lingual en-ru — 93.96, average across 12 languages on XL-WiC test — 79.32.
Requirements: 3.48B parameters; takes around 7 GB VRAM in bfloat16.
Limitations: requires the
trust_remote_code=True flag at runtime; the tagged word must fit within max_seq_length, otherwise the model falls back to returning an embedding of the entire sentence.









Comments