PubMedBERT SemRep accuracy classifier
fact-checking scientific papers
2 min readupdated
This model checks whether a sentence from a biomedical paper actually supports an extracted fact triple (subject, predicate, object). It outputs a CORRECT or INCORRECT label along with a confidence score, helping filter out automated parsing errors. Built on BiomedBERT (PubMedBERT) with 0.11B parameters, it reproduces the classifier from a study on COVID-19 drug repurposing via knowledge graphs. Researchers and NLP engineers can use it to clean up biomedical databases like SemMedDB.
Key features
- Fully reproduces the architecture, training code, and results from Zhang et al. (2021).
- Trained on 7,038 labeled assertions from PubMed abstracts covering over 20 relation types (including
INTERACTS_WITH,TREATS, andINHIBITS). - Compact 0.11B parameter footprint and MIT license make it easy to drop into local pipelines.
- Outputs softmax probabilities for both classes so you can tune the filtering threshold to fit your workflow.
What you can do
- Strip false relations out of SemMedDB before building biomedical knowledge graphs.
- Integrate the model into a SemRep post-processing pipeline using the
transformerslibrary. - Batch-validate large sets of extracted facts from English PubMed abstracts.
Example
import re
from transformers import pipeline
clf = pipeline("text-classification", model="xincan/pubmedbert-semrep-accuracy-classifier")
def norm(s):
return " ".join(re.sub(r"[\W_]+", " ", s).split()).lower()
print(clf(f"{norm('Control of brain tryptophan concentration in rats.')} [SEP] {norm('tryptophan TREATS rats')}"))
Read more
DetailsTest set metrics (703 examples): Precision 0.871, Recall 0.815, F1 0.842 (mean F1 across 10 runs is 0.859 vs. 0.854 in the original paper).
Trained on a single NVIDIA A40 GPU with a 512-token maximum context length.
Limitations: supports English only, does not verify the underlying scientific truth of a claim, and misses roughly 1 in 7 incorrect triples.










Comments