gt-embed-minimal-v1
Lightweight Human DNA Embeddings
2 min readupdated
Created by Bell Eapen, this compact model turns protein-coding DNA sequences (CDS) into 256-dimensional vector representations (embeddings). Under the hood is a 2-layer Transformer Encoder with ~2M parameters, trained via contrastive learning on ~17,000 human transcripts from GENCODE v50 (GRCh38 assembly). Its standout feature is codon-level tokenization (triplets of nucleotides that code for amino acids) via Genomic Tokenizer instead of single-character parsing, allowing the model to capture structure at the protein synthesis level. It is well-suited for bioinformaticians and developers looking for rapid prototyping and pipeline debugging.
Key features
- 26-token codon vocabulary: 20 tokens for amino acid codons and 6 special tokens (
[CLS],[SEP],[BOS],[MASK],[PAD],[UNK]). - Ultra-lightweight architecture: just ~2M parameters (2 layers, 4 attention heads, 256 hidden size), making it convenient for testing and running even on a CPU.
- Normalized output: produces L2-normalized 256-dimensional vectors for sequences up to 2,048 tokens (~6,144 nucleotides) long.
- Part of a model lineup: this is the minimal development release—the family also includes planned Small (~10M parameters) and Regular (~50M) variants for higher-precision tasks.
What you can do
- Gene prioritization: plug it into research pipelines when studying rare hereditary disorders and searching for disease-associated genes.
- Variant annotation: score and cluster novel genetic variants in protein-coding regions.
- Knowledge graphs: build vector links between genes and diseases in bioinformatics graphs.
Example
import torch
from genomic_tokenizer import GenomicTokenizer
tokenizer = GenomicTokenizer(model_max_length=2048)
cds_sequence = "ATGGCCGCCTACAAGCAGCAGTCTGGGGAAGACGGGCTACCTTGCCCCTGA"
tokens = tokenizer(cds_sequence)
input_ids = torch.tensor([tokens['input_ids']])
with torch.no_grad():
embedding = model(input_ids) # shape: [1, 256], L2-normalizedRead more
DetailsTraining: contrastive learning on ~17,000 human protein-coding transcripts (GENCODE v50, GRCh38.p14 genome).
Hardware & requirements: ~2M parameters, runs on CPU with PyTorch 2.5.1 and the genomic-tokenizer / gtembed packages.
Limitations: human genome only (excludes non-coding and regulatory regions), max length of 2,048 tokens, not clinically validated and not intended for medical diagnosis.
License: Apache 2.0.










Comments