Byte Transformer 85M: A tokenizer-free language model
2 min readupdated
The model reads and writes raw UTF-8 text bytes directly without a tokenizer (a program that splits text into chunks), where each step equals a single byte. The architecture was built for an experimental comparison against BPE-based models (Byte-Pair Encoding, a subword tokenization method) and H-Net at equal compute budgets. It features 85.2M parameters, 12 Transformer layers with an 8,192-byte context, and was trained on a subset of the FineWeb-Edu dataset. It suits researchers and developers exploring alternative tokenization approaches in language models.
Key features
- Tokenizer-free: processes text at the level of individual UTF-8 bytes.
- Architecture: a 12-layer Transformer with a width of 768 and 85.2M parameters.
- Training: utilized RMSNorm, rotary position embeddings, and SwiGLU.
- Comparison: serves as a baseline in a study comparing three different tokenization approaches.
What you can do
- Run raw-byte text generation using the script from the repository.
- Compare model efficiency against the BPE approach on your own text datasets.
- Use the code to explore how language models work without traditional tokenization.
Example
import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
path = snapshot_download("ShadeEngine/bytegpt-byte-85M")
sys.path.insert(0, path)
from model import ByteGPT, GPTConfig
meta = json.load(open(f"{path}/config.json"))
weights = load_file(f"{path}/model.safetensors")
for alias, source in meta["tied_weights"].items():
weights[alias] = weights[source]
model = ByteGPT(GPTConfig(**meta["config"]))
model.load_state_dict(weights)
model.eval().cuda()
prompt = bytes([254]) + "The French Revolution began in".encode("utf-8")
ids = torch.tensor(list(prompt), device="cuda")[None]
with torch.autocast("cuda", dtype=torch.bfloat16):
out = model.generate(ids, max_new_bytes=200, temperature=0.8, top_k=50)
print(bytes(out[0, 1:].tolist()).decode("utf-8", errors="replace"))Read more
DetailsFineWeb-Edu validation: 1.036 bits per byte
Python code: 1.329 bits per byte
HellaSwag accuracy: 35.7%
Hardware: training took about a day on two RTX 3090s
Limitation: a small research model, frequently outputs factual errors










Comments