XLL Acts Qwen3.5-9B
fine-tune for structural analysis of federal acts
2 min readupdated
An experimental checkpoint based on Qwen/Qwen3.5-9B (9.41B parameters), trained on US federal acts for text structuring tasks. The model takes indexed document text as input and formats its output according to a system prompt and output contract. It is not a production-ready build and does not guarantee legal or semantic accuracy, making it a fit for researchers parsing long official and legal documents.
Key features
- Built on the recent Qwen3.5-9B model with a context window of up to 32,768 tokens.
- Trained on 3,900 federal acts expanded into 10,960 chat presentations (2 epochs, 5,480 optimizer updates).
- Trained using completion-only loss with a focus on structural markup.
- Loads in native FP32 format with optional BF16 casting at inference, requiring no extra adapters or quantization.
What you can do
- Run experiments on structuring and parsing lengthy text documents and official acts.
- Build text-processing pipelines using the transformers library and the model's chat templates.
- Test how the model handles indexed text sequences within its context window.
Example
import json, re
from pathlib import Path
import torch
from huggingface_hub import HfApi, hf_hub_download
from transformers import AutoTokenizer, AutoModelForImageTextToText
repo = "thurstonpowers/xll-acts-qwen3.5-9b"
revision = HfApi().model_info(repo, revision="acts-20261118").sha
system = Path(hf_hub_download(repo, "PROMPT_TRAINED.txt", revision=revision)).read_text()
raw = Path("act.txt").read_bytes().decode("utf-8")
parts = [f'{m.start()}:{m.end()}|{json.dumps(m.group(), ensure_ascii=True)}'
for m in re.finditer(r'\s+|\w+|[^\w\s]', raw) if m.group().strip(' ')]
indexed = f"Document type: acts. Text length: {len(raw)} Unicode characters.\n" + "\n".join(parts)
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision, trust_remote_code=False)
ids = tokenizer.apply_chat_template([
{"role": "system", "content": system}, {"role": "user", "content": indexed}],
tokenize=True, add_generation_prompt=True, enable_thinking=False, return_dict=False)
max_new_tokens = 16384
if len(ids) + max_new_tokens > 32768:
raise ValueError("Whole input plus output reserve exceeds trained context; do not truncate")
model = AutoModelForImageTextToText.from_pretrained(repo, revision=revision,
dtype=torch.bfloat16, attn_implementation="sdpa", trust_remote_code=False).to("cuda").eval()
input_ids = torch.tensor([ids], device="cuda")
with torch.inference_mode():
result = model.generate(input_ids=input_ids, attention_mask=torch.ones_like(input_ids),
max_new_tokens=max_new_tokens, do_sample=False, use_cache=True,
pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id)
text = tokenizer.decode(result[0, len(ids):], skip_special_tokens=True)
prediction = json.loads(text)
Read more
DetailsBenchmark on 200 acts (100 CT + 100 federal): 143/200 (71.5%) produced a structurally valid tree, 55 returned invalid output, and 2 hit the generation limit.
Tested on an RTX 5090 in BF16 with batch size 1, greedy decoding, and flash SDP (65,536-token context, up to 16,384 output tokens).
Trained using transformers 5.15.1 and PyTorch 2.13.0+cu130.
Released under the Apache-2.0 license.










Comments