beetle
Open source · Apache-2.0 · Python
What is beetle?
beetle is a minimal, transparent framework for training tiny language models end-to-end — with zero downloads and a reproducible baseline a newcomer can run on a laptop CPU in a couple of minutes. No GPU, no dataset hunt, no hidden state.
It ships a family of architectures, a bilingual data and tokenizer pipeline, continual-learning extensions, and a HuggingFace-compatible checkpoint format. There are two entry points:
beetlelm-toy— the newcomer path: trains the real PicoDecoder on a tiny in-repo byte corpus on CPU, no downloads, light dependencies. Great for kicking the tires and CI.beetlelm-train— the full research trainer (Lightning, HuggingFace datasets, the architecture zoo, continual-learning modules) for multi-GPU clusters.
Checkpoints from both are standard HuggingFace and reload with from_pretrained.
The paper
The framework, model suite and psycholinguistic evaluations, in full.
Beetle: A Bilingual Model Suite for Modelling Second-Language Processing
1University of Cambridge · 2EleutherAI · 3University of Oxford · Correspondence: sas245@cam.ac.uk
Abstract Bilingual language models (LMs) offer a controlled setting for studying how training conditions shape second-language (L2) behaviour, but prior work typically varies exposure structure, scale, and architecture at once, making it difficult to attribute effects to any single factor. We introduce Beetle, a controlled LM pretraining framework in which tokeniser, target language, training budget, and exposure structure are each independently manipulable. Using Beetle, we train and release 285 bilingual and 45 monolingual open-source LMs with rich checkpoints across a range of exposure schedules, data scales and first languages (L1s). Evaluating on human bilingual and second-language reading-time prediction and grammaticality judgement, we find that staged, temporally structured curricula consistently improve alignment with language-learner reading time compared to balanced bilingual training — with the largest gains at smaller data scales and for typologically closer language pairs.
The framework & why these models exist
Beetle lets you manipulate pretraining conditions one at a time — that is the rationale for the whole suite.
Controlled manipulation of pretraining
Beetle is a language-model pretraining framework that allows controlled manipulation of pretraining conditions. We use it to isolate variables that prior L2-LM work manipulated separately: L2 onset timing, L2 exposure ratio, and continual-learning regularisation. Language exposure in bilingual pretraining is modelled as changing continuously over the course of training, so we can represent gradual or abrupt transitions between languages, periodic bursts of exposure, and clustered, context-dependent input. Holding architecture, tokeniser and data source fixed makes the effect of L1, curriculum and data scale attributable to a single factor.
Architecture
The default backbone, PicoDecoder (Diehl Martinez et al., 2025), is a 125M-parameter LLaMA-style causal decoder — a contemporary LLaMA-class stack (RoPE, SwiGLU, RMSNorm, grouped-query attention) rather than the GPT-2-style decoder of B-GPT.
Tokenisation
Tokenisation is a modular element of the framework, so it can be used to study the benefits of vocabulary overlap and different schemes (SentencePiece, UnigramLM, SuperBPE, BPE) and equal compression.
Beetle models use a BPE tokenizer (HuggingFace tokenizers) with a 50K vocabulary, trained with equal compression so both languages share the same compression rate — each model sees the same amount of information per language in each sequence.
Exposure curricula. We compare five curricula that manipulate how L2 input is distributed over training. Four differ mainly in when L2 is introduced and how the L1:L2 mixture then changes; B4 instead keeps L2 available throughout but concentrates it into discrete episodes. We report the total proportion of training tokens in L2, since curricula that introduce L2 at different points provide different amounts of it. Models are named by curriculum (b1–b5), scale and language pair (e.g. nld-eng).
Balanced
Constant 50:50 mixture throughout; L2 available from the start. L2 = 50%.
Simultaneous
L1 only, then a sigmoid to 50:50 at the halfway point. L2 = 25%.
Sequential
L1 only, then a sigmoid to an L2-dominant 33:67 mixture. L2 = 33.5%.
Classroom
80:20 overall, L2 clustered into intermittent episodes (available throughout). L2 = 20%.
Late
L1 only until 80% of training, then a sigmoid to 50:50 — an L1-attrition setting. L2 = 10%.
Languages, transfer & monolingual baselines
We release 285 bilingual + 45 monolingual models over 21 L1s, all with English as L2 (for reading-time and CEFR data). At 100M FineWeb we cover:
Cross-lingual transfer is measured against matched monolingual baselines (German, Dutch, Chinese at all scales; Russian, Italian, Turkish, Basque at 100M/2B) and against massively multilingual LLMs — Apertus 8B, XGLM 4.5B, Llama 3.1 1B, Gemma 3 270M, Qwen 3 0.6B. Larger scales focus on 9 L1s; BabyBabelLM covers German, Chinese, Dutch.
Learning dynamics & cross-lingual transfer
Because checkpoints are dense near phase boundaries, Beetle exposes when cross-lingual transfer emerges and how it depends on typological similarity, exposure timing and mixture. Sentence-level NLL trajectories on parallel FLORES-200 text track L1 and L2 across training, before and after L2 introduction.
How to use a Beetle model
Every model, tokenizer and dataset is standard HuggingFace. If you use one, cite the paper and note the curriculum, scale and language pair you loaded — that is what the model name encodes.
from transformers import AutoModelForCausalLM, AutoTokenizer
# B1 (balanced) Dutch-English, 100M FineWeb
name = "Beetle-FineWeb-100M/beetle-bilingual-balanced-b1-fineweb-nld-eng"
model = AutoModelForCausalLM.from_pretrained(name)
tok = AutoTokenizer.from_pretrained("Beetle-Data/tokenizer-nl-en")Motivation & related work
Beetle builds on prior bilingual / L2-LM and acquisition work, but differs by holding architecture, tokeniser and data fixed while varying only exposure structure — with a LLaMA-class backbone, fine-grained early checkpoints, matched monolingual controls, and 21 typologically diverse L1s.
Quickstart
1 · Install
$ git clone https://github.com/beetlelm/beetlelm.git
$ cd beetlelm
$ pip install -e .$ pip install "torch>=2.5.1" transformers numpy pyyaml click
$ pip install -e . --no-depsPrefer one command? ./install.sh --toy does a light install plus a smoke train.
2 · Train (CPU, zero downloads)
$ beetlelm-toy --config_path configs/tiny_cpu.yamlfrom beetlelm.toy import train
results = train("configs/tiny_cpu.yaml")
print(results["final_loss"], results["checkpoint_dir"])
# reload the checkpoint — standard HuggingFace
from beetlelm.pico_decoder.pico_decoder import PicoDecoderHF
model = PicoDecoderHF.from_pretrained("runs/tiny_cpu")Point data.train_file at any UTF-8 text file to train on your own corpus, or change one variable with --seed, --max_steps, --d_model.
Tutorials
Step-by-step guides — from a first CPU run to loading a pretrained beetle checkpoint.
Install beetle
Clone the repo and install in editable mode. The light install pulls only what the CPU quickstart needs.
git clone https://github.com/beetlelm/beetlelm.git
cd beetlelm
pip install -e .Train your first model
Train a real PicoDecoder end-to-end on CPU with zero downloads — done in a couple of minutes.
beetlelm-toy \
--config_path configs/tiny_cpu.yamlTrain on your own text
Point data.train_file at any UTF-8 file, and tune the run with a couple of flags.
beetlelm-toy \
--config_path configs/tiny_cpu.yaml \
--seed 42 --max_steps 500 --d_model 128Load a pretrained model
Every beetle checkpoint is standard HuggingFace — reload it with from_pretrained.
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained(
"Beetle-HumanScale/beetle-monolingual-humanscale-eng")Use a beetle tokenizer
Bilingual and human-scale BPE tokenizers are published on the Hub and load the same way.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(
"Beetle-Data/tokenizer-nl-en")
print(tok("Hello, beetle!"))Analyse & evaluate
Score learning dynamics and BLiMP-style minimal-pair accuracy, or play in Colab — no code required.
# minimal-pair evaluation
from beetle_analyze import blimp_accuracy
blimp_accuracy("runs/tiny_cpu")Psycholinguistic evaluation
We evaluate the models against human second-language reading times, grammaticality and structural priming.
Cross-lingual structural priming
| Curr. | FineWeb (24B) | HumanScale | ||
|---|---|---|---|---|
| diff | p | diff | p | |
| Bernolet (genitive) | ||||
| B1 | 1.051 | .0004 | 1.369 | <.0001 |
| B2 | 0.991 | .037 | 1.095 | <.0001 |
| B3 | 0.969 | .051 | 1.216 | <.0001 |
| B4 | 1.252 | .0008 | 0.565 | .005 |
| B5 | 1.075 | .0003 | 0.571 | <.0001 |
Reproduce it in Colab
Open the reading-time, grammaticality and priming analyses in Google Colab — no local setup.
German structural priming
RQ A — cross-lingual structural priming for German–English Beetle models (Bernolet genitive & dative).
Priming across scale
RQ B — how cross-lingual structural priming develops across 100M / 2B / 24B token budgets and over training checkpoints.
Multilingual priming
RQ C — structural priming across multiple L1s, comparing curricula and directionality (L1→L2 vs. L2→L1).
BLiMP grammaticality (human-scale)
Reproduce the human-scale BLiMP grammatical-judgement evaluations (P2/P3) for the Beetle model suite.
FLORES sentence NLL
Sentence-level NLL trajectories on parallel FLORES-200 devtest sentences for the 100M model variants.
Interpretability
Hierarchical / layerwise interpretability analyses from the paper’s learning-dynamics appendices.
CPU smoke test
A minimal CPU notebook to check the analysis stack end-to-end before running the GPU evaluations.
Models, tokenizers & datasets
Every beetle model, tokenizer and dataset — published open on the Hugging Face Hub.
The suite released with the paper: 285 bilingual and 45 monolingual models across curricula (b1–b5), scales (100M/2B/24B, humanscale) and language pairs. See the framework for the rationale, curricula and architecture; each model card on the Hub documents its curriculum, scale and language pair.
Monolingual
Human-scale beetle models trained on a single language (English, Dutch, German, Chinese …).
Bilingual & continual
L2-acquisition schedules (balanced, simultaneous, sequential, classroom, late) with EWC & LAMOL variants.
Data & tokenizers
BabyBabel corpora, pre-tokenized shards, ~28B raw dumps, and 30+ bilingual BPE tokenizers.
Type to filter across every repository below. Each chip links to its page on Hugging Face.
No repositories match your filter.
Beetle-Data
Bilingual data pipeline: BPE tokenizers, raw & pre-tokenized corpora.
Tokenizers 34
BabyBabel corpora 99
Pre-tokenized shards 108
Raw corpora (≈28B) 84
FineWeb corpora 1
Beetle-HumanScale
Human-scale (100M-word) tokenizers, corpora & trained beetle models.
Monolingual models 4
Bilingual & continual-learning models 27
BPE tokenizers 45
BabyBabel corpora 4
Datasets 83
Beetle-FineWeb
Beetle models trained on FineWeb (English/Dutch/German).
Bilingual & continual-learning models 16
Beetle-FineWeb-100M
Beetle models trained on the 100M-token FineWeb slice.
Monolingual models 21
Bilingual & continual-learning models 161
FineWeb corpora 6
Beetle-FineWeb-2B
Beetle models trained on the 2B-token FineWeb slice.
Monolingual models 10
Bilingual & continual-learning models 51
Beetle-FineWeb2-24B
Beetle models trained on FineWeb2 (24B tokens).
Bilingual & continual-learning models 43
Beetle-FineWeb3-24B
Beetle models trained on FineWeb3 (24B tokens).
Monolingual models 11
Datasets 1
The beetle ecosystem
Four small, reproducible repositories that fit together end-to-end.
beetlelm
The core framework: train tiny language models end-to-end on CPU with zero downloads and a reproducible baseline. The architecture zoo, tokenizer, and HuggingFace-compatible checkpoints live here.
Open repositorybeetle-data
A minimal, reproducible bilingual data-preparation pipeline: build a corpus, train a BPE tokenizer, pretokenize into shards, and report stats — for small language models.
Open repositorybeetle-analyze
A minimal, reproducible toolkit for analysing language-model evaluation output via BLiMP-style minimal-pair accuracy.
Open repositorybeetle-explorer
Load and play with Beetle language models in Colab — no coding required. A non-technical companion to the beetlelm training framework.
Open repositoryWhy beetle
Zero downloads
The toy path trains on an in-repo byte corpus. No dataset hunt, no network, works offline and in CI.
Reproducible
Deterministic on CPU given the seed. Expected numbers are committed as baselines you can diff against.
Transparent
Everything is driven by a readable YAML config. Comprehensive checkpoints for learning-dynamics research.
Scales up
Graduate from the toy trainer to Lightning, HF datasets, the architecture zoo, and continual learning.
Cite beetle
If beetle is useful in your research, please cite the paper:
@inproceedings{salhan2026beetle,
title = {Beetle: A Bilingual Model Suite for Modelling
Second-Language Processing},
author = {Salhan, Suchir and Arnett, Catherine and
Michaelov, James A. and Buttery, Paula},
booktitle = {Proceedings of EMNLP},
year = {2026},
url = {https://beetlelm.github.io/}
}Get involved & contact
beetle is open source and welcomes issues, pull requests, and questions.
Acknowledgements
The Beetle framework and models were trained by:
Site made by Suchir Salhan. beetle is released under Apache-2.0.