Beetle: A Bilingual Model Suite for Modelling Second-Language Processing

Beetle, built for AI researchers, multilingual NLP and model builders. Open models. Open training. Open questions. beetlelm.github.io

TL;DR: We introduce Beetle, a controlled language-model pretraining framework in which tokeniser, target language, training budget and exposure structure are each independently manipulable. Using Beetle we train and release 285 bilingual and 45 monolingual open-source models with rich checkpoints across a range of exposure schedules, data scales and first languages. Evaluating against human bilingual and second-language reading-time and grammaticality-judgement data, we find that staged, temporally structured curricula consistently improve alignment with language-learner reading time over balanced bilingual training, with the largest gains at smaller data scales and for typologically closer language pairs.

Bilingual language models offer a controlled setting for studying how training conditions shape second-language (L2) behaviour. But prior work typically varies exposure structure, scale, and architecture at once, making it difficult to attribute any effect to a single factor. If a staged bilingual schedule helps, is it the schedule, the amount of second-language data, or the model, and under which first language?

Beetle is designed to make these questions answerable. It is a controlled pretraining framework in which the tokeniser, target language, training budget, and exposure structure are each independently manipulable. Holding the others fixed makes the effect of first language, curriculum and data scale attributable to one cause, enabling systematic, comparable experimentation. The exposure structure is varied through five curricula, B1–B5, that change when the second language is introduced and how the per-language data distribution unfolds across training steps.

The Beetle framework and its B1–B5 exposure curricula — Beetle pretrains bilingual models under controlled conditions. Each facet is a structured exposure curriculum (B1–B5) that varies the per-language data distribution across training steps, while tokeniser, architecture and data source stay fixed.

Using this framework we train and release 330 open models (285 bilingual and 45 monolingual) across five curricula, three data scales (100M, 2B, 24B tokens) and 21 first languages, and evaluate them against human data on second-language reading-time prediction, grammaticality judgement, and cross-lingual structural priming. Structured, temporally staged exposure schedules align better with human second-language reading times than balanced bilingual training, with the largest gains at smaller scales and for typologically closer language pairs.

See for yourself: read some of our results · explore the models · read about the framework.

Read the paper · use the models · apply the framework

Beetle is three things at once: a scientific contribution, a set of open resources you can use today, and a framework for running your own controlled experiments.

① The paper

What we found. Structured, temporally staged exposure schedules align better with human second-language reading times, across 330 released models. Read the research →

② The models

Explore them now. 285 bilingual and 45 monolingual open models, thousands of checkpoints, 21 first languages, at 100M, 2B and 24B tokens. Download them, inspect training histories and compare behaviour. Browse the suite.

③ The framework

Run your own experiments. Manipulate tokeniser, target language, training budget and the B1–B5 exposure schedules independently to build your own controlled model families. Apply the framework.

The science behind Beetle

The full framework, all five curricula, every evaluation, and the complete results are in our EMNLP 2026 paper (Oral, top 3%). Interested in working with Beetle? Collaboration enquiries are welcome; the most exciting applications may be questions we have not asked yet.

Read the EMNLP 2026 paper Read the PDF sas245@cam.ac.uk

BibTeX

@inproceedings{salhan2026beetle,
  title     = {Beetle: A Bilingual Model Suite for Modelling Second-Language Processing},
  author    = {Salhan, Suchir and Arnett, Catherine and Michaelov, James A. and Buttery, Paula},
  booktitle = {Proceedings of EMNLP},
  year      = {2026},
  url       = {https://arxiv.org/abs/2609.22633}
}