The Beetle Framework

Overview

Introduction

beetlelm trains bilingual, monolingual and polyglot language models and saves checkpoints throughout training on a logarithmic schedule (step 0, 1, 2, 4, 8, ... 65536), so a finished run is also a record of how the model got there. Every checkpoint is a standard Hugging Face model. Reload one with from_pretrained and compare it to the next.

Design philosophy

beetlelm is deliberately small. The aim is not a new training architecture or a maximally tuned trainer, but a clear one you can read in an afternoon and fork. You can find optimizer.step() without hunting for it. A small, self-contained example runs locally with no downloads, so you can check the install and follow the training loop before you move to a cluster. The released models are trained on GPUs at 100M, 2B and 24B tokens.

Code structure

The architecture zoo

Models live under src/beetlelm/. The default is pico_decoder, a Llama-style decoder with grouped-query attention, rotary embeddings, SwiGLU and RMSNorm. Next to it sit bgpt, beetle_moe, beetle_ssm, geo_bilingual, hnet and mambabyte. Each is a plain, documented implementation you select from a config.

Data and tokeniser

The bilingual/ module builds corpora and trains tokenisers; bytes/ provides the byte-level tokeniser used by the quickstart. Training data comes from beetle-data (guide 2) and loads through Hugging Face datasets in streaming mode.

Continual learning

The continual/ module covers learning a second language after the first: EWC, MAML and SMLMT, replay, LoRA, and layer growth. These let you test whether new knowledge reinforces or interferes with what a model already knows.

Setup

Clone and install

beetlelm needs Python 3.10 or newer. The full install pulls the training stack (Lightning, datasets, Weights & Biases, the Hugging Face Hub).

$ git clone https://github.com/beetlelm/beetlelm.git
$ cd beetlelm
$ pip install -e .

Minimal install

For the small local example you do not need the full training stack. Install the few packages it uses and skip the rest. Weights & Biases is offline by default; nothing is uploaded unless you turn it on.

$ pip install "torch>=2.5.1" transformers numpy pyyaml click
$ pip install -e . --no-deps

Getting started

A local example

beetlelm-toy trains a 0.72M-parameter PicoDecoder on the bundled English and Spanish byte corpus and writes a Hugging Face checkpoint plus metrics.json to runs/tiny_cpu/. It is a small, self-contained run for checking the install and seeing the training loop work before you scale up.

$ beetlelm-toy --config_path configs/tiny_cpu.yaml

The same run is available from Python:

from beetlelm.toy import train
results = train("configs/tiny_cpu.yaml")
print(results["final_loss"], results["checkpoint_dir"])

The research trainer

beetlelm-train runs the full trainer on real datasets and the architecture zoo. Configs live in configs/ and cluster launchers in script/. Add --validate_only to check a config without a GPU.

$ beetlelm-train --config_path configs/pico_default.yaml
$ beetlelm-train --config_path configs/pico_default.yaml --validate_only

Configuration

A run is one YAML file. The example config is small enough to read in full. It picks the pico architecture, a 256-token byte vocabulary, and trains for 200 steps.

model:
  architecture: pico
  params: {n_layers: 4, d_model: 128, max_seq_len: 128}
tokenizer: {type: byte}        # 256-token UTF-8 vocabulary
training: {max_steps: 200, batch_size: 16, lr: 3.0e-3, device: cpu}

The released models use the same format at scale. This is a trimmed excerpt of a real 24B-token FineWeb config: a 14-layer, 768-dimensional PicoDecoder, the B1 exposure curriculum, trained for 91,552 steps on eight GPUs. The data_mix field is where you change the curriculum (B1 to B5); the language streams are pretokenised datasets from beetle-data.

model:
  architecture: pico
  params:
    n_layers: 14
    d_model: 768
    attention_n_heads: 12
    attention_n_kv_heads: 1       # grouped-query attention
    max_seq_len: 512
    vocab_size: 50000
data:
  data_mix: B1                      # the exposure curriculum (B1..B5)
  lang_sources: [Beetle-Data/de-28B]
  l2_streams:   [Beetle-Data/en-for-de-28B]
training:
  max_steps: 91552                 # 24B tokens / 262,144 per step
  target_effective_batch: 512      # 512 x 512 seq_len = 262,144 tokens/step
  seq_len: 512
  fabric: {devices: 8, precision: bf16-mixed, strategy: ddp}

Training budgets

The three scales are set by the token total and the step count, with the same effective batch. The defaults live in src/beetlelm/config/.

Exposure curricula (B1 to B5)

The point of Beetle is to change when and how much a model sees its second language. That schedule is a single field, data_mix, in the data block of a config. The language streams are pretokenised datasets from beetle-data; the curriculum decides how they are mixed across training.

data:
  data_mix: B3                          # the exposure curriculum
  lang_sources: [Beetle-Data/de-28B]        # L1 stream
  l2_streams:   [Beetle-Data/en-for-de-28B]  # L2 (English) stream
  pretokenized: true

The five curricula differ in when the second language starts, the final L1:L2 ratio, and how the mixture changes over training.

Define a new curriculum

A named curriculum is resolved by DataMixConfig in src/beetlelm/bilingual/. Internally every schedule is a sequence of phases over normalised training progress t in [0, 1]: each phase fixes a per-language sampling weight for a span of training. To define your own, give data_mix an explicit phase list instead of a named preset.

data:
  lang_sources: [Beetle-Data/de-28B]
  l2_streams:   [Beetle-Data/en-for-de-28B]
  data_mix:
    phases:                       # weights are [L1, L2] and sum to 1
      - {start: 0.0, end: 0.4, weights: [1.0, 0.0]}   # L1 only
      - {start: 0.4, end: 0.7, weights: [0.5, 0.5]}   # ramp in L2
      - {start: 0.7, end: 1.0, weights: [0.3, 0.7]}   # L2-dominant

Phase boundaries double as event markers: in continual-learning runs they trigger an EWC snapshot. The same phase form extends to more than two languages for trilingual and multilingual mixtures.

Launch a training sweep

A controlled study varies one factor at a time across many runs, so Beetle generates configs programmatically and submits them in bulk. configs/generate_p1_sweep.py writes a whole tree of YAMLs (per curriculum, per language pair, per strategy), and the launchers under script/launch/ submit them to a cluster with crash-resilient resume.

$ python configs/generate_p1_sweep.py          # write the config tree
$ LANGS="deu nld zho" TIERS="B1 B2 B3" \
      bash script/launch/launch_fineweb_24b_b1_b3_sweep.sh
$ bash script/launch/launch_multilang_b1b5_sweep.sh   # all curricula, many languages

Each launcher takes LANGS and TIERS (the curricula) as environment variables and skips runs that are already finished, so a sweep can be stopped and resumed.

Overview

A training run needs a tokeniser and tokenised data. beetle-data produces both. It streams raw multilingual corpora, decontaminates them against the evaluation benchmarks, trains a BPE tokeniser, and pretokenises into Arrow or Parquet shards. It ships the production pipeline and a toy path that runs locally with no downloads.

Setup

Install from PyPI, or clone for the editable version. The core install needs no GPU. The curriculum extra adds the GPU BeetleStream v2 stack and is optional.

$ pip install beetle-data
$ # or, from source:
$ git clone https://github.com/beetlelm/beetle-data
$ cd beetle-data
$ pip install -e .                 # core pipeline (no GPU needed)
$ pip install -e .[curriculum]     # + BeetleStream v2 GPU stack

Getting started

One command runs the whole pipeline on a bundled Spanish and English corpus, with no external downloads. It writes the tokeniser, the tokenised shards, and a stats file.

$ beetle-data run --out-dir beetle_out
# beetle_out/tokenizer.json, beetle_out/pretokenized/*.parquet, beetle_out/stats.json

The four steps are also separate commands. Each falls back to the bundled corpus; point --data-dir at a folder of <lang>.txt files to use your own text.

$ beetle-data build-corpus    --l1 es --out corpus.txt
$ beetle-data train-tokenizer --corpus corpus.txt --out tok.json --vocab-size 8000
$ beetle-data pretokenize     --tokenizer tok.json --corpus corpus.txt --out-dir shards --seq-len 128
$ beetle-data stats           shards

The same steps are available from Python.

from beetle_data import iter_bilingual, train_tokenizer, pretokenize, corpus_stats

tagged = list(iter_bilingual("es", "en"))      # [(text, "L1"|"L2"), ...]
tok = train_tokenizer((t for t, _ in tagged), vocab_size=8000)
stats = pretokenize(tagged, tok, out_dir="pretokenized", seq_len=128, lang="es-en")

Scaling up

The production pipeline streams corpora from Hugging Face and pretokenises at billion-token scale. The released bilingual runs draw on a 28B-token stream per language, of which 24B are used for training. Set HF_TOKEN first; no credentials are stored in the repo. Static mode runs three stages: build a benchmark index, stream and decontaminate against evaluation data with a 13-gram filter, then pretokenise to Arrow.

$ export HF_TOKEN=...
$ python pipeline/run_pipeline.py

Overview

beetle-analyze scores model evaluation output as BLiMP-style minimal-pair accuracy. Beyond the quickstart it includes the analyses reported in the Beetle paper: reading-time alignment against MECO, learner grammaticality on BLiSS, English grammatical competence on BLiMP, FLORES-200 perplexity trajectories, and cross-lingual structural priming. It also ships representational tools: CKA, probing, logit lens, and sparse autoencoders.

Setup

Clone and install the editable package. The optional extras pull heavier dependencies only when you need them.

$ git clone https://github.com/beetlelm/beetle-analyze
$ cd beetle-analyze
$ pip install -e .
$ pip install -e .[stats]        # + [eyetracking] [saes] as needed

Getting started

The quickstart scores the bundled toy data and prints an overall minimal-pair accuracy of 0.6000. Change the scoring rule with --scoring mean to get 0.80. Point --input at your own results, write outputs with --json-out and --plot-out, and add --bootstrap for confidence intervals.

$ beetle-analyze blimp
# overall minimal-pair accuracy: 0.6000
$ beetle-analyze blimp --scoring mean --json-out out.json
# overall minimal-pair accuracy: 0.80

Running the full suite

One runner drives the research analyses across a model family. Pick a family and the languages to score. Individual tools are also exposed as console scripts.

$ python analyze/run.py --all --family fineweb --langs deu nld zho eng
$ beetle-cka             # representational similarity
$ beetle-probe           # linear probes
$ beetle-dev-trajectory  # capability development over training

Reproduce the paper's results

Each evaluation in the paper has a runner under evals/. Run one against a model and language set, and read the result straight out.

$ python evals/eval_meco.py             # L1 reading-time alignment (MECO)
$ python evals/eval_meco_english_l2.py  # second-language (English) reading times
$ python evals/eval_bliss.py            # learner grammaticality (BLiSS)
$ python evals/eval_blimp.py            # English grammatical competence (BLiMP)
$ python evals/eval_flores_ppl.py       # FLORES-200 NLL trajectories
$ python evals/eval_priming.py          # cross-lingual structural priming

The exact paper runs, figures and significance tests are wrapped in scripts/. The priming tiers (run_priming_tier1_final.sh and up) and priming_emnlp_paper_figures.py regenerate the priming results; blimp_significance.py, meco_significance.py and priming_stats.py reproduce the reported statistics.

$ bash scripts/run_priming.sh
$ python scripts/priming_emnlp_paper_figures.py

What it measures

These are the evaluations reported in the Beetle paper, and the ones the toolkit reproduces.

• MECO, second-language reading times: does model surprisal predict how long learners spend reading?
• BLiSS, learner grammaticality: can the model tell acceptable from unacceptable sentences the way learners do?
• BLiMP, English grammatical competence across first languages and curricula.
• FLORES-200 negative log-likelihood trajectories, tracking modelling difficulty across training.
• Cross-lingual structural priming: does a structure seen in one language shift the model's expectations in another?

As an example of the last, the 24B-token models show a robust priming effect on the Bernolet genitive. The table reports the surprisal difference (in nats) between congruent and incongruent primes, with permutation p-values.

Full results and methods are in the EMNLP 2026 paper.

Evaluation data

The analyses build on these benchmarks and resources.

• MECO: multilingual eye-tracking reading times for first- and second-language readers.
• BLiSS: learner-style minimal pairs for acceptability discrimination.
• BabyLM 2025: related work on sample-efficient language modelling (2025.babylm-main.13).
• EMNLP 2025 demo: the demonstration paper for the Beetle tooling (2025.emnlp-demos.22).
• CDL 2026: related work on developmental language modelling (2026.cdl-1.15).

No-code explorer

Open notebooks/Beetle_Model_Explorer.ipynb in beetle-explorer, pick a collection, language, curriculum and checkpoint, and generate text. It needs only PyTorch, Transformers, the Hugging Face Hub and ipywidgets, so it runs in a free Colab session. Under the hood each model loads in one call:

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(repo, revision="step-N", trust_remote_code=True)

Open beetle-explorer on GitHub