ILM: Integer Language Model

Each benchmark is read with frozen code on its test data: the official test split, or for LoNLI, LAMA and SVAMP the part that its notes name. The runs use one desktop CPU with no GPU. The time is for the whole run, learning from all lessons and answering the test questions, as the best of three or more runs.

A time is the wall-clock time of the whole run with the release build: the choice among the tools, the controls, the learning from all lessons, and the read of the test data. Each time is the best of three or more runs in turn of the commands of the repository, in a fresh copy of it with the data fetched from the public sources.

ILM is at or near 100% on bAbI, SCAN and ProofWriter, in the upper range on CLUTRR, LAMA and LoNLI, and in the middle on SVAMP and E2E, where pre-trained transformers lead.

Where ILM stands

• bAbI, SCAN and ProofWriter: ILM reads every test split at or near 100%, from 600 to 1,000 lessons for each kind. We found no published learner that does this with the same lessons.
• CLUTRR: above GPT-4o with a graph and a symbolic solver (67.7), and at the level of the best row of the table, Faithful CoT with code-davinci-002 and votes (71.9). ILM learns from the step labels that the set gives.
• LAMA: above every model of BERT size, at the level of an 11-billion-parameter model with trained prompts.
• SVAMP: above the models that learn from scratch, below the pre-trained models and the large language models.
• E2E: at the level of the best systems of the E2E challenge, below fine-tuned GPT-2.

Each benchmark

The published numbers come from the papers named in each table, by arXiv number.

The board

The board measures all kinds of 25 public sets on their read parts, with the same program: 240 kinds in 112 tasks. A set gives only its files, its sizes, who wrote its text and its marks. ILM learns 150 kinds and 59 tasks in 17 sets, by the bar under How it works. 40 of the 59 tasks are learned from 200 lessons. The whole board takes 19 s.

ILM learns a kind of task from lessons. A lesson is a short text, a question and the exact answer. ILM counts which parts of the text go with which answer, and it keeps only what the counts show to be sure. The result is a set of tables, rules and small programs that a person can read.

Fifteen tools, one call

ILM has fifteen tools. Each tool has the same call: it learns from lessons, and then it answers a told text with a question. Each answer has a note that tells how the tool found it: the rule, the sentence or the program. Each tool is named for its way of reading, not for a task.

The choice among the tools

For each kind, every tool learns from the first two thirds of the lessons and reads the last third. The tool with the most right answers on that third is taken, and it learns again from all lessons. With equal results, the tool that comes first in the list is taken. No person chooses a tool for a kind, and no setting changes between kinds.

Beside the tools

Beside the tools, ILM has a fact reader for encyclopedia text, a question reader, a writer that turns given facts into a text, and one command (ilm ask) that routes each request and prints a trace of its steps.

Word classes from plain text

Some readers use a table of word classes: words that stand in the same places of a text fall into one class, for example first names, cities or months. ilm classes counts the table from plain text with no label. It groups the 40,000 most frequent words into 1,000 classes by the words that stand right before and right after them. The same text always gives the same table.

How a kind is measured

• The cases of a kind are cut into a learned part, a read part and a sealed part. All cases with the same told sentences stay in one part.
• A case is right only when the words of the answer are exactly the words of the true answer.
• Three controls stand beside the model: one answer for all cases, plain counting, and the question alone.
• A kind is learned at 95% or more of the read cases, and only when the model is 5 points or more above one answer for all and above the question alone. So a kind whose answers need no reading of the text is not learned.
• Read cases that are copies of lessons do not count. When a kind has copies, the share on its other read cases counts, and only with 50 or more of them.
• On the board, the bar is 90% for a set whose text people wrote (ATIS, CLUTRR, ParaRules, SNIPS and the word problems). A set can also mark a kind where the question alone gives so much that no model can be 5 points above it (ProofWriter, closed form, depth 5; ParaRules, depths 3 and 4). Such a kind passes at the bar alone.
• On the board, the read cases are read again with new words in place of the names, places and things.
• A control of changed cases asks whether the answer follows a change of the text. A task counts only when the learner follows this control.
• The sealed part is for one final run with frozen code. The final reads of the benchmarks read their test data in the same way.

Exact results

• Counts are whole numbers. The shares that decide a vote are exact fractions.
• Sums of doubles use the compensated sum or one fixed order of terms.
• Tables keep the order of first sight. Of equal candidates, the first one is taken.
• Work is cut into runs on all threads, and the runs are joined in their order. So the result is the same for every number of threads.

• ILM is not a general chat model. It learns one kind of task at a time from lessons of that kind, and each lesson needs the exact answer.
• On the board of 25 public sets (240 kinds in 112 tasks), it learns 59 tasks: every kind of the task at 95% right or more on held cases (90% where people wrote the text).
• On SVAMP and E2E, ILM is in the middle, and pre-trained transformers lead: on SVAMP, ILM has 38.3% and GPT-3 with chain of thought 68.9; on E2E, ILM has BLEU 65.37 and GPT-2 large with LoRA 70.4.
• On CLUTRR, ILM learns from the step labels that the set gives.
• Some comparisons do not read the same cases. The published LAMA numbers read all 34,039 facts, and ILM reads the 27,248 facts that are not lessons. The published SVAMP rows read all 1,000 problems, and ILM reads the 300 of the test file.
• During development, ILM used the training and validation data of each set. The board also reads a fixed part of the test split of some sets. Of that part, only the share of right answers was looked at, never a wrong case.
• The sealed part of LoNLI was read once. The LAMA probe was read four times, the CLUTRR test split three times and the E2E test set two times. Between two reads, the reader, the writer, the joins of the link tool and the table of word classes changed on development data only. The tables give the last read.
• The board does not count bAbI tasks 3 and 18 as learned: in the control of changed cases, a sure rule keeps the old answer when a supporting sentence is taken out.

The program is a Rust workspace of five crates. The code follows the rules of the crate premise: an idea is a type with its published source, every constant, type and function states its reason, and the code holds no comments and no unnamed numbers. The tests use made-up cases and need no downloaded data.

ILM reads public sets from the Hugging Face hub and GitHub at fixed revisions. The repository holds no data of the sets. Each set keeps its own license.

License: Apache License 2.0.

Run it with an AI coding agent

Give this prompt to an AI coding agent, such as Claude Code, Codex or Cursor. It follows the Quick start of the README: it builds ILM and runs bAbI on your computer.

Clone https://github.com/Datamine-Crypto/ILM and follow the Quick start in its README. Install what it needs, build ILM, then fetch, export and measure bAbI. Show me the right answers of each of the 20 tasks. For the other benchmarks, use docs/REPRODUCE.md.

ILM on GitHubILM on Hugging Face