Datamine Network Inc. is an AI and Crypto Research Company

• bAbI, 1,000 examples: mean error, lower is better
  Tasks failed (of 20): ILM 1, Universal Transformer 3, Transformer 5. 601 test questions with the story of a lesson are not read.
  ILM: 2.48%

  vs Transformer, best of 10 runs

  Transformer, best of 10 runs: 21.8%

  vs Universal Transformer, best seed for each task

  Universal Transformer, best seed for each task: 4.55%
• SCAN, length
  ILM: 100%

  Transformer: 0%
• SCAN, add jump
  ILM: 100%

  Transformer: 1%
• PCFG SET, productivity
  ILM: 100%

  Transformer: 50%
• PCFG SET, systematicity
  ILM: 100%

  Transformer: 72%
• CLUTRR graph, 10 relations
  During development the board read 527 of the 1,045 test graphs (shares only, never a wrong case).
  ILM: 100%

  Transformer: 70%
• Sentence transformations, generalization sets
  ILM: 100%

  Transformer: near 0%
• ToM-QA
  Not a read of unseen cases: the board read this test split during development (shares only, never a wrong case).
  ILM: 100%

  vs the best model of each form, from scratch (EntNet, RelNet, Multiple Observer)

  the best model of each form, from scratch (EntNet, RelNet, Multiple Observer): 88.63% to 100%
• ProofWriter, depth 5
  ILM: 99.52%

  vs ProofWriter (T5-11B), pre-trained, with proofs

  ProofWriter (T5-11B), pre-trained, with proofs: 99.6% and 99.7%
• LoNLI, sealed part
  Ahead of the models with no LoNLI lesson; not of RoBERTa-large tuned on 2% of LoNLI.
  ILM: 98.18%

  vs BERT-base, RoBERTa-large and DeBERTa-large, pre-trained, tuned on other NLI sets

  BERT-base, RoBERTa-large and DeBERTa-large, pre-trained, tuned on other NLI sets: 59.4% to 71.1%

  vs RoBERTa-large, pre-trained, tuned on 2% of LoNLI

  RoBERTa-large, pre-trained, tuned on 2% of LoNLI: near 100%

The papers give some of these as fractions; they are shown here as percentages. On SCAN, PCFG SET and the sentence transformations, ILM learns from 1,000 lessons, and the published models from 14,670 to 100,000.