Test-Model-Thing (TMT)
A small proof-of-concept language model, explicitly not built as an LLM, that predicts in a latent space, keeps internal recurrent state, and generates output byte by byte instead of token by token.
Test-Model-Thing combines several less common ideas in one small model: prediction in a latent space rather than directly over tokens, an internal state carried by recurrent trace units, byte-level input and output instead of a token vocabulary, continuous data streaming, and test-time training. It is built with MLX, so it runs on Apple Silicon devices, and community members have also run it on Linux and, through unofficial PyTorch ports, on Windows.
The author, a solo high-school developer, trained only a 4.5-million-parameter version of the model for about 12 hours on a small dataset drawn from a Wikipedia dump, and reports that it can close quotes and brackets correctly but tends to misspell characters, a consequence of generating byte by byte. The model's training and benchmark code is released for others to fork, extend, or train at larger scale.