Silia

A small transformer variant that merges the attention layer with the SwiGLU feed-forward layer to cut parameters at very small model sizes, without losing much performance.

Visit website github.com

Silia (Silu in Attention) is a neural network architecture that merges the attention layer with the SwiGLU feed-forward layer from the Transformer, aimed at models of about 100 million parameters or fewer, where a standard Transformer is argued to use more parameters than it needs. The paper-style write-up frames this as an under-explored regime: models called small, such as Qwen, Gemma and GPT-OSS, still run from 1 to 20 billion parameters.

Trained on the Fineweb-edu dataset, Silia is reported to reach competitive validation loss and generation quality against other models of equal or greater size, evaluated on the HellaSwag, PIQA and LAMBADA benchmarks.

It is written in Python and released under the MIT license.

More in Models

See all 405

mini-AGI
github.com
A continual learning language model that trains from scratch on a single 8 GB VRAM GPU, assembling its own architecture and adapting its size to the hardware and data available rather than shipping as a fixed pretrained model.
- Models
Applied Inference Lab: Compliance Control Embedding Stack
github.com
An open-source contrastive-embedding pipeline that maps technical CVE vulnerability descriptions to formal cybersecurity compliance controls in NIST SP 800-53 and CMMC, published with its model generation, fine-tuning and evaluation code.
- Models
- Security
peekaboolean
github.com
A small vision-language model, fine-tuned with LoRA on a 500-million-parameter backbone, that answers typed multiple-choice, ordinal-score and yes/no questions about an image with calibrated probabilities, in about 400 milliseconds on a laptop.
- Models
- Inference

More in Models

mini-AGI

Applied Inference Lab: Compliance Control Embedding Stack

peekaboolean

Cookies