BLLMC: Bangla LLM Collection
A modular PyTorch library for training and deploying Bangla language models from scratch, built around modern architecture components such as grouped-query attention, sliding-window attention, rotary embeddings and sparse mixture-of-experts.
BLLMC provides decoupled, composable building blocks for Bangla LLMs: attention mechanisms, positional embeddings, normalization and feedforward blocks that can be plugged into any model. Core layers include grouped query attention with shared KV heads, sliding window attention with KV caching, rotary position embeddings with NTK-aware scaling, learnable attention sinks, and sparse mixture-of-experts routing with a custom SwiGLU activation.
A model factory and registry lets users register and instantiate architectures such as GPT-2, Mistral, Llama 2, Llama 3 and GPT-OSS style models through a decorator pattern, with pluggable SentencePiece and tiktoken tokenizer backends that auto-download from the Hugging Face Hub. Generation supports greedy, temperature-based and top-k sampling strategies.
The project includes a Bangla-optimized data pipeline with a corpus downloader and automated train, validation and test splitting, and a trainer supporting mixed precision, cosine warmup scheduling, gradient clipping, gradient accumulation and checkpointing.