tinystories

Llama2 inference in one TypeScript file

Codebase for Linguistic Collapse: Neural Collapse in (Large) Language Models [NeurIPS 2024] [arXiv:2405.17767]

🎈 A series of lightweight GPT models featuring TinyGPT Base (~51M params) and TinyGPT2 (~95M params). Fast, creative text generation trained on whimsical stories.

Dataset Generation Code for SimpleStories

Implementation of a small GPT-style transformer from scratch in PyTorch. Learn how Large Language Models work by building, training, generating text, and visualizing attention.

Train and run a small Llama 2 model from scratch on the TinyStories dataset.

Train GPT style model on tinystories dataset

Code implementation for our paper "BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training" as part of the 2024 BabyLM Challenge

Small transformer trained from scratch

A ~30M GPT small language model trained from scratch on TinyStories — GPT-2-compatible so the ember inference server runs it.

This project fine-tunes GPT-2, a popular pre-trained transformer model, to generate short stories using the TinyStories dataset. The goal is to teach GPT-2 to produce creative and coherent stories based on prompts.

LLM Architecure, Open Source

A port of karpathy/llama2.c to MS-DOS

Specialized agentic coding LLM — 3B, pre-trained from scratch, runs locally on Apple Silicon. No API key, no data leaving your machine, zero cost per call.

Training a tiny GPT-like Transformer language model

A 110M-parameter Llama-style transformer trained from scratch on the TinyStories dataset, optimized for high-throughput training on 4GB VRAM consumer GPUs. The project features a custom asynchronous CUDA-stream prefetcher and KV-cache inference, achieving 10k+ TPS on an RTX 3050.

StoryByte — a ~1M-parameter GPT trained from scratch on TinyStories. The model behind the Let's Data Science 'Build a Tiny LLM — From Tokens to Text' course; its whole forward pass runs in ~40 lines of NumPy in a browser.

How small can English get? 8 tiny language models (fp16 vs ternary) trained from scratch on TinyStories, judged by a frozen LLM judge, and running live in your browser.

A 27.8M-parameter GPT trained from scratch on TinyStories — transformer, tokenizer pipeline and training loop written by hand in PyTorch. Trained weights included.

A PyTorch implementation of a Bigram Language Model using Transformer architecture for character-level text generation.