Build a Small LLM From Scratch: A Tested GPT in PyTorch
Build a compact GPT-style transformer in PyTorch, train it on Tiny Shakespeare, test the critical pieces, and see exactly how an 813K-parameter language model learns.
Build a compact GPT-style transformer in PyTorch, train it on Tiny Shakespeare, test the critical pieces, and see exactly how an 813K-parameter language model learns.
Tencent’s 770B Hy4 Preview combines sparse attention, four residual streams, native speculative decoding, and aggressive pricing—but remains firmly data-center scale.
