#mixtureofexperts

Building an AI That Never Stops Learning on 8GB of VRAM

How mini-AGI trains a byte-level, mixture-of-experts model from scratch on a single 8GB GPU, and solves catastrophic forgetting with one learning-rate tweak.