Google Research introduces sleep framework for LLMs to retain knowledge in continual learning

Google Research proposes a sleep framework for continual learning in large language models, enabling them to retain knowledge beyond the context window. The framework operates in alternating active and offline sleep phases, during which the model consolidates short-term context into stable long-term parameters using Mixture-of-Experts memory blocks. This approach allows LLMs to continuously acquire new facts, retain long-context knowledge up to 10M tokens, and improve complex reasoning without altering the core architecture.