EP080: Jamba Hybrid Solves Transformer Memory Limits

Описание: Jamba (https://arxiv.org/abs/2403.19887) is a new large language model developed by AI21 Labs that introduces a novel hybrid architecture. The model interleaves traditional Transformer layers with Mamba (a state-space model) layers, and integrates a Mixture-of-Experts (MoE) module to increase capacity without proportionally increasing compute requirements.


This hybrid approach addresses the fundamental limitations of pure Transformer models, which suffer from high memory and compute requirements for long contexts due to the growing key-value (KV) cache. It also improves upon pure Mamba models, which can sometimes struggle to match the in-context learning capabilities of Transformers.


Key highlights of the Jamba model include:

• Massive Context Window: Jamba supports an impressive context length of up to 256K tokens. By replacing some attention layers with Mamba layers, it requires an 8x smaller KV cache than a vanilla Transformer (needing only 4GB for a 256K context, compared to 32GB for Mixtral).
• High Efficiency: The model features 52B total parameters but only 12B active parameters per token due to its MoE routing. This allows it to easily fit on a single 80GB GPU.
• Superior Throughput: Jamba boasts up to 3x the throughput (tokens per second) of comparable models like Mixtral-8x7B, especially when processing long contexts.
• State-of-the-Art Performance: Across standard academic benchmarks and long-context evaluations, Jamba performs comparably to leading models of similar or larger sizes, such as Mixtral-8x7B and Llama-2 70B.

To encourage further community research and exploration into hybrid Attention-Mamba architectures, AI21 Labs has made the base model publicly available under a permissive Apache 2.0 license.