Jamba - Hybrid SSM-Transformer
State space models (SSMs) and transformers want different things. Transformers buy quality via attention at quadratic cost. SSMs buy linear-time inference and constant memory via a recurrence but lag quality. AI21's Jamba (March 2024) and…
Learn this with Professor KatalockOpens in KatalockAi chat: your AI engineering mentor teaches this lesson step by step, checks your understanding and reviews your code.