Magic claims to have built a 100M token context model that’s 1000x cheaper than Llama at the same context length. Although there are still some question marks regarding evaluation, the increasing context trend is clear.

Posted on 28 Sep 2024