Mistral AI launches Mistral 7B

 Mistral AI, the six month old Paris based startup that made headlines with its unique Word Art logo and a record-setting $118 million seed round reportedly the largest seed in the history of Europe  today released its first large language AI model, Mistral 7B.Mistral AI can handle English tasks while also delivering natural coding capabilities at the same time making another option for multiple enterprise-centric use cases.

Mistral AI team is proud to release Mistral 7B, the most powerful language model for its size to date. Mistral 7B is easy to fine-tune on any task. As a demonstration, Mistral AI providing a model fine-tuned for chat, which outperforms "Llama 2 13B" chat.

Performance of Mistral 7B and different Llama models on a wide range of benchmarks. For all metrics, all models were re-evaluated with our evaluation pipeline for accurate comparison. Mistral 7B significantly outperforms Llama 2 13B on all metrics, and is on par with Llama 34B (since Llama 2 34B was not released, we report results on Llama 34B). It is also vastly superior in code and reasoning benchmarks.

Mistral 7B uses a sliding window attention  mechanism (Child et al., Beltagy et al.), in which each layer attends to the previous 4,096 hidden states. The main improvement, and reason for which this was initially investigated, is a linear compute cost of 0. In practice, changes made to FlashAttention and xFormers yield a 2x speed improvement for sequence length of 16k with a window of 4k. A huge thanks to Tri Dao and Daniel Haziza for helping include these changes on a tight schedule

Fine-tuning Mistral 7B for chat

To show the generalization capabilities of Mistral 7B, we fine-tuned it on instruction datasets publicly available on HuggingFace. No tricks, no proprietary data. The resulting model, Mistral 7B Instruct, outperforms all 7B models on MT-Bench, and is comparable to 13B chat models

The Mistral 7B model is available today for download by various means, including a 13.4-gigabyte torrent (with a few hundred seeders already). The company has also started a GitHub repository and Discord channel for collaboration and troubleshooting.

For More Posts:

                          Click Here