# Falcon 180B
Falcon 180B: The Largest Open Language Model Surpasses Llama 2 and GPT 3.5
· *6 September 2023*
The Institute of Technological Innovations from the UAE has unveiled Falcon 180B, the largest open language model, displacing [Llama 2](https://neurohive.io/en/state-of-the-art/llama-2-and-llama-2-chat/) from the top spot in the rankings of pre-trained open-access language models by HuggingFace.
The Falcon 180B model
* Trained on 3.5 trillion tokens using the [RefinedWeb](https://arxiv.org/pdf/2306.01116.pdf) dataset.
* 180 billion parameters, which is 2.6 times more than the previous leader, LlaMA 70B,
* Requires 8 Nvidia A100 GPUs and 400GB of space for inference.
You can [test the model on HuggingFace](https://huggingface.co/spaces/tiiuae/falcon-180b-demo), and the [model’s code](https://huggingface.co/tiiuae/falcon-180B) is also available there.
### Model Architecture
Falcon 180B, a fine-tuned version of Falcon 40B, utilizes a multi-query attention mechanism for enhanced scalability.
The conventional multi-head attention scheme features one query, key, and value for each head, whereas the multi-query approach uses a single key and value for all “heads.”
\
The Falcon 180B model was trained on **4096 GPUs, which took approximately 7,000,000 GPU hours on Amazon SageMaker**.
Compared to LlaMA 2, training Falcon 180B required four times more computational power.
The dataset for Falcon 180B primarily comprises web data from the **RefinedWeb dataset (approximately 85%)**. Additionally, selected data, including dialogues, technical articles, and code, were used, making it a versatile model for NLP tasks (around 3%).
Falcon 180B outperforms Llama 2 70B and GPT-3.5 from OpenAI on the MMLU benchmark, although it falls behind GPT-4. It also competes successfully with Google’s proprietary PaLM 2-Large on benchmarks such as HellaSwag, LAMBADA, WebQuestions, Winogrande, PIQA, ARC, BoolQ, CB, COPA, RTE, WiC, WSC, and ReCoRD:
**Comparison of the PALM family of models and Falcon 180B**
Although Falcon 180B is available on the Hugging Face Hub, its commercial usage is highly restricted. It is advisable to review the license and seek legal counsel for commercial purposes.
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://metaverse-imagen.gitbook.io/ai-tools-research/ai-technology/generative-ai-architectures-and-models/generative-ai-and-llms-for-text/falcon-180b.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.