# MMLU (5-Shot CoT)

MMLU (5-Shot CoT) is a variant of the MMLU benchmark where the LLM is given only **5 examples** of each task before being evaluated on its performance. This is known as the **few-shot** setting, and it is a more challenging test of the LLM's ability to learn and generalize.

The **CoT** stands for **Chain of Thought**. In this setting, the LLM is given a step-by-step explanation of how to solve a task, rather than just the input and output. This helps the LLM to better understand the task and to develop a more generalizable solution.

MMLU (5-Shot CoT) is a challenging benchmark, but it is also a very important one. It helps researchers to develop LLMs that are more capable and versatile, and it also helps to ensure that LLMs are used responsibly and ethically.

Here are some examples of tasks that might be evaluated in MMLU (5-Shot CoT):

* Answer a question about a piece of text.
* Summarize a piece of text.
* Generate code to solve a problem.
* Translate a piece of text from one language to another.
* Write a poem or story.

The LLM is evaluated on its accuracy and fluency in completing these tasks.

MMLU (5-Shot CoT) is a relatively new benchmark, but it has already been used to evaluate some of the most state-of-the-art LLMs, such as Google's Flan-PaLM. Flan-PaLM achieved a score of 75.2% on MMLU (5-Shot CoT), which is the current state-of-the-art.

As LLMs continue to improve, we can expect to see even higher scores on MMLU (5-Shot CoT). This will help to ensure that LLMs are used to solve a wider range of problems and to benefit society in new and innovative ways.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://metaverse-imagen.gitbook.io/ai-tools-research/about-ai-tools-research/llm-performance-benchmarks/llm-benchmarks-and-tasks/mmlu-massive-multitask-language-understanding/mmlu-5-shot-cot.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.