> For the complete documentation index, see [llms.txt](https://metaverse-imagen.gitbook.io/ai-tools-research/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://metaverse-imagen.gitbook.io/ai-tools-research/about-ai-tools-research/frequently-asked-questions-faqs/faqs-on-llm-training-and-data-labelling/case-study-of-fine-tuning-an-llm/autotrain-llms-at-huggingface/what-are-gguf-format-model-files.md).

# What are GGUF Format Model Files?

GGUF, previously GGML, is a quantization method that allows users to use the CPU to run an LLM but also offload some of its layers to the GPU for a speed up. Although using the CPU is generally slower than using a GPU for inference, it is an incredible format for those running models on CPU or Apple devices.

**GGUF (GPT-Generated Unified Format) is a file format designed specifically for storing and running large language models (LLMs) for inference tasks.** It offers several advantages over previous formats like GGML, making it a promising choice for efficient LLM deployment.

**Key features:**

* **Single-file deployment:** Models are contained in a single file, simplifying distribution and loading.
* **Extensible:** New features can be added to GGML-based executors without breaking compatibility with existing models.
* **mmap compatibility:** Models can be efficiently loaded using memory-mapped files for fast access.
* **Optimized for inference:** Designed for efficient LLM inference, particularly on CPUs and GPUs.
* **Supports quantization:** Models can be quantized to reduce file size and computational requirements.
* **Metadata support:** Includes model metadata for better organization and understanding.
* **Improved tokenization:** Handles special tokens more effectively than GGML.

**Common uses:**

* Running large language models like GPT-4, Bloom, and Megatron-Turing NLG on various hardware platforms.
* Powering text generation, translation, question answering, and other language-related tasks.

**Compatibility:**

* **Supported executors:** llama.cpp, text-generation-webui, KoboldCpp, GPT4All, LM Studio, LoLLMS Web UI, Faraday.dev, llama-cpp-python, candle, and ctransformers.
* **Supported model frameworks:** PyTorch (via conversion).

**While we cannot provide images directly, here are examples of tools and libraries that work with GGUF files:**

* **lamma.cpp:** The primary GGUF executor, offering command-line and server options.
* **text-generation-webui:** A popular web interface for running GGUF models.
* **KoboldCpp:** A full-featured web interface with GPU acceleration.
* **GPT4All:** A free, open-source local GUI with GPU support.
* **LM Studio:** A user-friendly local GUI for Windows and macOS.
* **LoLLMs Web UI:** A web UI with various unique features and a full model library.
* **Faraday.dev:** An attractive, character-based chat GUI.
* **llama-cpp-python:** A Python library for using GGUF models with GPU acceleration and LangChain support.
* **candle:** A Rust ML framework focused on performance and ease of use.

<br>