# What is LoRA and How does LoRA work ### What is LoRA and How does LoRA work? **LoRA** stands for **Low-Rank Adaptation** of Large Language Models (LLMs). It is a novel technique proposed by Microsoft Research with a focus on addressing the challenge of Optimizing and Fine-Tuning Large Language Models. These models, such as GPT-3, with billions of parameters and GPT-4 with over a trillion parameters, are prohibitively expensive to adapt to specific tasks or domains. LoRA models are small models that apply tiny changes to stable checkpoint models. They are usually 10 to 100 times smaller than checkpoint models. That makes them very attractive to people having an extensive collection of models. LoRA’s effective adaptation technique maintains model quality while significantly reducing the number of trainable parameters for downstream tasks with no increased inference time. LoRA can be used for general-purpose fine-tuning, which allows it to be customized for different domains or datasets. Even though LoRA was initially proposed for Large-Language Models and demonstrated on Transformer blocks, the technique can also be applied elsewhere. LoRA applies small changes to the most critical part of models: The cross-attention layers. It is the part of the model where the image and the prompt meet (cross-attention layers that relate the image representations with the prompts that describe them). The researchers found that by focusing on the ‘Transformer attention blocks’ of large-language models, fine-tuning quality with LoRA was on par with full model fine-tuning while being much faster and requiring less compute. (Researchers found it sufficient to fine-tune this part of the model to achieve good training). --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://metaverse-imagen.gitbook.io/ai-tools-research/about-ai-tools-research/frequently-asked-questions-faqs/what-is-lora-and-how-does-lora-work.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.