Gradio Web UI for LLMs
Last updated
Last updated
A Gradio web UI for Large Language Models.
Its goal is to become the AUTOMATIC1111/stable-diffusion-webui of Text Generation.
(oobabooga/text-generation-webui)
3 interface modes: default (two columns), notebook, and chat.
Multiple model backends: Transformers, llama.cpp (through llama-cpp-python), ExLlamaV2, AutoGPTQ, AutoAWQ, GPTQ-for-LLaMa, CTransformers, QuIP#.
Dropdown menu for quickly switching between different models.
Large number of extensions (built-in and user-contributed), including Coqui TTS for realistic voice outputs, Whisper STT for voice inputs, translation, multimodal pipelines, vector databases, Stable Diffusion integration, and a lot more. See the wiki and the extensions directory for details.
Precise chat templates for instruction-following models, including Llama-2-chat, Alpaca, Vicuna, Mistral.
LoRA: train new LoRAs with your own data, load/unload LoRAs on the fly for generation.
Transformers library integration: load models in 4-bit or 8-bit precision through bitsandbytes, use llama.cpp with transformers samplers (llamacpp_HF
loader), CPU inference in 32-bit precision using PyTorch.
OpenAI-compatible API server with Chat and Completions endpoints -- see the examples.
Clone or download the repository.
Run the start_linux.sh
, start_windows.bat
, start_macos.sh
, or start_wsl.bat
script depending on your OS.
Select your GPU vendor when asked.
Once the installation ends, browse to http://localhost:7860/?__theme=dark
.
Have fun!
To restart the web UI in the future, just run the start_
script again. This script creates an installer_files
folder where it sets up the project's requirements. In case you need to reinstall the requirements, you can simply delete that folder and start the web UI again.
The script accepts command-line flags. Alternatively, you can edit the CMD_FLAGS.txt
file with a text editor and add your flags there.
To get updates in the future, run update_linux.sh
, update_windows.bat
, update_macos.sh
, or update_wsl.bat
.
https://github.com/oobabooga/text-generation-webui/wiki