GGUF or not GGUF, that is the question. If you prefer ExLlamaV2 over LlamaCPP you are in the right place! In this guide you will learn how to install ExLlamaV2 for AMD HIP/ROCm and boost your llm.
Prerequisite: Installation of ROCm and PyTorch
Before installing ExLlamaV2 you must install ROCm and PyTorch. I covered these topics on two articles:
Look at these articles to install them.
All the guides, including this one, assume the use of Linux so they will not work on Windows.
Installation of ExLlamaV2 enhanced by HIP/ROCm
The installation of ExLlamaV2 for AMD HIP/ROCm is quite simple. Let’s start by opening a terminal, cloning the repository and enter the directory just created:
git clone https://github.com/turboderp/exllamav2 \
&& cd exllamav2If you followed my tutorials you will need to enter the container and activate the python virtual environment where you had previously installed PyTorch:
# Run one command at a time
# Enter the container
distrobox enter almalinux-rocm
# We need to chance the default compiler's version from gcc-8 to gcc-9
scl enable gcc-toolset-10 bash
# Activate the Conda environment
conda activate py312After that you must install the requirements with the command:
pip install -r requirements.txtThen, run the following command to install the package:
pip install .For more information about ExLlamaV2 visit their main site.
I also need LlamaCPP
If you also want LlamaCPP to work on your AMD GPU powered by HIP/ROCm software here is a guide on how to install it. Spoiler: it’s really simple:
Conclusions
As you can see, installing ExLlamaV2 for AMD HIP/ROCm is pretty simple. If you liked this guide leave a comment and a like.
- Install ComfyUI for AMD HIP/ROCm on Windows
Learn how to install ComfyUI for AMD HIP ROCm on Windows and generate images and videos. - Install PyTorch for AMD HIP ROCm on WindowsLearn how to install PyTorch for AMD HIP ROCm on Windows and accelerate AI programs.
- How to install ROCm python packages on WindowsLearn how to install AMD HIP/ROCm on Windows trough python packages
- Install axolotl for AMD HIP/ROCm on LinuxLearn how to install Llama-Factory on AMD GPUs using HIP/ROCm. Step-by-step guide for setting up LLM fine-tuning with AMD hardware and open-source tools.
- Install Llama-Factory for AMD HIP/ROCm on LinuxLearn how to install Llama-Factory on AMD GPUs using HIP/ROCm. Step-by-step guide for setting up LLM fine-tuning with AMD hardware and open-source tools.
