You read online about KoboldCPP and you want to install them, but you have an AMD GPU. It will be definitely impossible to install theme with HIP/ROCm support, right? Well it’s wrong! In this guide we will see how to install KoboldCPP for your AMD GPU and accelerate models with the power of HIP/ROCm software.
Prerequisite: Installing ROCm on your computer
To actually use KoboldCPP you need to install ROCm on your computer. There are numerous way to do it. I suggest you to check this post where I explain how to install ROCm. It’s not difficult, give it a try! I also point out that this guide is expressly aimed to Linux not Windows.
Installing KoboldCPP for AMD HIP/ROCm
Let’s start right away with the installation. If you followed my previous tutorial on installing ROCm you will have to open a terminal and enter the container with the following command:
distrobox enter almalinux-rocmAnd activate gcc-10:
scl enable gcc-toolset-10 bashIf you have installed ROCm in other way keep going with the guide. Let’s clone the repository. With the following command you will create the directory that store the program. You will also install customtkinter package that is required to run the program:
git clone https://github.com/LostRuins/koboldcpp.git \
&& cd koboldcpp \
&& sudo dnf install -y python3.12-tkinter \
&& pip3.12 install --no-input customtkinterAfter that you have to compile it. The command is different depending on the GPU you use.
If you have an RDNA 2 GPU (Rx 6000):
make LLAMA_HIPBLAS=1 GPU_TARGETS=gfx1030 -j$(nproc)If you have an RDNA 3 GPU (Rx 7000):
make LLAMA_HIPBLAS=1 GPU_TARGETS=gfx1100 -j$(nproc)If you have an Rx 9070:
make LLAMA_HIPBLAS=1 GGML_HIP_FORCE_ROCWMMA_FATTN_GFX12=1 GPU_TARGETS=gfx1201 -j$(nproc)If you have an Rx 9060:
make LLAMA_HIPBLAS=1 GGML_HIP_FORCE_ROCWMMA_FATTN_GFX12=1 GPU_TARGETS=gfx1200 -j$(nproc)To update KoboldCPP run the following commands in a terminal in the directory llama.cpp. Remember to enter the container before doing so if you have it.
git pull \
&& make cleanAnd after that you have to repeat compilation procedure.
And with that we have finished the installation of KoboldCPP for your AMD GPU!
What to do next
Using the program is beyond the scope of this tutorial, if you need more information on how to use KoboldCPP visit the official site. Look at those llm front-end, such as Open WebUI, GPT4All and GPT4Free. Be careful, however, that you must install all the python packages needed by frontends in a virtual environment where there is PyTorch installed with HIP/ROCm support. You can find how to install correctly PyTorch for your AMD GPU on this tutorial, don’t worry, it’s super easy.
But I want to use LlamaCPP
KoboldCPP is perfect for some user cases but maybe you need a more generic software. You can use LlamaCPP in that case. KoboldCPP share a lot of code with LlamaCPP, in fact it will work on your AMD GPU really easy. You don’t believe me? Check out this tutorial where I explain how to install it for AMD HIP/ROCm:
- Install ComfyUI for AMD HIP/ROCm on Windows
Learn how to install ComfyUI for AMD HIP ROCm on Windows and generate images and videos. - Install PyTorch for AMD HIP ROCm on WindowsLearn how to install PyTorch for AMD HIP ROCm on Windows and accelerate AI programs.
- How to install ROCm python packages on WindowsLearn how to install AMD HIP/ROCm on Windows trough python packages
- Install axolotl for AMD HIP/ROCm on LinuxLearn how to install Llama-Factory on AMD GPUs using HIP/ROCm. Step-by-step guide for setting up LLM fine-tuning with AMD hardware and open-source tools.
- Install Llama-Factory for AMD HIP/ROCm on LinuxLearn how to install Llama-Factory on AMD GPUs using HIP/ROCm. Step-by-step guide for setting up LLM fine-tuning with AMD hardware and open-source tools.
