Repository: ollama
Issue: Support Radeon RX 5700 XT (gfx1010)
Labels: documentation,amd

Hi! Congrats for the great project!

We were trying to test ollama with AMD GPU support and we struggled a bit because the install guides are not clear that CUDA libraries are required for ollama (or llama.cpp) to work properly even with team red GPUs.

The error when running ollama run llama2 was (leaving here for reference):
...
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      ROCm0 KV buffer size =  1024.00 MiB
llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
llama_new_context_with_model:  ROCm_Host input buffer size   =    13.01 MiB
llama_new_context_with_model:      ROCm0 compute buffer size =   164.00 MiB
llama_new_context_with_model:  ROCm_Host compute buffer size =     8.00 MiB
llama_new_context_with_model: graph splits (measure): 3
**CUDA error: shared object initialization failed
  current device: 0, in function ggml_cuda_op_flatten at /home/devel/ollama/llm/llama.cpp/ggml-cuda.cu:9208
  hipGetLastError()
loading library /tmp/ollama3700311510/rocm_v6/libext_server.so
GGML_ASSERT: /home/devel/ollama/llm/llama.cpp/ggml-cuda.cu:241: !"CUDA error"**
[New LWP 4411]
[New LWP 4412]
[New LWP 4413]
[New LWP 4414]
[New LWP 4415]
...

after we installed the cuda libraries as per the instructions [HERE](https://developer.nvidia.com/cuda-downloads) the problem went away.

we also faced problems with ROCm 6.0.2 support for different gpu models (in our case it is a RX 5700 XT, arch gfx1010), the current binary packages doesn't contain TensileLibrary.dat (that somehow "maps" the kernels objects to use with different GPUs).

We had this error:

time=2024-02-09T17:05:38.481Z level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama3752973675/rocm_v6/libext_server.so"
time=2024-02-09T17:05:38.481Z level=INFO source=dyn_ext_server.go:145 msg="Initializing llama server"
**rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: Illegal seek for GPU arch : gfx1010**
free(): invalid pointer
SIGABRT: abort
PC=0x7f4bb13739fc m=3 sigcode=18446744073709551610

so I downloaded full rocm source and tried to build again, just to get the right command for compiling TensileLibrary.dat :

this was the command that cmake uses: 
'/home/devel/rocBLAS/build/virtualenv/lib/python3.10/site-packages/Tensile/bin/TensileCreateLibrary' '--merge-files' '--separate-architectures' '--lazy-library-loading' '--no-short-file-names' '--library-print-debug' '--code-object-version=default' '--cxx-compiler=hipcc' '--jobs=14' '--library-format=msgpack' '--architecture=gfx1012' '/home/devel/rocBLAS/library/src/blas3/Tensile/Logic/asm_full' '/home/devel/rocBLAS/build/Tensile' 'HIP'

this is the command I used to generate a new TensileLibrary.dat:
'/home/devel/rocBLAS/build/virtualenv/lib/python3.10/site-packages/Tensile/bin/TensileCreateLibrary' '--merge-files' '--no-short-file-names' '--library-print-debug' '--code-object-version=default' '--cxx-compiler=hipcc' '--jobs=14' '--library-format=msgpack' '/home/devel/rocBLAS/library/src/blas3/Tensile/Logic/asm_full' '/home/devel/rocBLAS/build/Tensile' 'HIP'

(i removed '--separate-architectures' '--lazy-library-loading'  as per instructions [in this bug](https://github.com/ROCm/Tensile/issues/1757) )

Hope this helps to others! Thanks again!