Repository: mudler/LocalAI

Issue #3076: Intel Arc A770 LLM Fails
Created: 2024-07-29T21:58:45Z
I am trying to get text generation working on my Intel Arc A770 8GB and I am running into issues sycl not utilising the GPU. When I attempt to run use any prompt or any model I run into the following messages in the logs:
instance-1  | 9:55PM DBG GRPC(llava-v1.6-mistral-7b.Q5_K_M.gguf-127.0.0.1:34249): stderr Native API failed. Native API returns: -2 (PI_ERROR_DEVICE_NOT_AVAILABLE) -2 (PI_ERROR_DEVICE_NOT_AVAILABLE)
instance-1  | 9:55PM DBG GRPC(llava-v1.6-mistral-7b.Q5_K_M.gguf-127.0.0.1:34249): stderr Exception caught at file:/build/backend/cpp/llama-avx2/llama.cpp/ggml/src/ggml-sycl.cpp, line:3523, func:operator()
instance-1  | 9:55PM DBG GRPC(llava-v1.6-mistral-7b.Q5_K_M.gguf-127.0.0.1:34249): stderr SYCL error: CHECK_TRY_ERROR(dpct::gemm_batch( *main_stream, oneapi::mkl::transpose::trans, oneapi::mkl::transpose::nontrans, ne01, ne11, ne10, alpha, (const void **)(ptrs_src.get() + 0 * ne23), dpct::library_data_t::real_half, nb01 / nb00, (const void **)(ptrs_src.get() + 1 * ne23), dpct::library_data_t::real_half, nb11 / nb10, beta, (void **)(ptrs_dst.get() + 0 * ne23), cu_data_type, ne01, ne23, cu_compute_type)): Meet error in this line code!
instance-1  | 9:55PM DBG GRPC(llava-v1.6-mistral-7b.Q5_K_M.gguf-127.0.0.1:34249): stderr   in function ggml_sycl_mul_mat_batched_sycl at /build/backend/cpp/llama-avx2/llama.cpp/ggml/src/ggml-sycl.cpp:3523
instance-1  | 9:55PM DBG GRPC(llava-v1.6-mistral-7b.Q5_K_M.gguf-127.0.0.1:34249): stderr /build/backend/cpp/llama-avx2/llama.cpp/ggml/src/ggml-sycl/common.hpp:103: SYCL error

It looks like my hardware does not support half (f16) precision?
LocalAI version:

v2.19.3
Environment, CPU architecture, OS, and Version:

> uname -a
Linux kishor-docker 6.8.0-38-generic #38+TEST2072755v20240712b1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 12 x86_64 x86_64 x86_64 GNU/Linux

Note: I had to use a custom test kernel from here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2072755/comments/4 to fix my Intel Arc from timing out. It looks like the 6.8 kernel has regressions and I am now running into it.
Edit:
I tried the mainline 6.10.2 kernel and I am still running into the same error. I am still seeing GPU hangs and resets.
Describe the bug

To Reproduce

Attempt to use text generation via the Web UI with any LLM model.
Expected behavior

Text to be produced and llama.cpp not to crash.
Logs

Logs:
https://gist.github.com/kprinssu/6cc5e9798018e05098ec091d1b5f4611
Linux Kernel logs:
[26569.857186] i915 0000:01:00.0: [drm] GPU HANG: ecode 12:10:85def5fa, in grpcpp_sync_ser [352969]
[26569.857197] i915 0000:01:00.0: [drm] grpcpp_sync_ser[352969] context reset due to GPU hang
[26572.958874] Fence expiration time out i915-0000:01:00.0:grpcpp_sync_ser[352969]:54!
[26589.959895] Fence expiration time out i915-0000:01:00.0:grpcpp_sync_ser[352969]:56!

Additional context
Docker Compose:
services:
  instance:
    image: quay.io/go-skynet/local-ai:v2.19.3-aio-gpu-intel-f32
    privileged: true
    ports:
      - 8085:8080
    volumes:
      - ./models:/build/models
      - ./images:/tmp/generated/images/
    devices:
      - /dev/dri:/dev/dri
    environment:
      - DEBUG=true
      - NEOReadDebugKeys=1
      - OverrideGpuAddressSpace=48
      - ZES_ENABLE_SYSMAN=1
    group_add:
      - 993
      - 996
      - 44


Note: I need to to the extra env vars above as sycl-ls did not list my GPU due to the 6.8 kernel (intel/compute-runtime#710 and intel/compute-runtime#710 (comment)).
Example Model Config:
name: gpt4-gpu
mmap: false
context_size: 4096

f16: true
gpu_layers: 4

parameters:
  model: llava-v1.6-mistral-7b.Q5_K_M.gguf

stopwords:
- "
"
- "<dummy32000>"
- "</tool_call>"
- "
"
- "
"

Duplicate issue #3382: intel igpu  not working 
Created: 2024-08-26T12:48:49Z
ana 6.1.103-Unraid #1 SMP PREEMPT_DYNAMIC Fri Aug  9 00:11:30 PDT 2024 x86_64 11th Gen Intel(R) Core(TM) i5-11400 @ 2.60GHz GenuineIntel GNU/Linux
with 64 GigaBamboozle of wham
gpu = RocketLake-S GT1 [UHD Graphics 730]
Describe the bug
the gpu does work as shown by llama-cpp -avx2  showing up in intel_gpu_top but  there is almost no usage  and seems to process forever
To Reproduce
go to the unraid "appstore"  choose the latest  intel gpu-aio-f16 image and  passthrough /dev/dri as a device
Expected behavior
the same experience as the cpu only  image only  faster
Logs
logs are attached
logs.txt
Additional context
i 'm using this  to flesh out dnd characters and 30 secs per prompt makes it take ages
thx  in advance
Max

Closing comment (2024-08-27T07:30:23Z): looking  a bit into it. it seems to be related to #3076