imperialwool commited on
Commit
a5ef5d4
1 Parent(s): e543d33

Update quart_app.py

Browse files
Files changed (1) hide show
  1. quart_app.py +3 -3
quart_app.py CHANGED
@@ -63,10 +63,10 @@ This is showcase how to make own server with Llama2 model.<br>
63
  I'm using here 7b model just for example. Also here's only CPU power.<br>
64
  But you can use GPU power as well!<br><br>
65
  <h1>How to GPU?</h1>
66
- Change <code>`CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS`</code> in Dockerfile on <code>`CMAKE_ARGS="-DLLAMA_CUBLAS=on"`</code>. Also you can try <code>`DLLAMA_CLBLAST`</code> or <code>`DLLAMA_METAL`</code>.<br>
67
- Powered by <a href="https://github.com/abetlen/llama-cpp-python">llama-cpp-python</a>, <a href="https://quart.palletsprojects.com/">Quart</a> and <a href="https://www.uvicorn.org/">Uvicorn</a>.<br><br>
68
  <h1>How to test it on own machine?</h1>
69
  You can install Docker, build image and run it. I made <code>`run-docker.sh`</code> for ya. To stop container run <code>`docker ps`</code>, find name of container and run <code>`docker stop _dockerContainerName_`</code><br>
70
  Or you can once follow steps in Dockerfile and try it on your machine, not in Docker.<br>
71
  <br>''' + f"Memory used: {psutil.virtual_memory()[2]}<br>" + '''
72
- <script>document.write("<b>URL of space:</b> "+window.location.href);</script>'''
 
 
63
  I'm using here 7b model just for example. Also here's only CPU power.<br>
64
  But you can use GPU power as well!<br><br>
65
  <h1>How to GPU?</h1>
66
+ Change <code>`CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS`</code> in Dockerfile on <code>`CMAKE_ARGS="-DLLAMA_CUBLAS=on"`</code>. Also you can try <code>`DLLAMA_CLBLAST`</code> or <code>`DLLAMA_METAL`</code>.<br><br>
 
67
  <h1>How to test it on own machine?</h1>
68
  You can install Docker, build image and run it. I made <code>`run-docker.sh`</code> for ya. To stop container run <code>`docker ps`</code>, find name of container and run <code>`docker stop _dockerContainerName_`</code><br>
69
  Or you can once follow steps in Dockerfile and try it on your machine, not in Docker.<br>
70
  <br>''' + f"Memory used: {psutil.virtual_memory()[2]}<br>" + '''
71
+ <script>document.write("<b>URL of space:</b> "+window.location.href);</script>''' + '''
72
+ Powered by <a href="https://github.com/abetlen/llama-cpp-python">llama-cpp-python</a>, <a href="https://quart.palletsprojects.com/">Quart</a> and <a href="https://www.uvicorn.org/">Uvicorn</a>.<br><br>'''