Repository: Mozilla-Ocho/llamafile

Issue #116: Server aborts when passing invalid JSON request
Created: 2023-12-17T10:38:00Z
Hi,
I tried to adjust the example API call used on this issue and I made a mistake and added a : after the opening { resulting in invalid JSON.
curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{:
  "model": "gpt-3.5-turbo",
  "messages": [
    {
      "role": "system",
      "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
    },
    {
      "role": "user",
      "content": "Compose a poem that explains the concept of recursion in programming. A maximum of 5 lines"
    }
  ]
}'
When I run that it results in this error:
error: Uncaught SIGABRT (SI_0) on marks-mbp-2.lan pid 97185 tid 262149
 ./llamafile-server-0.4
 Darwin Cosmopolitan 3.1.2 MODE=aarch64; Darwin Kernel Version 23.1.0: Mon Oct  9 21:27:24 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T6000 marks-mbp-2.lan 23.1.0
 cosmoaddr2line /Users/markhneedham/projects/learndatawithmark/llamafile-playground/llamafile-server-0.4 18a41911c b41180018a329540 464b000100282c64 100001a4f88 10000000478 10000063d74 1000006550c 10000013fd4 10000014f74 1000001ec58 10000045888 1000005e914 1000005f408
 0000000000000000 x0 47fdf043ec893f95 x8  0000000000000148 x16 000010008023f168 x24
 0000000000000000 x1 47fdf042837f4f95 x9  00000001e95ebda0 x17 000010008023f098 x25
 0000000000000008 x2 0000000000000020 x10 0000000000000000 x18 0000000000000001 x26
 00000100002ec000 x3 00000000000f4240 x11 0000000000000006 x19 0000000000000000 x27
 0000000000000000 x4 0000000000001050 x12 000000016ff67000 x20 00001000800ef500 x28
 00000100002e9be8 x5 00000000000160c6 x13 0000000000004203 x21 000010008023e1e0 x29
 00000100002ec000 x6 0000000000000000 x14 000000016ff670e0 x22 000000018a450cc0 x30
 0000000000000008 x7 000000009e054ffb x15 000010008023e6b8 x23 000010008023e1c0 x31
 000010008023e1c0 sp 18a41911c pc NULL-1975525588
 000010008023e1e0 fp b41180018a329540 lr NULL-1976507568
 000010008023e200 fp 464b000100282c64 lr ggml_table_silu_f16+55956
 000010008023e210 fp 100001a4f88 lr raise+48
 000010008023e220 fp 10000000478 lr abort+48
 000010008023e240 fp 10000063d74 lr bool nlohmann::json_abi_v3_11_2::detail::parser<nlohmann::json_abi_v3_11_2::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::json_abi_v3_11_2::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >>, nlohmann::json_abi_v3_11_2::detail::iterator_input_adapter<std::__1::__wrap_iter<char const*> >>::sax_parse_internal<nlohmann::json_abi_v3_11_2::detail::json_sax_dom_parser<nlohmann::json_abi_v3_11_2::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::json_abi_v3_11_2::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >>> >(std::__1::map*)+1064
 000010008023e260 fp 1000006550c lr nlohmann::json_abi_v3_11_2::detail::parser<nlohmann::json_abi_v3_11_2::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::json_abi_v3_11_2::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >>, nlohmann::json_abi_v3_11_2::detail::iterator_input_adapter<std::__1::__wrap_iter<char const*> >>::parse(bool, nlohmann::json_abi_v3_11_2::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::json_abi_v3_11_2::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >>&)+656
 000010008023e650 fp 10000013fd4 lr _ZZ4mainENKUlRKN7httplib7RequestERNS_8ResponseEE2_clES2_S4_+304
 000010008023e840 fp 10000014f74 lr _ZNSt3__110__function16__policy_invokerIFvRKN7httplib7RequestERNS2_8ResponseEEE11__call_implINS0_20__default_alloc_funcIZ4mainEUlS5_S7_E2_S8_EEEEvPKNS0_16__policy_storageES5_S7_+16
 000010008023ee50 fp 1000001ec58 lr std::__1::function<void (httplib::Request&, httplib::Response&)>::operator()(httplib::Response, void (httplib::Request&, httplib::Response&)) const+20
 000010008023ee60 fp 10000045888 lr _ZN7httplib6Server16dispatch_requestERNS_7RequestERNS_8ResponseERKNSt3__16vectorINS5_4pairINS5_11basic_regexIcNS5_12regex_traitsIcEEEENS5_8functionIFvRKS1_S4_EEEEENS5_9allocatorISH_EEEE+160
 000010008023ee70 fp 1000005e914 lr httplib::Server::routing(httplib::Request&, httplib::Response&, httplib::Stream&)+2200
 000010008023eec0 fp 1000005f408 lr httplib::Server::process_request(httplib::Stream&, bool, bool&, std::__1::function<void (httplib::Request&)> const&)+2508
 000010008023f010 fp 1000005f780 lr _ZN7httplib6detail26process_server_socket_coreIZNS0_21process_server_socketIZNS_6Server24process_and_close_socketEiEUlRNS_6StreamEbRbE_EEbRKNSt3__16atomicIiEEimlllllT_EUlbS6_E_EEbSC_imlSD_+580
 000010008023fd40 fp 1000005f8bc lr httplib::Server::process_and_close_socket(int)+144
 000010008023fe60 fp 100000159c8 lr _ZNSt3__110__function16__policy_invokerIFvvEE11__call_implINS0_20__default_alloc_funcIZN7httplib6Server15listen_internalEvEUlvE0_S2_EEEEvPKNS0_16__policy_storageE+36
 000010008023ff10 fp 1000001c5f8 lr std::__1::function<void ()>::operator()() const+20
 000010008023ff20 fp 1000001ca44 lr _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_+348
 000010008023ff30 fp 10000191d28 lr PosixThread+80
 000010008023ff90 fp 100001adfd0 lr __stack_call+24
[1]    97185 abort      ./llamafile-server-0.4 -m models/starling-lm-7b-alpha.Q4_K_M.gguf --nobrowser

Duplicate issue #160: Server crash when requesting stream mode?
Created: 2024-01-02T12:58:21Z
This seems to work after a long long time
# Command
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "messages": [{ "role": "user", "content": "tell me history of canada" }] }'
# Response
{"choices":[{"finish_reason":"stop","index":0,"message":{"content":"Canada is a country located in North America. It is the second largest country in the world by land area and shares its southern border with the United States of America. The history of Canada is a long and complex one, with many different cultures and influences shaping its development.\n\nThe first inhabitants of Canada were the indigenous peoples, who have lived in the region for thousands of years. These include the Inuit, who live in the Arctic, the First Nations, who live throughout the country, and the Metis, who are a mixed-race people with both indigenous and European ancestry.\n\nEuropean exploration and colonization of Canada began in the 15th century, with the arrival of Christopher Columbus. In the following centuries, many European powers, including the Spanish, French, English, and Dutch, established colonies in different parts of the country.\n\nThe French colony, known as New France, was established in the 17th century and was located in what is now the province of Quebec. The English colony, known as British North America, was established in the 17th and 18th centuries and was located in what is now the provinces of Nova Scotia, New Brunswick, and Prince Edward Island.\n\nIn the 19th century, Canada began to grow and develop as a nation. The Confederation Act of 1867 established the Dominion of Canada as a single country, and the country began to expand westward, with the addition of the provinces of British Columbia, Alberta, Saskatchewan, and Manitoba.\n\nThroughout its history, Canada has faced many challenges, including conflicts with indigenous peoples, economic struggles, and political divisions. However, the country has also made significant contributions to the world, including the development of the Canadian Shield, the creation of the Canadian flag, and the establishment of the Canadian Armed Forces.\n\nToday, Canada is a diverse and multicultural country, with a population of over 37 million people. It is a member of the Commonwealth of Nations and the United Nations, and is known for its natural beauty, its multiculturalism, and its commitment to social justice and equality.\n\nI hope this information helps you understand the history of Canada. Let me know if you have any other questions.","role":"assistant"}}],"created":1704195853,"id":"chatcmpl-IrL7TZLE86Y9NF27RWOb1QICsmCEEPA2","model":"uknown","object":"chat.completion","usage":{"completion_tokens":492,"prompt_tokens":35,"total_tokens":527}}
However this command seems to initiate crashes when requesting stream mode
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "stream": "true", "messages": [{ "role": "user", "content": "tell me history of canada" }] }'
Which shows something like:
error: Uncaught SIGABRT (SI_TKILL) at 0x3e80017571d on mofosyne-Z97MX-Gaming-5 pid 1529629 tid 1529668
  ./mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile
  No error information
  Linux Cosmopolitan 3.1.2 MODE=x86_64; #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 mofosyne-Z97MX-Gaming-5 5.15.0-91-generic

RAX 0000000000000000 RBX 0000100080bbe220 RDI 0000000000175744
RCX 0000000000604256 RDX 0000000000000000 RSI 0000000000000006
RBP 0000100080bbe130 RSP 0000100080bbe130 RIP 0000000000604256
 R8 0000000000000000  R9 0000000000000006 R10 0000000000604256
R11 0000000000000296 R12 0000000000000006 R13 0000100080bbe260
R14 0000100080350b30 R15 0000100080350b38
TLS 0000100080341430

XMM0  00000000000000000000000000000000 XMM8  0000100080bbebb80000100080bbebb8
XMM1  00000000000000000000000000000000 XMM9  0000100080bbebb80000100080bbebb8
XMM2  00000000000000000000000064656573 XMM10 0540f600fde0fd800a80fc00072002c0
XMM3  00000000000000008000000000000050 XMM11 03030303030303030303030303030303
XMM4  00000000000000000000100080bbebcc XMM12 0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f
XMM5  0000100080bbebcc0000100080bbebcc XMM13 20202020202020202020202020202020
XMM6  79746c616e65705f65636e6573657270 XMM14 fd600500fd0005a0fc800b600940ff60
XMM7  75727422203a226d616572747322207b XMM15 ffffa8d00001194000015f90ffffe8cc

cosmoaddr2line /home/mofosyne/Documents/LLM model/llamafile/mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile 604256 5f9ae1 411659 410bbb 42d2c3 4398e7 43acdb 466f19 477aba 478abe 479361 43bd55 43ef1f 5e5d7c 6269d1

0x0000000000604256: ?? ??:0
0x00000000005f9ae1: ?? ??:0
0x0000000000411659: ?? ??:0
0x0000000000410bbb: ?? ??:0
0x000000000042d2c3: ?? ??:0
0x00000000004398e7: ?? ??:0
0x000000000043acdb: ?? ??:0
0x0000000000466f19: ?? ??:0
0x0000000000477aba: ?? ??:0
0x0000000000478abe: ?? ??:0
0x0000000000479361: ?? ??:0
0x000000000043bd55: ?? ??:0
0x000000000043ef1f: ?? ??:0
0x00000000005e5d7c: ?? ??:0
0x00000000006269d1: ?? ??:0

10008004-1000806e rw-pa-    107x automap 6848kB w/ 64kB hole
10008070-100080b5 rw-pa-     70x automap 4480kB w/ 128kB hole
100080b8-100080c3 rw-pa-     12x automap 768kB w/ 52mB hole
10008400-10008800 rw-pa-  1'025x automap 64mB w/ 64mB hole
10008c00-10009090 rw-pa-  1'169x automap 73mB w/ 1788mB hole
10010049-1002056f r--s-- 66'855x automap 4178mB w/ 4009mB hole
10030000-10040461 r--s-- 66'658x automap 4166mB w/ 96tB hole
6fc00004-6fc00004 rw-paF      1x nsync 64kB w/ 64gB hole
6fd00004-6fd0000c rw-paF      9x zipos 576kB w/ 64gB hole
6fe00004-6fe00004 rw-paF      1x g_fds 64kB
# 8494mB total mapped memory
./mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile -m mistral-7b-instruct-v0.1.Q4_K_M.gguf

Closing comment (2024-01-05T03:37:50Z): This is a duplicate of #116. You're passing invalid JSON. Your stream parameter is "true" but the server wants a boolean. There's not much we can do to solve this right now, other than crashing with a friendlier error message. I'll go ahead and change that.