bug: Anthropic Sonnet Streaming is broken.

[chubshaun]

Jan version

v0.5.15

Describe the Bug

Previous version of Jan worked fine, pulled the latest version and checked out the release/v0.5.15 branch then ran make dev on node 20x. Streaming doesn't work with Anthropic and immediately throws an error. If I disable streaming, it continues to work properly. Tried with fresh git pull of jan github repo into another directory as well as deleting user directory ~/jan - made no difference.

Screenshots / Logs

2025-02-26T21:50:55.825Z [SPECS]::Version: 0.1.17406066532025-02-26T21:50:55.825Z [SPECS]::CPUs: [{"model":"Apple M1 Max","speed":2400,"times":{"user":979811350,"nice":0,"sys":588689430,"idle":1374367180,"irq":0}},{"model":"Apple M1 Max","speed":2400,"times":{"user":978609240,"nice":0,"sys":583894810,"idle":1380647590,"irq":0}},{"model":"Apple M1 Max","speed":2400,"times":{"user":429758180,"nice":0,"sys":345259560,"idle":2169453120,"irq":0}},{"model":"Apple M1 Max","speed":2400,"times":{"user":295453430,"nice":0,"sys":226645160,"idle":2426957610,"irq":0}},{"model":"Apple M1 Max","speed":2400,"times":{"user":206358840,"nice":0,"sys":134383370,"idle":2610573360,"irq":0}},{"model":"Apple M1 Max","speed":2400,"times":{"user":162611290,"nice":0,"sys":95718440,"idle":2693974750,"irq":0}},{"model":"Apple M1 Max","speed":2400,"times":{"user":136004540,"nice":0,"sys":67879610,"idle":2750864410,"irq":0}},{"model":"Apple M1 Max","speed":2400,"times":{"user":82690660,"nice":0,"sys":41243110,"idle":2832607020,"irq":0}},{"model":"Apple M1 Max","speed":2400,"times":{"user":54027460,"nice":0,"sys":23821180,"idle":2879600140,"irq":0}},{"model":"Apple M1 Max","speed":2400,"times":{"user":42532540,"nice":0,"sys":16908450,"idle":2898338010,"irq":0}}]
2025-02-26T21:50:55.826Z [SPECS]::Endianness: LE
2025-02-26T21:50:55.826Z [SPECS]::Total Mem: 34359738368
2025-02-26T21:50:55.826Z [SPECS]::OS Platform: darwin
2025-02-26T21:50:55.826Z [SPECS]::Free Mem: 92274688
2025-02-26T21:50:55.826Z [SPECS]::OS Release: 23.6.0
2025-02-26T21:50:55.826Z [SPECS]::Parallelism: 10
2025-02-26T21:50:55.826Z [SPECS]::OS Version: Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:30 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6000
2025-02-26T21:50:55.826Z [SPECS]::Machine: arm64
2025-02-26T21:51:15.463Z [CORTEX]:: Spawn cortex at path: jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/cortex-server
2025-02-26T21:51:15.463Z [CORTEX]:: Spawning cortex subprocess...
2025-02-26T21:51:15.464Z [APP]::Starting process: jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/cortex-server --start-server --port 39291 --config_file_path jan-data-folder/.janrc --data_folder_path /~/jan --loglevel INFO
2025-02-26T21:51:17.884Z [APP]::Process output: Set log level to INFO

20250226 21:52:35.236317 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:52:35.283692 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:52:35.291267 UTC 219572987 INFO  RetrieveThread: 01JN23XNZQ24JHR62G1GTKTWNH - thread_service.cc:43
20250226 21:52:35.292088 UTC 219572987 INFO  RetrieveThread: 01JN23XNZQ24JHR62G1GTKTWNH - thread_service.cc:43
20250226 21:52:35.293940 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:02.163881 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:02.166784 UTC 219572987 ERROR Failed to delete message: Message not found - message_service.cc:116
20250226 21:53:02.168264 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:02.181287 UTC 219572987 INFO  CreateMessage for thread 01JN23XNZQ24JHR62G1GTKTWNH - message_fs_repository.cc:15
20250226 21:53:02.182998 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:02.218856 UTC 219572989 INFO  Origin:  - main.cc:258
20250226 21:53:02.220083 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:03.941766 UTC 219572989 INFO  RetrieveThread: 01JN23XNZQ24JHR62G1GTKTWNH - thread_service.cc:43
20250226 21:53:03.943533 UTC 219572989 INFO  RetrieveThread: 01JN23XNZQ24JHR62G1GTKTWNH - thread_service.cc:43
20250226 21:53:03.946207 UTC 219572989 INFO  Origin:  - main.cc:258
20250226 21:53:03.974904 UTC 219572989 INFO  Origin:  - main.cc:258
20250226 21:53:06.945953 UTC 219572989 INFO  RetrieveAssistant: 01JN23XNZQ24JHR62G1GTKTWNH - assistant_service.cc:28
20250226 21:53:06.949320 UTC 219572989 INFO  Origin:  - main.cc:258
20250226 21:53:07.874686 UTC 219572989 INFO  RetrieveAssistant: 01JN23XNZQ24JHR62G1GTKTWNH - assistant_service.cc:28
20250226 21:53:07.876805 UTC 219572989 INFO  Origin:  - main.cc:258
20250226 21:53:10.676554 UTC 219572989 INFO  Origin:  - main.cc:258
20250226 21:53:10.688911 UTC 219572989 ERROR Failed to delete message: Message not found - message_service.cc:116
20250226 21:53:10.690864 UTC 219572989 INFO  Origin:  - main.cc:258
20250226 21:53:10.702290 UTC 219572989 INFO  CreateMessage for thread 01JN23XNZQ24JHR62G1GTKTWNH - message_fs_repository.cc:15
20250226 21:53:10.704337 UTC 219572989 INFO  Origin:  - main.cc:258
20250226 21:53:10.739885 UTC 219572989 INFO  Origin:  - main.cc:258
20250226 21:53:13.052145 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:13.058977 UTC 219572987 INFO  RetrieveThread: 01JN23XNZQ24JHR62G1GTKTWNH - thread_service.cc:43
20250226 21:53:13.060147 UTC 219572987 INFO  RetrieveThread: 01JN23XNZQ24JHR62G1GTKTWNH - thread_service.cc:43
20250226 21:53:13.062360 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:13.097928 UTC 219572987 INFO  CreateMessage for thread 01JN23XNZQ24JHR62G1GTKTWNH - message_fs_repository.cc:15
20250226 21:53:13.099939 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:17.712423 UTC 219572987 INFO  CreateMessage for thread 01JN23XNZQ24JHR62G1GTKTWNH - message_fs_repository.cc:15
20250226 21:53:17.714158 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:20.345746 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:20.351803 UTC 219572987 INFO  RetrieveThread: 01JN23XNZQ24JHR62G1GTKTWNH - thread_service.cc:43
20250226 21:53:20.352826 UTC 219572987 INFO  RetrieveThread: 01JN23XNZQ24JHR62G1GTKTWNH - thread_service.cc:43
20250226 21:53:20.354573 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:20.393037 UTC 219572987 INFO  CreateMessage for thread 01JN23XNZQ24JHR62G1GTKTWNH - message_fs_repository.cc:15
20250226 21:53:20.395026 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:23.802583 UTC 219572987 INFO  RetrieveAssistant: 01JN23XNZQ24JHR62G1GTKTWNH - assistant_service.cc:28
20250226 21:53:23.807277 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:26.145077 UTC 219572987 INFO  CreateMessage for thread 01JN23XNZQ24JHR62G1GTKTWNH - message_fs_repository.cc:15
20250226 21:53:26.146879 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:26.161989 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:29.165825 UTC 219572987 INFO  RetrieveThread: 01JN23XNZQ24JHR62G1GTKTWNH - thread_service.cc:43
20250226 21:53:29.167740 UTC 219572987 INFO  RetrieveThread: 01JN23XNZQ24JHR62G1GTKTWNH - thread_service.cc:43
20250226 21:53:29.171215 UTC 219572987 INFO  Origin:  - main.cc:258
20250226 21:53:29.218272 UTC 219572987 INFO  Origin:  - main.cc:258

What is your OS?

• MacOS
• Windows
• Linux

[chubshaun]

Updating for anyone else who encounters this issue - there is a fix on Discord. The response format (under Remote Engine -> Anthropic -> Response Format Conversion) needs to be updated with the following code block:

{% if input_request.stream %} {"object": "chat.completion.chunk", "model": "{{ input_request.model }}", "choices": [{"index": 0, "delta": { {% if input_request.type == "message_start" %} "role": "assistant", "content": null {% else if input_request.type == "ping" %} "role": "assistant", "content": null {% else if input_request.type == "content_block_delta" %} "role": "assistant", "content": {{ tojson(input_request.delta.text) }} {% else if input_request.type == "content_block_stop" %} "role": "assistant", "content": null {% else if input_request.type == "content_block_stop" %} "role": "assistant", "content": null {% endif %} }, {% if input_request.type == "content_block_stop" %} "finish_reason": "stop" {% else %} "finish_reason": null {% endif %} }]} {% else %} {{tojson(input_request)}} {% endif %}