Problem with documents not being able to be registered in high-quality mode

[tomy-kyu]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Content

There is a problem with the knowledge function.
When I try to register a document in high-quality mode, I get a “500 Internal Server Error: PluginInvokeError”.
However, it seems possible to register in inverted index mode. After that, it seems possible to switch to high-quality mode, but when I try to add more documents from there, the same thing happens.
When I checked for similar issues, I found #14708, but unlike this case, I am not using either Tongyi or Cohere. I am using the Embedding model from AOAI.
The current revision is f368a679b.

If anyone knows how to deal with this, please let me know.

Mar  4 12:56:52 ws01 b74769e89457[1364543]: 2025-03-04 03:56:52.847 ERROR [Dummy-27] [app.py:875] - Exception on /console/api/datasets/indexing-estimate [POST]
Mar  4 12:56:52 ws01 b74769e89457[1364543]: Traceback (most recent call last):
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/controllers/console/datasets/datasets.py", line 452, in post
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     response = indexing_runner.indexing_estimate(
Mar  4 12:56:52 ws01 b74769e89457[1364543]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/indexing_runner.py", line 290, in indexing_estimate
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     documents = index_processor.transform(
Mar  4 12:56:52 ws01 b74769e89457[1364543]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 58, in transform
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     document_nodes = splitter.split_documents([document])
Mar  4 12:56:52 ws01 b74769e89457[1364543]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/rag/splitter/text_splitter.py", line 96, in split_documents
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     return self.create_documents(texts, metadatas=metadatas)
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/rag/splitter/text_splitter.py", line 81, in create_documents
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     for chunk in self.split_text(text):
Mar  4 12:56:52 ws01 b74769e89457[1364543]:                  ^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 68, in split_text
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     chunks_lengths = self._length_function(chunks)
Mar  4 12:56:52 ws01 b74769e89457[1364543]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 38, in _token_encoder
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     return embedding_model_instance.get_text_embedding_num_tokens(texts=texts)
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/model_manager.py", line 244, in get_text_embedding_num_tokens
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     self._round_robin_invoke(
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     return function(*args, **kwargs)
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 65, in get_num_tokens
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     return plugin_model_manager.get_text_embedding_num_tokens(
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/plugin/manager/model.py", line 313, in get_text_embedding_num_tokens
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     for resp in response:
Mar  4 12:56:52 ws01 b74769e89457[1364543]:                 ^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     self._handle_plugin_daemon_error(error.error_type, error.message)
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     raise PluginInvokeError(description=message)
Mar  4 12:56:52 ws01 b74769e89457[1364543]: core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"error_type":"unmarshal_error","message":"unmarshal json failed: json: cannot unmarshal number into Go struct field GetTextEmbeddingNumTokensResponse.num_tokens of type []int"}
Mar  4 12:56:52 ws01 b74769e89457[1364543]: During handling of the above exception, another exception occurred:
Mar  4 12:56:52 ws01 b74769e89457[1364543]: Traceback (most recent call last):
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     rv = self.dispatch_request()
Mar  4 12:56:52 ws01 b74769e89457[1364543]:          ^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     resp = resource(*args, **kwargs)
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/.venv/lib/python3.12/site-packages/flask/views.py", line 110, in view
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     resp = meth(*args, **kwargs)
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/controllers/console/wraps.py", line 147, in decorated
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     return view(*args, **kwargs)
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/libs/login.py", line 94, in decorated_view
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     return current_app.ensure_sync(func)(*args, **kwargs)
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/controllers/console/wraps.py", line 27, in decorated
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     return view(*args, **kwargs)
Mar  4 12:56:52 ws01 b74769e89457[1364543]:            ^^^^^^^^^^^^^^^^^^^^^
Mar  4 12:56:52 ws01 b74769e89457[1364543]:   File "/app/api/controllers/console/datasets/datasets.py", line 468, in post
Mar  4 12:56:52 ws01 b74769e89457[1364543]:     raise IndexingEstimateError(str(e))
Mar  4 12:56:52 ws01 b74769e89457[1364543]: controllers.console.datasets.error.IndexingEstimateError: 500 Internal Server Error: PluginInvokeError: {"error_type":"unmarshal_error","message":"unmarshal json failed: json: cannot unmarshal number into Go struct field GetTextEmbeddingNumTokensResponse.num_tokens of type []int"}

[tomy-kyu]

I've solved it myself.
In conclusion, I've confirmed that the problem is with the way the Embedding model is handled in the AOAI Plugin, not in the Dify main body, so I'm closing this discussion.

[tomy-kyu]

langgenius/dify-official-plugins@a086d4e
The above merge has now been successfully implemented. Thanks to the author for their support.