Hugging face token-classification output causes non JSON serializable error

[ajsalow]

Looks like there was a recent MR to support all types of hugginface inputs/outputs however token-classification outputs are failing for me. I was able to reproduce with a couple token-classification models, cmarkea/distilcamembert-base-ner for example threw the following error.

Using docker image index.docker.io/seldonio/mlserver@sha256:7f7806e8ed781979bb5ef4d7774156a31046c8832d76b57403127add33064872

2022-12-16 19:42:28,391 [mlserver] INFO - Loaded model 'huggingface-token-class' succesfully.
2022-12-16 19:42:28,436 [mlserver] INFO - Loaded model 'huggingface-token-class' succesfully.
2022-12-16 19:43:05,194 [mlserver] DEBUG - Payload id='e05f34eb-014f-4b76-9c00-8ec9b29ff3aa' parameters=Parameters(content_type=None, headers={'host': 'localhost:8080', 'user-agent': 'python-requests/2.28.1', 'content-length': '131', 'accept': '*/*', 'accept-encoding': 'gzip, deflate, br', 'content-type': 'application/json', 'x-forwarded-for': '127.0.0.1', 'Ce-Specversion': '0.3', 'Ce-Source': 'io.seldon.serving.deployment.mlserver.default', 'Ce-Type': 'io.seldon.serving.inference.request', 'Ce-Modelid': 'huggingface-token-class', 'Ce-Inferenceservicename': 'mlserver', 'Ce-Endpoint': 'huggingface-token-class', 'Ce-Id': 'e05f34eb-014f-4b76-9c00-8ec9b29ff3aa', 'Ce-Requestid': 'e05f34eb-014f-4b76-9c00-8ec9b29ff3aa', 'Ce-Namespace': 'default'}) inputs=[RequestInput(name='args', shape=[1], datatype='BYTES', parameters=None, data=TensorData(__root__=['My name is Clara and I live in Berkeley, California.']))] outputs=None
2022-12-16 19:43:05,272 [mlserver] DEBUG - Prediction [[{'entity': 'I-MISC', 'score': 0.58051825, 'index': 5, 'word': '▁Clara', 'start': 10, 'end': 16}, {'entity': 'I-LOC', 'score': 0.9975036, 'index': 10, 'word': '▁Ber', 'start': 30, 'end': 34}, {'entity': 'I-LOC', 'score': 0.99758935, 'index': 11, 'word': 'ke', 'start': 34, 'end': 36}, {'entity': 'I-LOC', 'score': 0.99756134, 'index': 12, 'word': 'ley', 'start': 36, 'end': 39}, {'entity': 'I-LOC', 'score': 0.99869055, 'index': 14, 'word': '▁Cali', 'start': 40, 'end': 45}, {'entity': 'I-LOC', 'score': 0.99901426, 'index': 15, 'word': 'for', 'start': 45, 'end': 48}, {'entity': 'I-LOC', 'score': 0.9989453, 'index': 16, 'word': 'nia', 'start': 48, 'end': 51}]]
2022-12-16 19:43:05,273 [mlserver.parallel] ERROR - An error occurred calling method 'predict' from model 'huggingface-token-class'.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/mlserver/parallel/worker.py", line 122, in _process_request
    return_value = await method(*request.method_args, **request.method_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mlserver_huggingface/runtime.py", line 109, in predict
    str_out = [json.dumps(pred, cls=NumpyEncoder) for pred in prediction]
  File "/opt/conda/lib/python3.8/site-packages/mlserver_huggingface/runtime.py", line 109, in <listcomp>
    str_out = [json.dumps(pred, cls=NumpyEncoder) for pred in prediction]
  File "/opt/conda/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/opt/conda/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/conda/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/conda/lib/python3.8/site-packages/mlserver_huggingface/common.py", line 135, in default
    return json.JSONEncoder.default(self, obj)
  File "/opt/conda/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type float32 is not JSON serializable
INFO:     None:0 - "POST /v2/models/huggingface-token-class/infer HTTP/1.1" 500 Internal Server Error

[ajsalow]

I think I have the fix. Will open an PR shortly.

[pepesi]

From the exception, That image used an old version code, not the latest. May the pr is not released

[ajsalow]

@pepesi I'm not sure what you mean? The image I used is the latest tagged release, from a week ago and #692 was merged in that version of the code. What's more is I was able to reproduce locally when latest code on the main branch . Please see the MR I have opened to resolve the exception being thrown for token-classification tasks.

[pepesi]

That code is not released on branch release/1.2.x, you can view the code on here

[ajsalow]

Ahh ok, I see now it wasn't in the tag but there is still an issue with master. Toke-classification output is still wrong. These changes are needed https://github.com/SeldonIO/MLServer/pull/905/files

[pepesi]

Not sure what happened, could you show more detail?

[ajsalow]

I gave steps to reproduce in the PR. Load the model mentioned here or any token-classification model and send it a request. The PR fixes the issue, numpy floats were not being converted before being serialized to json.

[pepesi]

I think the pr #905 may have no effect on the problem, and the key is that you used an old version code. because when encoding a response, content_type in metadata will not take effect, just find codec by the content, not content_type. (code like below)

MLServer/mlserver/codecs/utils.py

Lines 90 to 104 in a685106

	def encode_inference_response(
	payload: Any,
	model_settings: ModelSettings,
	) -> Optional[InferenceResponse]:
	# TODO: Allow users to override codec through model's metadata
	codec = find_request_codec_by_payload(payload)
	if not codec:
	return None
	model_version = None
	if model_settings.parameters:
	model_version = model_settings.parameters.version
	return codec.encode_response(model_settings.name, payload, model_version)

Could you please just use the clean master code on your local and try it again?

[adriangonz]

Hey @ajsalow @pepesi ,

I can confirm that @pepesi's PR didn't make it to the 1.2.0 release unfortunately (by just a week or so), and since it's a major change it's not in 1.2.1, which is just a patch release.

So, just to double check, did @pepesi's suggestion sort out your issue?

PS: Thanks a lot for following up on this one @pepesi 👍

[ajsalow]

Yes it did, thanks.