Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Another encoding issue #2

Open
frdmn opened this issue Nov 10, 2014 · 3 comments
Open

Another encoding issue #2

frdmn opened this issue Nov 10, 2014 · 3 comments

Comments

@frdmn
Copy link
Contributor

frdmn commented Nov 10, 2014

As mentioned in #1, there's still an issue with some surrogates:

<*sql> Could not save SAY to database caused by: <class 'UnicodeEncodeError'> 'utf-8' codec can't encode character '\udcf3' in position 15: surrogates not allowed

and

<*sql> Could not save SAY to database caused by: <class 'UnicodeEncodeError'> 'utf-8' codec can't encode character '\udcd0' in position 0: surrogates not allowed
@frdmn
Copy link
Contributor Author

frdmn commented Nov 11, 2014

Some more:

<*sql> Could not save SAY to database caused by: <class 'UnicodeEncodeError'> 'utf-8' codec can't encode character '\udcb4' in position 3: surrogates not allowed
<*sql> Could not save SAY to database caused by: <class 'UnicodeEncodeError'> 'utf-8' codec can't encode character '\udcb4' in position 7: surrogates not allowed
<*sql> Could not save SAY to database caused by: <class 'UnicodeEncodeError'> 'utf-8' codec can't encode character '\udcfc' in position 164: surrogates not allowed

@buxxi
Copy link
Owner

buxxi commented Nov 11, 2014

It says EncodeError and not DecodeError, so I guess it fails on this: message.encode('utf-8') @ line 112

Does that mean that the incoming message is something that can't be encoded to utf-8, maybe utf-16?

@frdmn
Copy link
Contributor Author

frdmn commented Nov 11, 2014

@buxxi Not sure if it's that elegant, but we could suppress those encoding issues for now:

message = message.encode('utf-8', 'ignore').decode('utf-8', 'replace')

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants