Linkifier doesn't handle some unicode strings

[Tyriar]

I was this in VS Code today:

Failing on 140:

xterm.js/src/Linkifier.ts

Line 223 in bdb6647

const attr: number = char[CHAR_DATA_ATTR_INDEX];

I have had reports of some links not working, perhaps this is causing it (by failing on a row before?).

[jerch]

Oh thats weird, a char should not be empty, at least I am not aware of any code that initializes a line with no cells defined.
Do have some data to repro it?

[Tyriar]

@jerch nope, but it definitely happened as it was in the console 😄. I think it's related to the issue of unicode characters taking up multiple indexes in a string but only 1-2 cells. This fix will fix the issue and if we spot a link with a white underline we will have a test case for it.

[jerch]

@Tyriar
This doesnt work at all: echo http://💩.la
The '💩' is a surrogate char in UTF16, the linkifier does not recognize this at all, no errors.

Edit:
This is not recognized either: http://café.de
The 'é' is a combined char. No errors either.

[Tyriar]

@jerch I was thinking more: echo '💩 http://something.com' breaking the text index

[jerch]

Yepp thats it - if you insert spaces before the poo in a way that "http..." wraps to the next line the error above appears. This triggers the error in the demo:

echo '                                        💩 http://something.com'

Might be related to the right border nightmare, no clue yet.
It also happens with the combining char instead of the surrogate.
Oh well 3.6 does not throw the error, still the underline is not in the right place.

Edit:
This is quite broken due to the way the string index offset gets calculated - it does not respect fullwidth/surrogate/combining chars (shows misalignments by those chars). Same goes for the selection manager (just search for poo symbol). The error at hand basically tries to access cols behind the last col, should not have worked before either but it did with faulty results, no clue why lol. Well the fix is halfway done.

[Tyriar]

I merged the PR but let's reopen this to keep tracking it.

[Tyriar]

I'm seeing wrapping being broken on macOS/bash when I use 💩:

[jerch]

@Tyriar Could you test if the input problem with the PR #1678 persists?

[Tyriar]

@jerch yep it's in my queue 😃

Also just saw this (not running your change):

Test text: ~/playground/jekyll❯ echo './Makefile'