NeoVim support

[jeaye]

NeoVim support would require rewriting a good deal of the VimL + Lua binding work. It would also require integration with NeoVim's asynchronous API.

Preliminary work was done here: neovim/neovim#719 (comment)

[tony]

For inspiration https://github.com/critiqjo/lldb.nvim and https://github.com/Shougo/deoplete.nvim.

Both of these solutions use Python 3. We had an issue before where we discussed lua vs python etc.

I'm confused. I'm reading various sources that are conflicting. Lua is supposed to play such a big role in neovim - why are deoplete.vim and lldb.nvim in python?

[jeaye]

@tony I've found that confusing as well. Lua is supposed to play a huge part in NeoVim's replacement of VimL, but that's different from the usage of if_lua, I think. It was really surprising to see.

[tony]

@jeaye I just got my vim-config working with deoplete.vim. Some fast stuff.

Well it's not just that, I was of the impression @shougu was also a big lua fan.

Also, neovim requires installing a module from pip.

That said, some if the issues we were talking about in an earlier issue #26. For neovim, maybe we need to dig a bit deeper to find out why they're picking python.

As for neovim, aside from the tests I don't see much adoption of Lua anywere. It may be faster / more technically pure / etc. It looks beautiful on paper. In practice, I really don't want to be in a situation where I'm doing everything in python but neovim's usage of lua ends up becoming, in context, another viml.

[tony]

Update, fixed issue to #26.

[jeaye]

As for neovim, aside from the tests I don't see much adoption of Lua anywere.

Of course there is, see neovim/neovim#243. Also, https://github.com/neovim/neovim/wiki/Progress

[dedowsdi]

I'm using neovim, but i'd like to give you my star in advance. Come on!

[tbastos]

NeoVim currently lacks color_coded, while vanilla vim lacks a decent front-end to LLDB (NeoVim has a nice one: https://github.com/critiqjo/lldb.nvim). As a C/C++ programmer it seems NeoVim already offers more value than Vim. I would have switched if I wasn't a sucker for semantic code highlighting 😜

I like Lua more than Python, but I agree it would make sense to switch to Python to reuse some of YCM's code. Ideally every clang-based plugin should use the same backend.

Hope you can make some progress on the NeoVim front soon! Thanks for this excellent plugin!

[oblitum]

Even YCM devs regret using python. I've seen that a lot in the gitter channel. Maybe it's time to just wait for what's coming in Vim 8? Shougo has worked on a nice remote api for Vim plugins now that should be stable on Vim 8, and as NeoVim theoretically should also support a remote API without requiring to touch python for that (I think I've heard people built plugins for it in Haskell!). It could be a good opportunity to do this remotely on both sides, with the language the author feels better.

[jeaye]

My current goal is for color_coded to merge with YCM when its API supports the delivery of highlighting tokens; I'd then like to simplify color_coded by removing all C++ and expanding to NeoVim and Emacs. It should focus on just getting the tokens and applying it in a platform-specific manner.

[oblitum]

By the way, this new feature of Vim is +channel and +job. Of course, adopting this would require Vim 8, which may not be ready on distros. These features are currently available in recent builds.

[oblitum]

@jeaye do you plan to try to add that API to YCM? I've not heard of that from YCM side.

[oblitum]

Even with ycmd supporting such API for highlighting, there won't be benefit for color_coded in there, except in case its use of libclang got merged with the current use of libclang for completion, so that in the end there would be just one libclang overhead, which is what I find of most beneficial. If that's not the aim, I personally don't care color_coded keeps on its own.

[oblitum]

To back my previous statement: https://github.com/neovimhaskell/nvim-hs

[tbastos]

@jeaye Sounds good, is anyone working on that?

@oblitum The goal is to avoid having to compile libclang twice, invoking libclang twice per file, having to maintain two sets of configs (ycm_extra_conf and .color_coded), and also right now YCM supports compilation databases while color_coded doesn't.

[jeaye]

Sounds good, is anyone working on that?

It was discussed quite a bit here: ycm-core/ycmd#291 I'm not sure if that PR is actually going to replace color_coded or not; if we get that YCM API for highlight tokens, though, I'll do my part to migrate color_coded.

[alexozer]

Doesn't it seem pretty unmodular / un-Unix-like to merge a syntax highlighter into a code completer? I mean both could use a system-wide libclang, right? Or do libclang versions matter a lot?

[jeaye]

@alexozer It certainly does, but YCM's goal often appears to be "handle all C-family related features to provide IDE-like usage." See my comment here ycm-core/ycmd#291 (comment) Valloric followed up: ycm-core/ycmd#291 (comment)

Given what Valloric said, I'm still a bit puzzled as to why the PR is providing highlighting support, but it seems his goals are in line with my own. If that's the case, we should be seeing YCMD provide a token API which color_coded can use. I hope that's the case.

[oblitum]

@jeaye Thanks for pointing out that discussion, didn't know it was going on.

[phcerdan]

Just FYI, this plugin seems relevant: chromatica

[dkasak]

davits/DyeVim might be relevant to this, as a proof of concept using the new ycmd API, when ycm-core/ycmd#291 finally gets merged.

[jeaye]

I find it so surprising that people would rather redo all of the work other projects have done instead of help improve them. Thanks for the link, @dkasak.

[davidkennedydev]

@phcerdan chromatica is much buggy.
We are at end of 2017 this issue continues open?
@jeaye Do you can describe what's currently incompatible with neovim?

[jeaye]

@davidUser https://github.com/neovim/neovim/wiki/FAQ#can-i-use-lua-based-vim-plugins-eg-neocomplete

[jeaye]

Sorry, I'll add more info beyond the link.

Where the Lua and NeoVim checks are happening

https://github.com/jeaye/color_coded/blob/master/plugin/color_coded.vim#L11-L22

Where Lua is used

The bulk of the Lua code is mixed in-line with VimL here: https://github.com/jeaye/color_coded/blob/master/autoload/color_coded.vim This is used, since Lua is a light-weight bridge between VimL and C++.

What needs to be done

NeoVim provides a Lua interface, but not the same Lua interface as regular Vim. If the checks linked above are commented out, you'll start to see errors when referencing things like vim.eval, which don't exist for NeoVim. Each of these differences needs to be accounted for. In reality, it's not much work, since the autoload/color_coded.vim is only a couple hundred lines and most of it is VimL, not Lua. I don't work in C or C++ anymore these days, so it hasn't been done on my end. Please whip this up and provide a PR!

What that'd get us

Just fixing these Lua differences would allow color_coded to work in NeoVim, but it won't take full advantage of NeoVim's async features. That's fine, to start, since us Vim folks have been using it like this for years, but the async goodness would be likely follow. As for how to do that, I'm not sure, since I haven't done any NeoVim plugin work. I think the most important thing is just getting color_coded to work at all on NeoVim.

[davidkennedydev]

Ok, I had change all inline Lua scripts by the nvim RPC calls, and wrap the vim by a vim with nvim RPC calls.
https://github.com/DavidUser/color_coded/tree/feature/nvim-support

Everything are working until the C++ calls to vim.command(':call color_coded#add_match( ... this pass through my wrap and I can see the calls on log but this call command instead other commands don't have a result.
If someone can help me I really don't know what is happening.

ps:

• Copy the calls by hand and run on nvim works
• Run the command on lua script works
• Just the direct C++ lua::eval doesn't

Copy the log and paste on terminal:

[jeaye]

Wow, awesome work, @davidUser. It looks like you've nearly got it! For solving this add_match issue, I think we may need to talk to someone with some more NeoVim expertise. I have some ideas though, if you're interested.

Summary of related code

In short, for any other followers, it looks like Lua is being invoked here: https://github.com/DavidUser/color_coded/blob/0a9c56fe6ec559923e45fac08c30fa0c285ec2d7/include/vim/commands.hpp#L20-L33

to call the add_match function in VimL here: https://github.com/DavidUser/color_coded/blob/0a9c56fe6ec559923e45fac08c30fa0c285ec2d7/autoload/color_coded.vim#L138-L144

but it's failing (well, not having any visible change). The lua::eval function itself is very simple: https://github.com/DavidUser/color_coded/blob/0a9c56fe6ec559923e45fac08c30fa0c285ec2d7/include/lua/eval.hpp#L24-L25

Ideas for solving

• I think this is a LuaJIT problem.

What if you lua::eval something like vim.command("let g:foobar = 1") from C++? Does it ever get set so you can read it from :echo or VimL? If not, this may be similar to the issue discussed in #66, where color_coded is statically linking with one Lua and (n)vim is dynamically linked with another. You can verify that ldd $(nvim) and ldd color_coded.so. They should have the same liblua, but I'm thinking that NeoVim is using LuaJIT and color_coded is just using Lua. The history behind this is here: #20.

If NeoVim has worked around this LuaJIT problem, then the solution should be as "simple" as linking with that instead.

Thank you for all of your dedicated work!

[davidkennedydev]

@jeaye

What if you lua::eval something like vim.command("let g:foobar = 1") from C++? Does it ever get set so
you can read it from :echo or VimL?

Doesn't work, g:foobar not defined. Same problem.

color_coded is statically linking with one Lua and (n)vim is dynamically linked with another. You can verify that ldd $(nvim) and ldd color_coded.so

Using ldd no liblua to nvim, but not liblua to vim too. And it work's fine on vim.
using strings + grep i find something like a liblua.so on vim but not on nvim.
So really seems that nvim use something different of lualib, like lua JIT.

[jeaye]

Yes indeed, nvim is using LuaJIT on my end. Are you saying that nvim isn't dynamically linking to any liblua on your end? If so, and it's statically linked, then we have the same problem, but it's manifested as nvim having its own statically linked Lua while color_coded is using the dynamically linked one. Still, two Luas.

For vim, I think ldd won't do the trick, but vim --version will. nvim doesn't show all of that extra detail in the version output.

My output on Arch:

$ ldd $(which nvim) | grep lua
	libluajit-5.1.so.2 => /usr/lib/libluajit-5.1.so.2 (0x00007fe68a0ba000)

$ vim --version | grep lua
+cursorshape       +lua/dyn           +ruby/dyn          +X11

[davidkennedydev]

Should both use dynamic libraries or I can just compile color coded with the same static library?

[jeaye]

Both should be using dynamic. If either is using static, then there will be a duplicate library. The only way they can both refer to the same one is to both refer to the same dynamic one.

[davidkennedydev]

So should I rebuild nvim with same dynamic library to work's. 👎
It's a integration really brittle, when the kind of linkage of nvim to lua affect if it work's. Maybe use lua like a C++ bridge to nvim isn't really a good idea.
I'll work on direct C++ call to nvim, now I understand why vim should had language agnostic API.

[jeaye]

You're absolutely right, @davidUser. NeoVim's language-agnostic API is solving this exact problem, in a way which Vim may never. color_coded is designed this way as a result of what Vim provides, Lua being the "simplest" binding language which works reasonably across various platforms (I originally used Ruby, but that didn't work as reliably on MacOS -- Python is even more of a pain than Lua).

It seems like Lua is a fine way to allow VimL to call into C++, but going back through Lua into NeoVim isn't going to be reliable.

Thanks, again, for the superb work. I know this has been frustrating.

[jeaye]

Happy new year! @davidUser how's your progress been ging with the language-agnostic API? Your help with this has been superb.

[gngshn]

Is there any update for this issue? thanks

[davidkennedydev]

I'm being very busy, any contribution is welcome.
You can find the work in progress on https://github.com/davidkennedydev/color_coded/tree/feature/nvim-support.