Roadmap

[davidanthoff]

@fredo-dedup I thought it might be a good idea to have a place to discuss the overall roadmap for this package? I'd be really interested to help polish the few remaining things, make things stable etc. I still feel that this has the potential to be a super important package in the data science stack in julia and would like to help :)

Do you have some specific plans with the package?

[fredo-dedup]

The things that are/were on my mind about this package are :

1. finding the most lightweight infrastructure to allow printing to file. The node dependency creates a huge number of files (I can't understand why npm loads that much files, when webpack is able to squeeze dependencies in super small js files). And loading the full chrome executable is not much better either.
2. finding the right julia syntax. We can't much depart from the tree structure implicit in the JSON spec files unless we enjoy rewriting everything and have a complex translation layer. But on the other hand the tree structure is a bit clumsy sometimes, we should probably provide shortcut functions. One advantage of the current solution using functions for every possible JSON field in the schema is that functions can be linked to a doc string and it can guide the user to find his way when using this package. Using named tuples (coming in v0.7) would fit well the tree structure but we would lose the doc string of functions.

But the main point is that I have very little time to work on this (it's not for lack of wanting to work on it !).
Would you like me to add you to the project's owners ?

[davidanthoff]

I did tag a new release of https://github.com/davidanthoff/NodeJS.jl a couple of days ago that doesn't output the filenames of everything it extracts during build.jl. It doesn't cut down on the number of files that are actually installed, but it certainly makes the user experience a lot better, and the install doesn't clutter up CI logs etc. My gut feeling is that this part is actually pretty stable and reliable, so maybe that is ok? What does seem to cause a fair bit of problems is the Cairo.jl and Rsvg.jl dependency, that we currently use to convert from svg to all the other output formats. Those packages seem to be somewhat volatile. Maybe the situation will become better with BinDeps2. I'm also still looking around for other ways to do that.

I agree with everything you write re 2). I've also thought about the named tuple option, but as you write, the doc thing seems important. I think maybe carefully adding helper functions here and there is the right thing to do for now...

Yes, I'd like to help out, so if you could add me to the project, it would be great! I think for starters I would just try to clean up things and stabilize stuff, for anything bigger I'd open issues do first discuss with you. Just getting your input/thoughts on these things would be super helpful, and then I could hopefully just make the changes myself.

[fredo-dedup]

I just invited you as a Collaborator. Thank you very much for your contributions past and future !

[davidanthoff]

Thanks!

Btw., take a look at https://github.com/davidanthoff/DataVoyager.jl. That tool generates vega-lite specs. I'm almost done integrating that whole story tightly with the package here, which I think will be super powerful.

[fredo-dedup]

I used it yesterday at work, and yes it seems really useful to define the plotting settings for later re-use.
We would just need to convert the JSON copied in the clipboard to a Spec struct.

[davidanthoff]

So I now merged both your new syntax and my new @vlplot macro into master.

Here is one thing that I would really like to do: I'm doing a julia online tutorial in two weeks, and I would love to also show VegaLite.jl. I kind of like my new macro syntax, so I think I'll just continue to polish that, and maybe we can cut a new release before I do the tutorial?

I think for now we can just leave both the macro and your new syntax in there, and then make a decision later whether we drop one or not.

I do hope to make a lot of progress over the next couple of days with the package. I thought maybe the best way to do this is that I open PRs with my work, and then merge them. That would give you a relatively easy way to later trace what I've done. I think I will mostly just polish the macro stuff and leave most of the things you did as they are, so hopefully that won't introduce anything you are not on board with.

[davidanthoff]

I think for now we can close this issue and continue with individual issues for anything that comes up? I feel the package is broadly in really good shape and probably doesn't need many large scale decisions anymore.