Add MySql and Sqlite drivers/packages to CI testing

[jamescheney]

So that we can say that they are "supported" (and decrease the likelihood that they bit rot).

[dhil]

I believe the current status is as follows

• Both PostgreSQL and SQLite3 get installed during CI
• Only PostgreSQL is tested during CI
• MySQL remains unsupported

[jamescheney]

I think that's correct.

Testing SQLite 3 could just be a matter of adapting the existing tests to create/initialize and connect to a SQLite 3 database - theoretically both flat and shredded queries should work now thanks to #495.

[dhil]

Reeled in from #505

It seems Ubuntu 16.04 (as provided by Azure Pipelines) only offers SQLite3 version 3.11, whilst we require at least version 3.25 (c.f. release notes). So I suppose we need to find a third-party repository which offers a more recent version of SQLite3 before we can enable the shredding tests.

[jamescheney]

SQLite 3.25 (which added support for ROW_NUMBER etc.) is only needed if we want to test shredding for SQLite3 on the CI. I suggest that we don't wait around indefinitely and add SQLite3 and MySQL CI support but just run the plain database tests, not the shredding ones. Whenever the default SQLite3 and/or MySQL versions available have support for the needed features, enabling the shredding tests for them should be a one-or-two-liner.

[jamescheney]

It also looks like some work will be needed for the OCaml MySQL library to become compatible with Mysql 8...

ygrek/ocaml-mysql#13

so for now the ambition will just be to get the old mysql driver working again with Mysql 5.*, and adapt the CI to run / test the flat query examples with mysql and sqlite3.

[jamescheney]

WIP in branch issue322 has reinstated the links-mysql package, with changes needed to make it compile. However, when attempting to dynamically load the links-mysql library I get this error:

links> @load "factorials.links";
compiling to IR
compiled IR
debug: found loadable database driver
- /Users/jcheney/src/links/_build/install/default/lib/links-mysql/links_mysql.cmxs
- /Users/jcheney/src/links/_build/install/default/share/links-mysql/links_mysql_dependencies.json

***: Error: Cannot load plugin dependency '/Users/jcheney/.opam/4.07.0/lib/mysql/mysql.cmxs' (link error: no implementation available for Stdlib__stdLabels)
 

I've tried to make the build process for links-mysql as close as possible to that for links-postgresql; the OPAM mysql installation is fresh. Any thoughts @dhil or @frank-emrich ?

[jamescheney]

It seems the ocaml mysql library does use the StdLabels libraries (though not in any essential way) so presumably Dynlink is complaining because the rest of Links doesn't use them... I've tried a couple of ways to force these modules to be referenced but without success so far.

[jamescheney]

Adding the mysql library to core/dune and adding a reference to it from somewhere in Links.core makes the dynamic loading succeed, which enables testing that the old mysql driver still works. However, we don't want to do this, both to avoid the unnecessary dependency and because the OPAM mysql package won't install unless mysql libraries are actually present, thus forcing an additional dependency on system libraries that we don't want.

I ran across the original code @Yuren-Zhong wrote to force loading of dependencies:

https://github.com/Yuren-Zhong/Modularizing_and_Dynamic_Linking_in_Links/blob/master/dynload.ml

but doing the same thing here doesn't work.

However, adding the -linkall flag to dune does work. @Yuren-Zhong's attempt to use this ran into a problem with cohttp that no longer appears to be an obstacle. That said, -linkall has its downsides too: for example the size of links.exe goes from 12MB to 15MB.

[dhil]

WIP in branch issue322 has reinstated the links-mysql package, with changes needed to make it compile. However, when attempting to dynamically load the links-mysql library I get this error:

This issue does not occur when compiling with OCaml 4.10.0. I think the issue here is that for some relative recent version of OCaml the standard library got repackaged. We can either bump the minimum required version of OCaml to compile Links or possibly try to be a bit more clever about dynamic loading for OCaml < 4.x where x is the minor version that repackaged the standard library.

[jamescheney]

OK. I am using OCaml 4.07. Note however that since I wrote the comment you replied to on Sunday, I found a workaround using the -linkall flag. Is this flag present or absent in the root dune file in your checkout of branch issue322? If it is present, then does it still work if you remove it?

[dhil]

It is absent. I deliberately checked out an earlier version of your branch. I have tested with OCaml 4.10.0 (no error) and 4.06.1 (error occurs).

[jamescheney]

Thanks! Since there's a workaround I don't think this is a sufficient reason for requiring a higher ocaml version but am happy with that resolution if everyone else prefers that.

If there's a way to make this flag conditional on ocaml version < 4.10 (which is known to not need it) then this would document that the flag can go away once Links's required ocaml version is >= 4.10.