--extra-lib-dirs and --extra-include-dirs ignored?

[edsko]

I tried compiling hackage-server with new-build, and hackage-server relies on text-icu, which in my case is installed in the brew Cellar and hence needs additional lib and include paths. However, doing

cabal-nix new-build \
  --extra-include-dirs=/Users/e/homebrew/Cellar/icu4c/54.1/include \
  --extra-lib-dirs=/Users/e/homebrew/Cellar/icu4c/54.1/lib

had no effect, nor had setting the extra-include-dirs and extra-lib-dirs flags in the global section of the cabal.project file. This worked though:

packages: hackage-server.cabal

package text-icu
  extra-include-dirs: /Users/e/homebrew/Cellar/icu4c/54.1/include
  extra-lib-dirs: /Users/e/homebrew/Cellar/icu4c/54.1/lib

package hackage-server
  tests: True

(which arguably is the most logical way to configure this anyway, so perhaps this is merely a matter of warning the user that global lib/include flags are not taken into account for subpackages?).

[hvr]

It appears the same problem exists with global flags not being honored

[dcoutts]

This is a question about what packages do top level options apply to? All packages all the way down, all packages local to the project or what?

The current answer is all packages local to the project. So in this example since text-icu is not local to the project then the top level options do not apply to it.

[hvr]

Btw, here's a quite common use-case I heard from an OSX user w/ homebrew:

package HsOpenSSL
   extra-lib-dirs: /usr/local/Cellar/openssl/1.0.2h_1/lib
   extra-include-dirs: /usr/local/Cellar/openssl/1.0.2h_1/include

[ezyang]

What if we give a warning if these options are passed in from command line? They could be legitimate, but I think in most cases they won't be, and a user can always stick it in cabal.project.local to suppress it.

[gbaz]

Sorry -- dumb and slightly off-topic question here -- but I just want to make sure that the options set in the ~/.cabal/configure file for these two are always respected even under new-build?

[ezyang]

.cabal/config is respected by new-build. See also http://ezyang.com/nix-local-build.html#configuring-builds-with-cabal.project for all the config that is used in this determination.

[peti]

So, let's assume that I want all Cabal build to find header files in /usr/local/include -- regardless of whether it's a local package or an external package I'm building. How would I configure that? Listing that location as an extra-include-dir in ~/.cabal/config used to work, but with new-build it no longer does. What solution exists to there to handle this use-case?

[hvr]

@peti btw, one problem I recently realised is that such a global setting would force every package, including GHC boot libraries to be recompiled, as it would force a new nix-hash (as those flags enter the hash computation). And for non-reinstallable packages such as base cabal would likely end up having to give up, as it couldn't satisfy the request. So I'm not sure we can even provide what you want in a principled truly "global" way. The reason it worked in the past was that Cabal applied flags in ~/.cabal/config only to newly compiled packages, which was maybe convenient but also quite hacky/fragile at the same time, as it introduced statefulness.

[peti]

I see. Yes, that makes sense.

I am basically happy specifying those extra-include-dirs paths for specific packages only, i.e. not globally for everything. However, those paths are specific to the system I'm building on, so I want to configure them in ~/.cabal/config (or something like that). But new-build won't honor such things in the global config file; it honors package-specific configuration options only in the per-project cabal.project file. But system paths don't belong there. They belong to the system.

Now, if I could add

package hopenssl
  extra-include-dirs: /home/simons/src/openssl/include
  extra-lib-dirs: /home/simons/src/openssl/lib

to ~/.cabal/config and have new-build use that information every time it compiles hopenssl for any of my projects, then I would be happy. Is that feasible?

[hvr]

@peti Personally, I think that'd quite reasonable to have in ~/.cabal/config (or nearby).

I do wonder though how to be able to locally override the global setting. Especially with openssl, I've sometimes needed temporarily to use a newer local versions rather than the system-wide installed one (e.g. hookup for example needs a more recent OpenSSL version than is installed systemwide in Debian 8; however I want every other project that doesn't need hookup to use the system-wide openssl version; now imagine if the system-wide was installed in a non-standard location requiring extra-includes), and cabal traditionally considers settings to be of the free monoid kind, and thus appends settings rather than replacing them.

On the other hand, I guess a local package hopenssl configuration could simply override all of the global package hopenssl stanza (as if there never was a global one).

I think we have enough here to turn this into a specification for a feature-request ticket... :-)

[peti]

@hvr, I see what you mean. extra-xxx-dirs are difficult to treat as Moniods. I can think of the following strategies to deal with this issue:

1. Just mconcat all extra paths into one large list and make it the user's responsibility to write a sensible configuration. If the global config file sets an extra include dir for any package foo, then that directory will be part of the search path for all builds of foo, regardless of what more local configuration files say. Therefore, if I want a search path included in some builds of foo but not all, then I cannot set one in the global configuration file.

2. In addition to (1), but we could guarantee that more locally configured search paths occur in that final list before the moreddition to (1), but w globally configured ones. Therefore, if I configure /tmp/foo/include globally and /tmp/bar/include locally, then my project will compile with -I/tmp/bar/include -I/tmp/foo/include. This is dangerous, of course, but it's a pretty common strategy to use. After all, all compilers have /usr/include and their search path, yet we re-direct the compiler elsewhere just by pre-pending another location to that list so that the files in /usr/include are no longer found.

3. We could introduce another kind of search path, say default-extra-include-dirs, which is cleared whenever a extra-include-dirs directive appears anywhere else. That way, I could specify a search path that is used for "almost all builds", i.e. all builds that don't specify something else. Multiple occurrences of default-extra-include-dirs would be mappended, of course.

4. Provide an explicit clear-<option-name> directive that throws away all occurrences of that configuration option that occur in a less local place. So, when I run cabal configure --clear-extra-include-dirs, then that search path will be empty -- regardless of what ~/.cabal/config, cabal.project, or cabal.project.local say. I'm not quite sure how the syntax would work in configuration files, but I suppose there is some sensible way to express that there, too.

[bitc]

If anyone is looking for a workaround: I found that setting the C_INCLUDE_PATH environment variable works (LIBRARY_PATH may also work). This doesn't seem to be documented anywhere, but I saw it mentioned here: #2705 (comment)

For example, I have a project that has a dependency on the lzma package. When running regular cabal new-build it would give an error that it can't find the header file "lzma.h". Using --extra-include-dirs did not help.

But this did work:

$ C_INCLUDE_PATH=/Users/bitc/Downloads/xz-5.2.3/src/liblzma/api cabal new-build

[peti]

Note that search paths configured via environment variables like C_INCLUDE_PATH and others will not be recorded in the package configuration file for the generated library, which means that the build result might be unusable in shell environments that don't have these variables. In case of header files, this is probably not going to pose a problem, but when shared libraries need to be found at link-time, then it certainly will.

[harendra-kumar]

I was compiling GHC and it errored out because it could not find gmp.h. I found that GHC scripts were invoking cabal configure but I did not know where in the ghc make scripts I can specify something like --extra-include-dirs. I knew gmp.h was in `/opt/local/include' but there was no way I could tell cabal about it. I knew there was some environment variable to tell cabal about where to look for c headers, but could not find it anywhere in the documentation. After a lot of search, the comment above about C_INCLUDE_PATH helped. Why is this not documented? This is the second time I ran into this problem and every time I find this at some obscure place.

[vmchale]

The command-line flags still seem to do nothing. Not sure why it exists.

[emilypi]

What's the status of this? I went to install hopenssl today, and the flags were ignored completely.

cabal install hopenssl --extra-include-dirs=/usr/local/opt/openssl/include --extra-lib-dirs=/usr/local/opt/openssl/lib

Is still not honored.

[hvr]

I went to install hopenssl today, and the flags were ignored completely.

They are not completely ignored; they currently only apply to local packages (i.e. locally unpacked ones e.g.) but not to non-local ones;

you'd also notice this if you ran cabal v2-configure --extra-include-dirs=/usr/local/opt/openssl/include --extra-lib-dirs=/usr/local/opt/openssl/lib they'd be included at the top-level of the cabal.project.local file, i.e.

$ cat cabal.project.local
extra-include-dirs: /usr/local/opt/openssl/include
extra-lib-dirs: /usr/local/opt/openssl/lib

and the top-level properties in cabal.project have the semantics to apply to local packages only;

if you want something to apply to non-local packages you have to declare it in a package stanza; e.g. if you use the wildcard package stanza:

package *
  extra-include-dirs: /usr/local/opt/openssl/include
  extra-lib-dirs: /usr/local/opt/openssl/lib

it'd apply to all packages, local and non-local; but due to the Nix-style configuration tracking approach, this would also force every package to be recompiled, even if that include/lib-path would made no difference whatsoever to a package which doesn't even depend on hsopenssl.

Hence the recommendation is to write specific

package HsOpenSSL
  extra-include-dirs: /usr/local/opt/openssl/include
  extra-lib-dirs: /usr/local/opt/openssl/lib

as that would require only the least amount of cache-misses & recompilation, and redundant entries in your nix-style store.

However, I need to investigate what the situation is re having globally configured settings in ~/.cabal/config which would be supposed to act globally.

[emilypi]

@hvr This came up while installing cabal2nix, which requires hopenssl, not as a local package, but as a part of the cabal install process. If i can't install an executable using --extra-lib-dirs and --extra-include-dirs, I would consider this flag broken for installs.

[phadej]

The problem is that v2-build and v2-install flags have do the same things, otherwise one consider that inconsistency a bug.

When new-build was added, the flag interface wasn't thought out. I'm slightly in favor making flag settings global (so it's also uniform e.g. if you install outside or inside the project) etc. I'm sure some (e.g. Herbert) would disagree. But for cabal.project scenarios one can write stuff to cabal.project.local, in non project situations, as Emily points out, you cannot.

Yet. I invite anyone to look up in how the command line interface is built. It's somewhat tricky to see what options are there, and almost impossible in how they are wired up. :)

[phadej]

To clarify previous a bit: I'm afraid there is no active cabal contributors who understand how parsing of ~/.cabal/config, cabal.project and cli argument parsing are related (they do share fair a bit of code). Maybe making changes is easy, maybe not. Also quite hard to tell what changes what all things will affected.

I'll get to that eventually, but I'm simply postponing that because I'm afraid I'll lose productivity for a month or two trying to understand and (as my gut feeling says) try to untangle that code.

If no-one championed these issues for past 5 years, I don't think they are urgent.

[hvr]

The problem is that v2-build and v2-install flags have do the same things, otherwise one consider that inconsistency a bug.

I'd be worth articulating why exactly;

For one, cabal build strictly operates on local source-code, so it's biased towards local tweaks; cabal configure is strongly tied to cabal build; and many of the flags we pass to cabal build are designed to act local (e.g. cabal build -O0 applies -O0 only to local packages -- because its intent is to allow for fast edit/recompile/test cycles on your local source-code -- but your rationale would imply to make that global too and pay a significant inconvenience as you'd have to recompile your complete dependency tree into -O0 and possibly end up with humongous object files & executables...)

...but then we have cabal install which is a bit of a hybrid; there's two modes of operation here; either you're cabal installing from within a source-tree (and thus under a project scope), or you can be outside of a project/source-tree. And I'd argue that here depending on which of those two you're in, you want CLI flags to apply to different scopes -- just like the operation cabal install performs is highly context-sensitive to whether you're inside or outside a project.

So in the outside-a-project case of a cabal install, which I didn't realise at first when I read @emilypi comment (i.e. I was still in the cabal build mindset):

cabal install -z --extra-include-dirs=/usr/local/opt/openssl/include --extra-lib-dirs=/usr/local/opt/openssl/lib hopenssl

it's clearly understandable you'd want a way to inject some --extra-* flags globally to affect the compilation. Maybe we just need a 2nd set of flags --global-extra-* ... :-)

[goldfirere]

Thanks @jneira. I had not tried package *. And, really, I'd be quite happy for all packages to be affected by this setting, and so ~/.cabal/config really is the right place for it. But it was frustrating in the meanwhile to be saying --extra-include-dirs=/opt/homebrew/include and have a dependency not find its header in that directory. I understand the intent of having the setting apply only to the root package, but it's not what I would naively expect -- some user guidance here would be welcome.

[jneira]

I understand the intent of having the setting apply only to the root package, but it's not what I would naively expect -- some user guidance here would be welcome.

That is hard problem and ux is suffering from it. The cause imo is nobody agree about what should be the "default" behaviour when you dont specify any scope. That includes raw cli arguments and cabal.project top level fields. Some people thinks it should be applied to all packages and other ones will be surprised if they are applied to dependencies, depending of their use case at hand.

Otoh nobody wants to set explicitly everywhere the scope and remove non scoped arguments will break all the ecosystem really hard.

🤷

[mouse07410]

I say the default for the cli arguments should be "this package and whatever packages get built as dependencies for it".

cabal.project will not work for package/project that Cabal installs from Hackage (like cabal install cryptonite) - only for those you keep the source of on your disk.

[jneira]

@mouse07410 nice, i think it matches what are we discussing here: #7297 (comment)

[asivitz]

Do I understand this correctly? If I try to build zlib for my project like this

cabal v2-build --extra-lib-dirs="..." zlib

The option actually has no effect, and that's intended behavior?
If I have this correct- I'll write a PR to mention this (IMO very surprising) behavior.

[jneira]

well it is somewhat specified in docs: https://cabal.readthedocs.io/en/stable/cabal-project.html#package-configuration-options

They can be specified at the top-level, in which case they apply only to local package,

afaiu cli args are equivalent to top level options and maybe it should be stressed in docs

[asivitz]

Well this is the error message I get:

* Missing (or bad) C library: z
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.If the
library file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.

So it's suggesting a flag that doesn't work.

[jneira]

@asivitz ugh, thanks for pointing it out, the error message should be ammended too, replacing the suggestion with another one which works generically or noting flags will only work for local packages

[asivitz]

Yea, I'm actually not sure what I can do here. Do I pass --ghc-options="-L/some/path"? That doesn't work for me, but I could be hitting another problem.

I'm all mixed up b/c this also involves Nix, and something is being very stubborn about finding this library. But it's hard to debug b/c all of Nix, Cabal, and GHC seem potentially resistant to accepting new library paths. It seems to be passed along, but then when it finally reaches the CC call, it's dropped.

[jneira]

Does not the workaround mentioned in the issue first comment work for you?
It would be something like:

package zlib
  extra-lib-dirs: /path/to/extra-lib-dirs

[asivitz]

The main reason I can't go that route is that I'm trying to get a configuration that will work across multiple environments. So I wouldn't want to embed that configuration into something I'd check into source control. And also, the actual lib is dynamically created (through Nix). So, a command line argument is really what I need here, as opposed to a config value. (I guess I could dynamically generate the project file, but, I really don't want to do that.)

I'm currently trying a different approach in Nix. Although it seems like I should be able to do this, somehow, maybe I can avoid the problem entirely.

[gbaz]

You can add that configuration to cabal.project.local and hence not place it in what is put it in source control, fwiw.

I can't think of a good way to specify this in command line options. We would need a full design to have subpackage options stanzas specifiable on the command line, which feels pretty awkward compared to creating a file.

[jneira]

or maybe we could play with the explicit target, the counterintuitive thing arise when you do a cabal command my-package --whatever and such option is not applied to my-package

[asivitz]

That would certainly address my use case. It makes intuitive sense to me as well

[andreabedini]

Trying to summarise:

• The --extra-lib-dirs= command line argument, just like the extra-lib-dirs: cabal.project option, only applies to local packages and will not apply to dependencies.

• This is due to the fact that cabal includes fields like extra-lib-dirs: and extra-include-dirs: in the hash it uses to reuse already build dependencies. Applying that option globally will cause recompiling everything.

• In a project setting, there is a syntax to apply these options to any arbitrary package, whether local or not:

package [package name or * for all packages]
  extra-lib-dirs: ...

While in a project, both cabal build and cabal install will read and use those options. The options are now scoped in a project so any recompilation is intentional.

• If you are developing a project, it is best practice to add these paths to cabal.project.local and not include that into source version control, since those options are most likely specific to your system.

• Users of cabal install outside a project do not have a simple equivalent. One could argue that the use case of using cabal install as a non-project/global command was left out of the transition to v2/nix-style commands. This includes the use case of "installing" packages from Hackage, which was and seems to be still quite common.

• If you try to cabal install a Hackage package that needs a system library that cannot be found in the default search paths, you don't have a way to fix that with a command line option.

• My understanding is that adding/changing extra-lib-dirs in the global cabal config works (but be aware you will recompile all dependencies everytime you change it). Edit: I tested this and it does work.

• It seem impossible at the moment to use the above per-package syntax in the global cabal config. This is due to techinical debts that have not been solved (:heart: @jgotoh for working on Migrate cabal.project parsing to parsec #7748). I am under the impression this would give an good solution, since you would be able to scope the extra options to specific packages, avoding mass recompilations.

• The other solution is influencing gcc library search paths. In the end, all cabal does is passing extra-libraries to gcc, which has its own library search mechanism. In particular you can provide gcc with extra paths to look for libraries in, using the LIBRARY_PATH environment variable. The same applies with extra-include-path and C_INCLUDE_PATH, see gcc manual page.

• Notice this will be invisible to cabal's hashing mechanism, so changes in LIBRARY_PATH and C_INCLUDE_PATH will not trigger recompilation.

This all I know and can understand, please correct me if I made mistakes or missed something important.

[alt-romes]

I investigated and discussed this issue in yesterday's Cabal dev meeting.

As has been noted in this ticket's discussion, and summarised by @andreabedini and the commit message of 652a77c (this is a commit that took the wrong approach, but describes what packages do each flags apply to), flags given in the command line to build, such as --extra-lib-dirs and --extra-include-dirs, are local flags, just like flags at the top-level of the cabal.project file.
Local flags apply to local packages, such as those in your local project for which you have a .cabal file.

However, local flags do not apply to non-local packages (such as those pulled from hackage and installed in the store). Therefore, specifying --extra-lib-dirs in the command line will not apply those options to the packages you depend on such as HsOpenSSL. They are not ignored, they /only/ apply to the /local/ packages you are building!

To apply flags to a package you depend on, such as HsOpenSSL, you need to use either global configuration (~/.cabal/config) or per-package configuration, such as:

package HsOpenSSL
    extra-lib-dirs: ...
    extra-include-dirs: ...

or e.g. using an * instead of the package name to apply those options to all non-local dependencies of the project you are building.

In the meeting we agreed that, ultimately, the behaviour of local flags applying to local packages only is the correct one, and that we shouldn't change that behaviour.

Instead, we agree the solution to this ticket is to improve documentation and be very clear in cabal configure/build about the scope of configuration flags, add explicit examples of these use cases such as configuring the extra-lib-dirs of a specific package, and perhaps even add a small note to the documentation of --extra-lib-dirs saying something like "Note that this flag only applies to the packages local to the project, but not its dependencies. Read X documentation for more details."