cmd/go: go get should not add a dependency to go.mod

[natefinch]

go get is not the right command for adding a dependency to the current module. We have a command for dealing with modules, and it's go mod, that's the command that should be used to add dependencies.

This is the first paragraph of help on go get:

Get downloads the packages named by the import paths, along with their dependencies. It then installs the named packages, like 'go install'.

None of that says "add the library you specified to the dependencies of the current project". And why would it? it's for getting code from somewhere else and bringing it to this machine.

Notably, go get will even add go commands as a dependency... so like if you run go get golang.org/x/tools/cmd/goimports and your current directory happens to be a go module, guess what? Your go.mod now has

golang.org/x/tools v0.0.0-20180911133044-677d2ff680c1 // indirect

This UX is extremely confusing for the ~2 million people who used go before modules. It's also way too implicit and magic regardless of the experience. Adding a dependency to your project should be an explicit action:

go mod add golang.org/x/tools/cmd/goimports  

What did you do?

from inside a module directory

$ ls
go.mod
$ go get github.com/natefinch/lumberjack
go: finding github.com/natefinch/lumberjack latest
go: downloading github.com/natefinch/lumberjack v0.0.0-20180817145747-7d6a1875575e
~/dev/test$ more go.mod
module app

require github.com/natefinch/lumberjack v0.0.0-20180817145747-7d6a1875575e // indirect

What did you expect to see?

No change to my go.mod

What did you see instead?

go get added a library to my go.mod file.

System details

go version go1.11 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/finchnat/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/finchnat"
GOPROXY=""
GORACE=""
GOROOT="/Users/finchnat/sdk/go1.11"
GOTMPDIR=""
GOTOOLDIR="/Users/finchnat/sdk/go1.11/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
GOROOT/bin/go version: go version go1.11 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.11
uname -v: Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.13.6
BuildVersion:	17G65
lldb --version: lldb-902.0.79.7
  Swift-4.1

[myitcv]

From the bottom of go help get:

...

This text describes the behavior of get when using GOPATH
to manage source code and dependencies.
If instead the go command is running in module-aware mode,
the details of get's flags and effects change, as does 'go help get'.
See 'go help modules' and 'go help module-get'.

From go help module-get:

The 'go get' command changes behavior depending on whether the
go command is running in module-aware mode or legacy GOPATH mode.
This help text, accessible as 'go help module-get' even in legacy GOPATH mode,
describes 'go get' as it operates in module-aware mode.

...

[myitcv]

@natefinch I think the documentation on this covers the points you raise, but please do say if you think it could be better signposted.

cc @bcmills

[natefinch]

It's not about the documentation. It's a complete change to what go get has meant for the last 6+ years. Which has meant "download and compile some code so I have it locally". That's still a useful command to have in the universe of go modules.

Changing what an established command does, in such an important way, is a really bad idea.

I think it's pretty clear, if I go get golang.org/x/tools/cmd/goimports ... what I mean is "download and compile goimports, and put it somewhere for general use". The fact that this command also stealthfully adds golang.org/x/tools/ to my current directory's dependencies is surprising and extremely unwelcome.

Like I said in the bug report... we have a command for dealing with modules. Why would we not just add this functionality there? go mod add github.com/... is perfectly clear, and wouldn't clash with current usage.

[komuw]

if I go get golang.org/x/tools/cmd/goimports ... what I mean is "download and compile goimports, and put it somewhere for general use".

I think this is; #25922 and #24250

[cespare]

I agree with this. I've already been surprised three separate times now by running go get and having tools unexpectedly added to my go.mod, or having a new go.mod file dropped into a directory where I didn't intend.

Before modules, go get was used for two pretty distinct use cases: (1) downloading tools and (2) downloading libraries. It seems much, much less surprising and user-friendly for post-modules go get to specialize on use case (1) and leave (2) for other tools. In my experience with modules so far, I've handled (2) by simply adding new imports and running go test/go build.

[myitcv]

I think it's pretty clear, if I go get golang.org/x/tools/cmd/goimports ... what I mean is "download and compile goimports, and put it somewhere for general use".

In a GOPATH world I would agree. But we're talking a modules world here. From my perspective I'm fully anticipating any go command to do something with my go.mod.

Put another way (and this is where #24250 comes in), I would expect, in a modules world, to have to do something "special" in order to performa a "global" install (by definition outside the context of my go.mod)

[vdemario]

I was also surprised by this behavior. The suggestion that go get will have a different meaning in a "modules world" doesn't solve the use case of go get golang.org/x/tools/cmd/goimports or installing any other tool. Do I need to change directories every time I need to install a new tool? What if a go.mod is created anywhere I am because in the modules world any place is a valid place for a Go project?

Regardless of these examples, what bothers me the most is that we're using the same command that has existed for several years to do a different thing. Are we all supposed to accept that every tutorial, every doc, everything that has ever been said about go get is not valid anymore?

The global place for installing any library or tool still exists, so why do we need to do something special to get back the behavior that already exists?

[kaedys]

Have to chime in that that I agree with @natefinch and @vdemario here. Using go get for installing tools has been the standard practice since go get was created. Changing that semantic is extremely problematic from a usability perspective, especially since the Go tooling currently lacks a command equivalent to "download this Go library, compile it, and install it somewhere I can use, but do not add it to my current project".

We're adding entirely new behavior via the go mod command. It seems to make the most sense to have explicitly adding a dependency to be a subset of that command, rather than hijacking an existing command and completely obliterating one of the two most common ways that command is currently used.

[vdemario]

Also, apparently trying to install a tool outside a module doesn't even work:

/tmp
$ set -gx GO111MODULE on # fish shell syntax
/tmp
$ go get golang.org/x/tools/cmd/goimports
go: cannot find main module; see 'go help modules'

[myitcv]

@vdemario please see, as linked above, #24250

[kaedys]

That issue clarifies this one even more. Prior to Go 1.11, the semantics for go get were identical to go install, except that it pulled the code from a remote repository before running go install on them. The module functionality in Go 1.11 has completely abandoned the go install piece (actually, it's still there, but it's mostly an anachronism at this point) because go get has been hijacked for adding a dependency to a module.

And that's a bit of a problem, since go get was validly used just as commonly for the go install functionality as it was for the abstraction of git clone. Abandoning the install functionality is extremely jarring and leads to some very unexpected side-effects, like the necessity of running go mod tidy after installing a tool that's not also a dependency, and as @vdemario just pointed out, the complete inability to install tools unless you're currently in a module (which you then need to tidy).

[myitcv]

@kaedys

The module functionality in Go 1.11 has completely abandoned the go install piece (actually, it's still there, but it's mostly an anachronism at this point) because go get has been hijacked for adding a dependency to a module.

And that's a bit of a problem, since go get was validly used just as commonly for the go install functionality as it was for the abstraction of git clone. Abandoning the install functionality is extremely jarring and leads to some very unexpected side-effects, like the necessity of running go mod tidy after installing a tool that's not also a dependency

go get does still perform an install. Given:

export GOPATH=$(mktemp -d)
cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello
export GOBIN=$PWD/.bin
go get golang.org/x/tools/cmd/stringer

Then:

$ ls $GOBIN
stringer

like the necessity of running go mod tidy after installing a tool that's not also a dependency, and as @vdemario just pointed out, the complete inability to install tools unless you're currently in a module (which you then need to tidy).

This falls under the umbrella of #24250.

As @komuw linked above, for discussion on the current "best practice" for defining tool dependencies, please see #25922

[kaedys]

But it's not about tool dependencies. It's about tools you want to install on a global level, unrelated to the current module. That's the issue at the root of this, that under the Go module framework, go get is inherently and unavoidably linked to the module you are currently operating on (and completely fails if you're not on a module). There's no reason to abandon the ability to install tools unrelated to the current module (or outside the context of a module), and there's no reason to assume that every package retrieved is necessarily a dependency of the current module, especially given the pre-1.11 semantics and usage of go get.

[myitcv]

It's about tools you want to install on a global level

Indeed, that's why I linked to #24250.

Just to emphasise what @rsc said in #24250 (comment):

$ vgo get github.com/golang/dep/cmd/[email protected]
cannot determine module root; please create a go.mod file there

This clearly must work eventually.

(we can now substitute go get for vgo get)

So there is agreement that this "global" concept indeed very much has merit. It's just that this didn't make Go 1.11.

There's no reason to abandon the ability to install tools unrelated to the current module (or outside the context of a module), and there's no reason to assume that every package retrieved is necessarily a dependency of the current module, especially given the pre-1.11 semantics and usage of go get.

Hopefully the link above covers this point?

[natefinch]

Why aren't there go mod add and go mod update commands? That would seem to be the obvious place to go looking for the functionality of adding or updating a dependency. Then there would be no need to overload go get for adding and updating dependencies.

[bcmills]

@natefinch

We have a command for dealing with modules, and it's go mod, that's the command that should be used to add dependencies.

You are mistaken. The go mod command is for module-specific operations, not all operations “dealing with modules”. As its documentation says, “support for modules is built into all the go commands, not just go mod. For example, day-to-day adding, removing, upgrading, and downgrading of dependencies should be done using go get.”

go get in GOPATH mode already performs a grab-bag of tasks: it “downloads the packages named by the import paths, along with their dependencies. It then installs the named packages, like go install.” go get has the same behavior in module mode, except that it downloads entire modules and it does not yet work outside of a module (that's #24250).

In my opinion, it is unfortunate that go get conflates downloading packages with installing binaries, but we're essentially stuck with it: we need to maintain compatibility with existing, documented command lines, and existing command lines don't only use go get to install tools — they also use it to download package dependencies.

[cespare]

@bcmills

[...] existing command lines don't only use go get to install tools — they also use it to download package dependencies.

This is kind of a hobbyist workflow, I suppose, but most serious Go users are not using go get to get package dependencies; they are using vendoring, dep, or some other solution.

OTOH I suspect that many or most Go users make use of the "fetch/update a Go tool" functionality of go get.

[natefinch]

existing command lines don't only use go get to install tools — they also use it to download package dependencies.

....sure. I'm not sure what that has to do with go get foo/bar/baz modifying the local go.mod?

go get foo/bar/baz explicitly tells the go tool "ignore the directory that I'm in. Instead, find this package, make sure it and its dependencies are downloaded locally, build it, and install it."

There's no reason why it couldn't do the exact same thing in the go modules world. You're explicitly telling the go tool not to build the local directory.

I agree that go build (implying go build .) should build the local directory, and update go.mod etc. That's fine. You're explicitly telling it to work on the local directory.

When you give it a package path, though, you're explicitly telling it not to work on the local directory. That's the disconnect.

[vdemario]

I agree that go getin GOPATH mode performs a grab-bag of tasks and I also agree that it is unfortunate that go get conflates downloading packages with installing binaries.

Considering that, why does the module-aware go get also gets the new responsibilities of adding, removing, upgrading, and downgrading of dependencies?

It seems counter-intuitive that a command that already does too much is getting new responsibilities and side effects (such as updating go.mod).

[bcmills]

go get foo/bar/baz explicitly tells the go tool "ignore the directory that I'm in. Instead, find this package, make sure it and its dependencies are downloaded locally, build it, and install it."

But go get -u foo/bar/baz upgrades arbitrarily many dependencies: it may affect the current directory, depending on what else is involved.

As weird as it is for go get of a tool to update the current module but install the tool globally, it seems even weirder for go get -u not to upgrade dependencies depending on the mode, or for go get -u to affect the module where go get would not.

[bcmills]

It seems counter-intuitive that a command that already does too much is getting new responsibilities and side effects (such as updating go.mod).

Every command that loads packages updates go.mod, and go get -u already adds and upgrades dependencies even in GOPATH mode. We're not adding new responsibilities to go get so much as to go globally.

(We did add one new responsibility to go get: it can now downgrade dependencies, whereas the go command couldn't downgrade dependencies at all before.)

[natefinch]

it seems even weirder for go get -u not to upgrade dependencies, or for go get -u to affect the module where go get would not

I feel like this is a disconnect between people who use a single gopath for everything (i.e. google) and people who exclusively use vendored dependencies.

Every serious public project for a long time has used vendored dependencies. go get -u would never update any of these dependencies, because they all use vendoring.

[natefinch]

IMO go get -u <some library> just doesn't really make any sense in the modules world. It's a holdover from gopath. It works for binaries because it updates the binary in gobin.

I'd really prefer if the commands to update dependencies were explicit and obvious named. go update foo/bar or go add foo/bar.

Why hang on to this fairly vague word "get" when we're talking about actions that never existed before? go get has baggage that we can leave behind.

[bcmills]

I feel like this is a disconnect between people who use a single gopath for everything (i.e. google) and people who exclusively use vendored dependencies.

Most Google-internal users pretty much aren't using GOPATH (or the go tool) at all. This isn't a Google-vs.-outside issue so much as a hobbyist-vs.-enterprise one.

Every serious public project for a long time has used vendored dependencies. go get -u would never update any of these dependencies, because they all use vendoring.

Yes, we clearly need to work on the migration path for vendoring. That's one of my top priorities for 1.12.

[bcmills]

I completely agree that get is a weird holdover from the pre-modules world, but I don't think we can leave that baggage behind. Go takes a very serious approach to backward-compatibility, and breaking install instructions that use go get -u would be a pretty big incompatibility.

That said, maybe it's worth leaving the get baggage behind when we're talking about installing tools: for example, go install as it stands is very nearly useless in module mode. Perhaps go install pkg@version should install the named version of the tool without updating the local go.mod, and then anyone who wants to update their instructions can switch over to that.

[natefinch]

go get -u as instructions to install a library have been invalid ever since the widespread use of the vendor directory. I'd be interested to see how many widely used libraries actually put that in their readme.

[natefinch]

This isn't a Google-vs.-outside issue so much as a hobbyist-vs.-enterprise one.

Sorry! I almost deleted the google bit there. definitely was not trying to make this a google vs. the outside world.

[leighmcculloch]

I've been using something very similar to install Go binaries from source instead of using go get. From the examples posted it looks like we're all doing similar things to compensate for go gets behavior. https://github.com/leighmcculloch/gobin

[myitcv]

I support what Russ/Bryan are doing with the go tool and the approach they are taking (I am the author of https://github.com/myitcv/gobin). They have many different aspects to consider/balance, from supporting previous behaviour of the go tool, to not bloating the go tool or its sub commands, etc.

#27653 (comment) is a good summary of how that balance is currently struck.

None of this precludes experiments in other tools, and https://github.com/myitcv/gobin is one such experiment. It may live on, it may die... any number of outcomes. And most importantly none of this precludes the go tool gaining extra capabilities in the future, based on these experiments.

@leighmcculloch - hopefully the usage docs for github.com/myitcv/gobin help to show what that tool is capable of.

[mvdan]

Those of you who followed this thread might be interested in #30515, to add a "global install" command of some sort.

[dcarbone]

Another point of concern here is the unexpected removal of values from go.mod.

I am building a WAMP server that utilizes https://github.com/gammazero/nexus, which in turn relies in https://github.com/ugorji/go. ugorji's module is atypical in that there is no code nor mod file in the root of the repo, rather everything is under /codec.

ugorji pushed a breaking change to their master branch in march 2019. The version nexus pins in its own checked-in vendor dir is from 2018. The ugorji repo does have tagged versions, but it is impossible to actually USE them with go modules, it seems. The code is under /codec, as is the mod file, but when attempting to pin v1.1.2 (the latest tag as of this post) via go get github.com/ugorji/[email protected], this entry is seemingly entirely ignored when executing go build, go mod vendor, or go mod tidy. In all cases, the latest version of master is added to the sum file, subsequently used, and then even more subsequently breaks everything and causes me to see red.

The only solution I have found to this extremely idiotic issue is to do the following:

replace (
        github.com/ugorji/go/codec v0.0.0-20190204201341-e444a5086c43 => github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8
	github.com/ugorji/go/codec v0.0.0-20190309163734-c4a1c341dc93 => github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8
	github.com/ugorji/go/codec v0.0.0-20190315113641-a70535d8491c => github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8
	github.com/ugorji/go/codec v0.0.0-20190316192920-e2bddce071ad => github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8
)

require (
        github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8 // indirect
)

This works as I basically take every version since the last tag (v1.1.2) of the repo and replace it the tagged version.

However. When any command dealing with deps is run (get, vendor, tidy) the entry from the require block is removed for me, which then causes builds to fail with this exceptional error: build github.com/ugorji/go/codec: cannot find module for path github.com/ugorji/go/codec

I fully appreciate that I am stupid, am probably doing it wrong, etc., however it is still annoying. I would greatly appreciate a "no cleanup" or "no edit" option or something, so I am not bit by build errors at midnight because I forgot to re-un-edit my program's mod file.

[bcmills]

@dcarbone

When any command dealing with deps is run (get, vendor, tidy) the entry from the require block is removed for me, which then causes builds to fail with this exceptional error: build github.com/ugorji/go/codec: cannot find module for path github.com/ugorji/go/codec

go mod vendor and go mod tidy should never change your configuration from one in which everything builds to one in which something does not.

If you observe otherwise, please file a separate issue with steps to reproduce the error — this issue is closed, and closed issues are not tracked or triaged.