cmd/go: module: go mod vendor does not copy test source files

[brenol]

What version of Go are you using (go version)?

$ go version
1.13

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/brenol/.cache/go-build"
GOENV="/home/brenol/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY="bitbucket.org"
GONOSUMDB="bitbucket.org"
GOOS="linux"
GOPATH="/home/brenol/goworkspace"
GOPRIVATE="bitbucket.org"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/brenol/goworkspace/src/github.com/twitchscience/kinsumer/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build321165660=/tmp/go-build -gno-record-gcc-switches"

What did you do?

git clone github.com/twitchscience/kinsumer
go mod init
go mod vendor

What did you expect to see?

go mod vendor checkout the code that is being used by the go.mod file (in the correct version/tag)

What did you see instead?

go mod vendor does not checkout the code that is being used.

[thepudds]

Would you be able to give a couple examples about which modules/version do not match your expectations? For example, perhaps you expect that foo should be at v1.2.3, but it appears that v1.5.0 of foo was vendored, or something like that?

Also, go list -m all is the definitive way to see the final versions of modules that will be used in a build. What does that show, and does it seem to agree with what was vendored?

[brenol]

sure.

I expected the module github.com/twinj/uuid which was in master after running go mod vendor branch to point to its v1.0.0 tag (inside the vendor directory).

This would greatly help while debugging as I would be able to see that the source code in it was importing github.com/myesui/uuuid package.

However, after doing go list -m all and I don't see the offending package that made me open this issue: gopkg.in/stretchr/testify.v1 which would mean that mod is resolving gopkg.in/stretchr/testify.v1 to github.com/stretchr/testify and is complaining about the original package;, even though it's not listed in the go list -m all.

$ go list -m all
github.com/twitchscience/kinsumer
github.com/aws/aws-sdk-go v1.24.3
github.com/cactus/go-statsd-client/statsd v0.0.0-20190906215803-47b6058c80f5
github.com/davecgh/go-spew v1.1.0
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
github.com/myesui/uuid v1.0.0
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.4.0
github.com/twinj/uuid v1.0.0
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
golang.org/x/net v0.0.0-20190918130420-a8b05e9114ab
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a
golang.org/x/text v0.3.0
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
gopkg.in/yaml.v2 v2.2.2

[bcmills]

Note that go mod vendor does not check out the code in question. It copies over only the subset of files that are relevant to go build (not a complete VCS checkout).

[bcmills]

I would be able to see that the source code in it was importing [github.com/myesui/uuid] package.

go mod vendor intentionally does not copy in the test-only source files or dependencies of vendored packages.

go mod why and go mod graph — not go mod vendor — are the general tools for understanding module dependencies. In this case, fixing the gopkg.in/stretchr/testify.v1 issue (by requesting a compatible version explicitly) enables go mod why -m github.com/myesui/uuid to successfully report the reason for the dependency, which is indeed via a test of a dependency:

kinsumer$ go1.13 mod init github.com/tritchscienc/kinsumer
go: creating new go.mod: module github.com/tritchscienc/kinsumer

kinsumer$ go1.13 get gopkg.in/stretchr/[email protected]
go: finding gopkg.in v1.2.2
go: finding gopkg.in/stretchr/testify.v1 v1.2.2
go: downloading gopkg.in/stretchr/testify.v1 v1.2.2
go: extracting gopkg.in/stretchr/testify.v1 v1.2.2
go: finding github.com/stretchr/testify v1.4.0
go: downloading github.com/stretchr/testify v1.4.0
go: extracting github.com/stretchr/testify v1.4.0
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/stretchr/objx v0.1.0
go: downloading github.com/davecgh/go-spew v1.1.0
go: downloading gopkg.in/yaml.v2 v2.2.2
go: extracting github.com/pmezard/go-difflib v1.0.0
go: extracting github.com/stretchr/objx v0.1.0
go: extracting github.com/davecgh/go-spew v1.1.0
go: extracting gopkg.in/yaml.v2 v2.2.2
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/davecgh/go-spew v1.1.0
go: finding github.com/stretchr/objx v0.1.0
go: finding gopkg.in/yaml.v2 v2.2.2

kinsumer$ go1.13 mod tidy
go: finding github.com/cactus/go-statsd-client/statsd latest
go: finding github.com/cactus/go-statsd-client v3.2.0+incompatible
go: finding github.com/twitchscience/kinsumer latest
go: finding github.com/aws/aws-sdk-go v1.24.3
go: downloading github.com/cactus/go-statsd-client/statsd v0.0.0-20190922113730-52b467de415c
go: downloading github.com/cactus/go-statsd-client v3.2.0+incompatible
go: downloading github.com/twitchscience/kinsumer v0.0.0-20190917015803-a7e05fb3da3b
go: downloading github.com/aws/aws-sdk-go v1.24.3
go: extracting github.com/twitchscience/kinsumer v0.0.0-20190917015803-a7e05fb3da3b
go: extracting github.com/cactus/go-statsd-client v3.2.0+incompatible
go: finding golang.org/x/sync latest
go: downloading golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e
go: extracting github.com/cactus/go-statsd-client/statsd v0.0.0-20190922113730-52b467de415c
go: extracting golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e
go: finding github.com/twinj/uuid v1.0.0
go: downloading github.com/twinj/uuid v1.0.0
go: extracting github.com/twinj/uuid v1.0.0
go: downloading gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: extracting gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: extracting github.com/aws/aws-sdk-go v1.24.3
go: downloading github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
go: finding github.com/myesui/uuid v1.0.0
go: finding golang.org/x/net latest
go: downloading golang.org/x/net v0.0.0-20190921015927-1a5e07d1ff72
go: downloading github.com/myesui/uuid v1.0.0
go: extracting github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
go: extracting github.com/myesui/uuid v1.0.0
go: extracting golang.org/x/net v0.0.0-20190921015927-1a5e07d1ff72
go: downloading golang.org/x/text v0.3.0
go: extracting golang.org/x/text v0.3.0

kinsumer$ go1.13 mod why -m github.com/myesui/uuid
# github.com/myesui/uuid
github.com/tritchscienc/kinsumer
github.com/twinj/uuid
github.com/twinj/uuid.test
github.com/myesui/uuid

kinsumer$

[bcmills]

go mod vendor seems to be working as designed here — there certainly is a diagnostic problem, but I believe it will be better addressed in go mod why with #26977.