doc: cmd/go: document use with modules "from the ground up"

[shaunc]

What version of Go are you using (go version)?

$ go version
go version go1.11.3 darwin/amd64

Does this issue reproduce with the latest release?

NA

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
NA

What did you do?

I am a go newbie trying to set up several folders with go code in a monorepo (which also contains code built in other languages). Some of these folders will contain services, others library utilities.

What did you expect to see?

I would like documentation based on the new "module" paradigm that explain:

• module
• package
• file
• folder

and their relationship to each other "from the ground up" (without assuming knowledge of how things used to work). In particular, in a monorepo,

• should there be an overall "go.mod", or one in each folder (or both??)
• if there is only one "go.mod" how do I specify different dependencies for each folder? If not,
  is there a way to specify common dependencies to force version synchronization (or if there isn't
  doc should flag as gotcha).
• how do I refer to code in one folder from another?

What did you see instead?

The tutorials don't cover modules. The official doc for modules are geared to explain what is different for experienced go users. There seems to be no good documentation anywhere on
use of modules in monorepos.

[ALTree]

Thanks for the request.

I took the liberty of removing the proposal label, since I don't think this is something that needs to be handled with the proposal process. You've just made a reasonable request for some new documentation, after all.

[bcmills]

CC @myitcv @thepudds

[thepudds]

@bcmills @myitcv

If we focus on someone new to Go, I think in a pre-module world these types of questions might be best answered by the introductory golang.org "How to Write Go Code" page? (There are also plenty of great introductory books, blogs, videos, etc., but that might be the best single document within the official documentation for these types of questions?)

There is an issue #28215 for updating that for modules, and I had seen that making progress, but looks like it might be stuck right now with a CLA issue. (I just added a comment there regarding that).

If we focus on someone new to Go, in this interim 1.11/1.12 period with modules defaulting to off, it might be best to have two documents:

1. If someone does not yet want to use modules: "How to Write Go Code" (the existing document)
2. If someone wants to use modules: "How to Write Go Code with Modules" (or some other title representing some new document that covers similar ground in a modules world)

The two documents could point to each other with 2-3 introductory sentences to help someone decide which one they want to read.

Or said another way, I would guess trying to explain both GOPATH and modules in a single document targeting someone new to Go would probably add more complexity than it is worth, and it seems premature (at least right now) to only have a modules-only version of "How to Write Go Code"?

I think some documents on golang.org are tied to the Go release cycle, but other documents are updated outside the release cycle. I'm not sure which category describes "How to Write Go Code".

In any event, to help fill in the vacuum a bit in the most immediate term, I guess I would propose:

• Create a new wiki page titled something like "How to Write Go Code (with Modules)" or something along those lines.
• Use the CL from website: Update "How to Write Go Code" with go modules #28215 as a starting point for populating it.
• Indicate along the top that it is a draft.
• Try to get some help from the community to get it into reasonable shape.
• Eventually, the content there gets deleted with a pointer to the 'real' "How to Write Go Code" whenever that is ready.

Alternatively, perhaps there is a plan to have the 'real' module documentation targeting new Go users popping out in the near-ish term, in which case an interim wiki page would not be not as useful.

I will say, though, there seems to be some hunger within the ecosystem for official modules documentation targeting new Go users, so maybe an interim wiki page could help bootstrap the material?

Does that sound reasonable? Or alternative suggestions?

[thepudds]

@shaunc

Regarding your questions above, here is a rough and non-comprehensive attempt to answer.

This is not an answer to your primary request for a "Modules from the ground up" targeting someone new to Go. That would be longer, and doesn't quite exist (at least not in an official form that is a polished as the current non-module introductory "How to Write Go Code" official page). Rather, this is quick attempt to answer your more specific sub-questions above.

First of all, for most projects it is best to at least start with one overall go.mod in your repository, and not put a go.mod file in each folder. It adds substantially more complexity to have multiple go.mod files in a given repository. If you have been working with a mono-repo, you likely can work with a single module.

A more detailed answer below (which I'm collapsing for now, so you need to click on "Longer Answer")...

There is much more that could be said, or said better, but my main question for you would be -- if you read that longer answer below, what are your next 3-4 most immediate questions (which might be re-asking one of your initial questions, or might be a follow-up question, etc.)?

Longer Answer (click here)

What is a module? What is the relationship between repositories, modules, and packages?

A module is a collection of related Go packages that are versioned together as a single unit.

Modules record precise dependency requirements and create reproducible builds.

Most often, a version control repository contains exactly one module defined in the repository root. (Multiple modules are supported in a single repository, but typically that would result in more work on an on-going basis than a single module per repository).

Summarizing the relationship between repositories, modules, and packages:

• A repository contains one or more Go modules.
• Each module contains one or more Go packages.
• Each package consists of one or more Go source files that all reside in a single directory.
• Source code declares its own package with a package foo statement.
• Source code imports code from another package via an import path supplied in an import statement such as import "github.com/repo/foo" (covered more below).

Modules must be semantically versioned according to semver in the form v(major).(minor).(patch), such as v0.1.0 or v1.2.3. If using Git, tag released commits with their versions.

How are modules arranged on disk?

A module is defined by a tree of Go source files with a go.mod file in the tree's root directory. (Module source code may be located outside of GOPATH, which was the pre-modules mechanism for organizing Go source code).

Most often, there is a single go.mod per repository located in the repository root, but go.mod files can be located elsewhere.

Here is an example go.mod file defining the module github.com/my/thing:

module github.com/my/thing

require (
    github.com/some/dependency v1.2.3
    github.com/another/dependency/v4 v4.0.0
)

A module declares its identity in its go.mod, which provides the module path. The import paths for all packages in a module share the module path as a common prefix. The module path and the relative path from the go.mod to a package's directory together determine a package's import path. For example, if you are creating a module for a repository github.com/my/repo that will contain two packages with import paths github.com/my/repo/foo and github.com/my/repo/bar, then the first line in your go.mod file typically would declare your module path as module github.com/my/repo, and the corresponding on-disk structure could be:

repo/
├── go.mod
├── bar
│   └── bar.go
└── foo
    └── foo.go

How are packages imported?

In Go, code in another package needs to be imported via an import statement in order to be used. This is true before modules, and this continues to be true with modules.

With modules, packages are imported using the full path including the module path. For example, if a module declared its identity in its go.mod as module example.com/my/module, a consumer could use the import statement:

import "example.com/my/module/mypkg"

This imports package mypkg from the module example.com/my/module.

How do I specify different dependencies for different packages within the same module?

This is a more nuanced question, so this is a rougher answer (which might miss some case, but hopefully at least conveys the gist)...

If you have a single go.mod with multiple packages, that go.mod will track the dependencies across all packages, which usually works well in practice.

If we take the example where package foo and bar reside in a single module, it is not a problem for example if:

• foo and bar both require the same v1+ major version of some dependency X (including given X should be compatible within a v1+ major version, according to semver)
• foo and bar require different major versions of dependency Y (including given Y can be separately tracked within a single go.mod as require Y/v2 and require Y/v3)

It would however be a problem if:

• package foo requires a v0 version of some dependency Z that is incompatible with the v0 or v1 version of Z required by bar (because at most one v0/v1 version of dependency Z would be tracked within a single go.mod),
• if foo and bar require different incompatible versions of a shared dependency that does not follow https://semver.org. (semver is a requirement for any code opting in to modules, but pre-existing code will not have always followed semver).

[shaunc]

@thepudds -- Thanks!

In the intervening days I more or less figured this out, but its great to have a description in one place to refer to.

My initial confusion, not related to my question, as it is covered in the existing doc, was not understanding the tight coupling between packages and folders. (I thought of packages something like C++ namespaces -- in theory orthogonal to file layout. Perhaps I was led in this direction
by the fact that applications have a "main" package that is not named for the folder. If I have a
library with an executable utility script, I reasoned, one could have both "main" and "foo" packages
in same directory, so folders were an organizational convenience.... Anyway I sorted that out but existing documents could perhaps be clearer about package/folder and "main" as exception.)

The monorepo currently is a mix of typescript and python, with a bit of cython. In fact this initial experiment is to replace a cython-backed service with go, as cython is hard to maintain.

One thing I like about the node/typescript tooling, which I understand you don't (yet?) have, is that you can have one overall "package.json" for the monorepo as a whole then individual ones for subprojects, and require() looks up the tree for dependencies. Thus you can mostly maintain one overall set of dependencies, saving work and gnashing of teeth because of incompatibilities, while still being able to override when you need to. (NB -- don't know if support for this is on the horizon; it could be alternately with an "extends" directive in go.mod if you don't like implicit dependency on filesystem hierarchy.)

Anyway -- thanks to you and the go community!

[bcmills]

We now have quite a bit of module documentation at https://go.dev/doc.

I'm sure it could be further improved and clarified, but I would rather we file more targeted individual issues for the remaining improvements.