[Go Modules]

Tags: #go #golang #modules #dependencies #vendoring

Go Modules.md

Go 1.11 introduced a new concept of ‘Modules’ which brings first class support for managing dependency versions and enabling reproducible builds.

A Module is a way to group together a set of packages and give it a version number to mark its existence (state) at a specific point in time.

Go Modules use Semantic Versioning for their numbering scheme.

Think of a ‘module’ as a project repository. Your project consists of many packages, and a module allows you to adequately group and version them.

New Project

Make a test project somewhere on your computer:

mkdir ~/code/test-project

Note: creating it outside of your $GOPATH demonstrates the process of using modules more clearly (in my case my $GOPATH is actually ~/code/go and so I’m putting this project just outside of that).

Here is our application, which uses one explicit non-standard library dependency (logrus):

package main

import (
	"fmt"

	"github.com/sirupsen/logrus"
)

func main() {
	logrus.Info("a log")
	fmt.Println("hello world")
}

Creating Modules

Let’s now create a new module:

$ go mod init my-test-project

go: creating new go.mod: module my-test-project

Note: my-test-project is an arbitrary value, but most people use similar structure to what they’re used to (e.g. github.com/your_name/your_project).

Let’s look at the go.mod module file that is created:

$ cat go.mod 

module my-test-project

Build

If we try and build our application, the go build command will recognise we’re inside of a module and parse any imports our code is referencing and download those into $GOPATH/pkg/mod

$ go build

go: finding github.com/sirupsen/logrus v1.2.0
go: downloading github.com/sirupsen/logrus v1.2.0
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/konsorten/go-windows-terminal-sequences v1.0.1
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/stretchr/testify v1.2.2
go: finding github.com/stretchr/objx v0.1.1
go: finding golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33
go: finding golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
go: downloading golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
go: downloading golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33

Dependency Location

If we check $GOPATH/pkg/mod we’ll see everything installed:

$ tree -L 3 $GOPATH/pkg/mod

/Users/markmcdonnell/code/go/pkg/mod
├── cache
│   ├── download
│   │   ├── github.com
│   │   └── golang.org
│   └── vcs
│       ├── 3b0ce44c45d17d58d917f0d7afa63ad02299664b3d7c8e3b3c6de03178872203
│       ├── 3b0ce44c45d17d58d917f0d7afa63ad02299664b3d7c8e3b3c6de03178872203.info
│       ├── 76a8992ccba6d77c6bcf031ff2b6d821cf232e4ad8d1f2362404fbd0a798d846
│       ├── 76a8992ccba6d77c6bcf031ff2b6d821cf232e4ad8d1f2362404fbd0a798d846.info
│       ├── 9950c06efbb2d90e85a58f1fbd6f3eb2db497b7c539a93fb5555656c5aba3c13
│       ├── 9950c06efbb2d90e85a58f1fbd6f3eb2db497b7c539a93fb5555656c5aba3c13.info
│       ├── b58cd1804573f08b6cfc86bbbad2960dd009cb14e98e8b74221958153f37a31b
│       ├── b58cd1804573f08b6cfc86bbbad2960dd009cb14e98e8b74221958153f37a31b.info
│       ├── b9a4b9bbdb4a59723f2348415ad7ffda91568455a1cfd92e97976132bdfbaf57
│       ├── b9a4b9bbdb4a59723f2348415ad7ffda91568455a1cfd92e97976132bdfbaf57.info
│       ├── ce05746539f15caa8470a1cb206cefcfc18421bcd2c6e35153546df051d6a96e
│       ├── ce05746539f15caa8470a1cb206cefcfc18421bcd2c6e35153546df051d6a96e.info
│       ├── de5fd3af413a4f3f969455ae522b4002fcb7bb4c158f339396dfc77710c9007d
│       ├── de5fd3af413a4f3f969455ae522b4002fcb7bb4c158f339396dfc77710c9007d.info
│       ├── ed2f58bca3966d01dc4666baa48276a4fab360938a8d941050d58e371e2bba77
│       └── ed2f58bca3966d01dc4666baa48276a4fab360938a8d941050d58e371e2bba77.info
├── github.com
│   └── sirupsen
│       └── logrus@v1.2.0
└── golang.org
    └── x
        ├── crypto@v0.0.0-20180904163835-0709b304e793
        └── sys@v0.0.0-20180905080454-ebe1bf3edb33

Checksum

If we look at the go.sum file generated we’ll see all the specific versions and hashed content:

$ cat go.sum

github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sirupsen/logrus v1.2.0 h1:juTguoYk5qI21pwyTXY3B3Y5cOTH3ZUyZCg1v/mihuo=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793 h1:u+LnwYTOOW7Ukr/fppxEb1Nwz0AtPflrblfvUudpo+I=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33 h1:I6FyU15t786LL7oL/hn43zqTuEGr4PN7F4XJ1p4E3Y8=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=

If you change versions of a dependency, then you’ll still see the hash from the previous version you used.

The reason for this is that if you revert back to the previous version you can check the hash hasn’t changed.

Why would you need to check the hashed content hasn’t changed, surely reverting to a previous ‘version’ worked before so it should work now right?

Well, someone could have forced pushed a change to a previous version and so the hash of the content will be different, and you’ll now be able to tell. It might be the change that was force pushed over the original ‘version’ you were using means the behaviour of that dependency breaks your application.

This is why the file has a .sum extension, because this is exactly what a checksum is.

Vendoring

You can have your project support older versions of Go that don’t support modules by also vendoring your dependencies:

$ go mod vendor

Which results in a vendor directory being generated:

$ tree -L 3 vendor/

vendor/
├── github.com
│   ├── konsorten
│   │   └── go-windows-terminal-sequences
│   └── sirupsen
│       └── logrus
├── golang.org
│   └── x
│       ├── crypto
│       └── sys
└── modules.txt

Why?

We can identify where a specific dependency is used:

$ go mod why -m github.com/sirupsen/logrus

# github.com/sirupsen/logrus
my-test-project
github.com/sirupsen/logrus

The why command (when used with -m) finds a path to any package in each of the modules.

Example Dockerfile

# =============================================================================
# build stage
# =============================================================================
FROM golang:1.11 AS golang

RUN go get golang.org/x/lint/golint

WORKDIR /app

COPY go.mod go.sum ./
ADD internal/lib ./internal/lib
RUN go mod download
COPY . .

# Lastly, build our actual application binary
RUN go build -ldflags="-s -w" -o /bin/app ./cmd/app

# =============================================================================
# app stage
# =============================================================================
FROM debian:stable-slim

RUN apt-get update && \
    apt-get install -y ca-certificates && \
    rm -rf /var/lib/apt/lists/*

COPY scripts/test /app/scripts/test
COPY --from=golang /bin/app /bin/

CMD ["/bin/app"]

Replacing Dependencies

// go.mod
module <your_new_rig_service>

require (
    buzzfeed/settings v0.0.0
)

replace (
    buzzfeed/settings => ./internal/lib/settings
)