Repository: go
Issue: cmd/go: support easy way to install a remote binary while respecting remote 'replace' and 'exclude' directives
Labels: NeedsDecision,early-in-cycle,modules

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.12.1 linux/amd64
</pre>

### Does this issue reproduce with the latest release?
Yes, including tip.

### What did you do?

```
cd /tmp
go get foo
```
where:
* `foo` is a module with `replace` directives chosen by the module author, and
* `foo` is a binary command, and
*  I want `foo` to build as specified by the module author

### What did you expect to see?

The `replace` directives in the `go.mod` for `foo` respected in my resulting `foo` binary.

### What did you see instead?

The `replace` directives in the `go.mod` for `foo` are currently ignored.

### Summary

The suggestion in this issue is to provide some easy way to install a remote binary while respecting `replace` and `exclude` directives in the remote module. 

This seems necessary based on observed usage, as well as this seems part of the promise from the proposal and elsewhere that _"a module author is in complete control of that module's build when it is the main program being built"_. That promise of control by the module author seems especially important when the author is publishing a binary for use by others.

### Background

As far I as I was able to follow the design discussions:
* By design, modules are a less expressive system than a more traditional approach, and
* It seems the complete control given to the top-level build via `replace` and `exclude` directives was important aspect of balancing out that reduction in expressiveness.

For example, the ["Build Control"](https://go.googlesource.com/proposal/+/master/design/24301-versioned-go.md#build-control) section in the official proposal has a discussion about this, including (with emphasis added):

> Minimal version selection gives the top-level module in the build additional control, allowing it to exclude specific module versions or replace others with different code, but those exclusions and replacements only apply when found in the top-level module, not when the module is a dependency in a larger build.
> 
> _A module author is therefore in complete control of that module's build when it is the main program being built_, but not in complete control of other users' builds that depend on the module. _I believe this distinction will make this proposal scale to much larger, more distributed code bases than the Bundler/Cargo/Dep approach_.

There are similar sentiments expressed in the initial vgo blog series, [such as](https://research.swtch.com/vgo-mvs):

> Minimal version selection is very simple. It achieves simplicity by
> eliminating all flexibility about what the answer must be: the build
> list is exactly the versions specified in the requirements. _A real system
> needs more flexibility, for example the ability to exclude certain module
> versions or replace others_.

Those arguments seem reasonably compelling. It seems, however, the current Go modules system does not quite deliver on that promise when, for example:
* The author of module `foo` decided to use a `replace` or `exclude`, and
* The module `foo` is _"the main program being built"_ via a `go get foo` executed outside another module.

In that scenario, the current modules system ignores the remote `replace` or `exclude` in `foo`. In other words, that scenario seems to illustrate not delivering on the promise of "_A module author is therefore in complete control of that module's build when it is the main program being built_". 

### "Don't use replace" as an alternative solution

When the concern in this issue has been raised in the past, sometimes the response has been something like "People shouldn't really use `replace` when releasing a module". However, I think that falls short as a solution.

@rogpeppe has stated for example that `juju` cannot currently be built without `replace` directives.

In general, one could imagine that the need for `replace` directives going down over time as the ecosystem adapts modules and semver more faithfully, but it is hard to imagine the need for need for `replace` directives going so low that the need for `replace` could be approximated by zero. For example:
* Use of semver is never perfect, especially in the face of human error.
* Changes can [break](http://www.hyrumslaw.com/) a consumer without changing the statically checkable API.
* `v0` is a "compatibility free zone", yet people in practice and by necessity still use `v0` dependencies. MVS can deliver incompatible results in the face of multiple `require` directives for a `v0` dependency.
* Etc.

I think the on-going need for `replace` is especially true given the purposeful reduction in expressivity elsewhere in the modules system.

### "git clone" as an alternative solution

In discussions on this topic, a response is sometimes made along the lines of "If an author of a binary needs to use `replace` directives, they can always just update their readme to ask users to not do `go get` or `go install` and instead do a `git clone` followed by `go install`". A readme in theory could also include `git clone --branch`. However, I think a `git clone` solution falls short given:

* The future benefits of GOPROXY mirrors and notaries, etc.
* The benefits of the `go` command automatically picking a good semver tag for you, with a default of `@latest` (e.g., the semver-aware logic described in ["Module aware go get"](https://golang.org/cmd/go/#hdr-Module_aware_go_get)).
* The need to update the readme over time (e.g., changing the recommended semver tag if the readme supplies a specific recommended semver tag to use with `git`, or changing the readme if it normally specifies a more standard `go get` variation but then is only temporarily updated to specify using `git clone` in order to respect a `replace` directive for a few months while waiting for resolution of an upstream problem, etc.).
* `go get some/cool/cmd` has proven to be popular within the Go community, including as a concrete "gateway" to Go for people who are not yet developing Go themselves.
* `go get` nicely hides most VCS differences.

Hugo is an example of an early modules adopter that currently has a `replace` directive in its `go.mod`. It has the following [installation instructions](https://github.com/gohugoio/hugo):

> Since Hugo 0.48, Hugo uses the Go Modules support built into Go 1.11 to build. The easiest is to clone Hugo in a directory outside of GOPATH, as in the following example:
> 
> ```
> mkdir $HOME/src
> cd $HOME/src
> git clone https://github.com/gohugoio/hugo.git
> cd hugo
> go install
> ```

That might be the right choice for Hugo based on its needs and the current state of modules in Go 1.12, but those instructions for example ignore the benefit of semver tags.

Personally, I would view it as a step backwards if something along those lines became the recommended solution if you have a `replace` directive.

### Other possible solutions

There are likely many possible solutions, but here are three sample solutions to help start conversation here.

Under all three of these options, it could be reasonable reject any filesystem-based replace directives in a remote go.mod that reach outside the module (as suggested by Bryan in https://github.com/golang/go/issues/24250#issuecomment-419098182). To help avoid placing a testing burden on authors to check for that, `go release` could warn if that condition is found, which is likely a good thing to do anyway as suggested in the `go release` issue in https://github.com/golang/go/issues/26420#issuecomment-471701258.

#### Option 1

If #30515 is adopted with a new `-b` flag (for binary or bare), and `go get -b foo` ends up meaning "install foo while ignoring any current module's `go.mod` and without altering the current module's `go.mod`", it could be natural for `go get -b foo` to also respect any `replace` or `exclude` directives in `foo`. The rationale could be that there is no other top-level module being considered aside from `foo` when `-b` is specified.  

The same could apply if a different spelling than `-b` is selected for #30515 (e.g., perhaps #30515 is resolved via a `go get -global`, `go get -clone`, `go install`, etc.).

#### Option 2

If #30515 is not adopted, then `go get foo` when run outside of a module could be redefined in 1.13 to respect any `replace` or `exclude` directives in `foo`. The rationale could be that there is no other top-level module being considered aside from `foo` when doing `go get foo` outside of a module.  

This behavior was suggested by multiple people for Go 1.12 (including by Bryan in https://github.com/golang/go/issues/24250#issuecomment-419098182), but the ultimate Go 1.12 behavior was different, including Bryan commented in [CL 148517](https://go-review.googlesource.com/c/go/+/148517) that his 1.12 change was "as minimal a change as I could comfortably make to enable 'go get' outside of a module for 1.12".  (Part of the context for Bryan's comment in the CL is I think that minimal change might have been implemented after the 1.12 freeze).

#### Option 3

The behavior in Option 1 and Option 2 could both be the 1.13 behavior. In other words:

* `go get foo` when run outside of a module in 1.13 would now respect any `replace` or `exclude` directives in `foo` (in addition to respecting the remote module's `require` directives and not changing any local `go.mod`).
* `go get -b foo` in general means "run as if in a directory outside any module". (Under this definition, it would imply `replace` or `exclude` directives in a remote `foo` are respected).

### Relationship to other issues

This has been discussed in multiple issue such as #27643, #24250, #30515 and others, but usually as a side topic. Perhaps this issue here can be closed as a duplicate of #30515, but @mvdan, the author of #30515, asked that this aspect be discussed in a different issue than #30515, so hence this issue is being opened now.