Repository: go
Issue: proposal: net/http: Transport tracing hooks
Labels: Proposal,Proposal-Hold

The current HTTP client instrumentation library httptrace tracks the life cycle of a single request with hooks present at different stages in the outgoing HTTP Request's journey. This limits the use of `httptrace` in many ways which we believe could be solved by having an instrumentation library which works at a “transport level”. This means instead of placing hooks in the request’s context, one defines these hooks for the `Transport`. This would also support the current functionality of `httptrace`.

### Proposal

Here is a basic example of the proposed change. Let’s take the example of the `GotConn` hook in `httptrace.ClientTrace`. The current method for adding and using this hook is 

```
req, _ := http.NewRequest("GET", "http://example.com", nil)
trace := &httptrace.ClientTrace{
	GotConn: func(connInfo httptrace.GotConnInfo) {
		fmt.Printf("Got Conn: %+v\n", connInfo)
	},
}
req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
resp, _ := http.DefaultClient.Do(req)
```

We propose restructuring `http.Transport` to contain the hooks instead of `ClientTrace`. `Transport` contains a new struct `Stats` which contains the hooks and other data like counts of idle/active connections and waiting requests. 

```
type Stats struct {
	GotConn func(connInfo GotConnInfo)
        
	// among other hooks and data
}

type Transport struct {
	Stats *Stats
	// and all other fields to stay the same
}
```

And one would make a request in the following way:- 
```
req, _ := http.NewRequest("GET", "http://example.com", nil)
stats := &http.Stats{
	GotConn: func(connInfo http.GotConnInfo) {
		fmt.Printf("Got Conn: %+v\n", connInfo)
	},
}
transport := http.Transport{Stats: stats}
client := http.Client{Transport: transport}
resp, _ := client.Do(req)
```

Within `Transport`, when a request has received a connection, the `GotConn` hook would run as below. This is in contrast to extracting `ClientTrace` from the `Request`’s context and using its hooks.

```
func (t *Transport) methodName() {
    // a connection has been delivered to the request
    if t.Stats.GotConn != nil {
	  t.Stats.GotConn(GotConnInfo{…})
    }
…
}
```

The above basic proposal can be extended to all the hooks already present in `httptrace`. We propose the following additional hooks to be added to `http.Stats` above:-

1. **IdleConnWaitIn, IdleConnWaitOut, ConnsPerHostWaitIn, and ConnsPerHostWaitOut** - These hooks will be called when a request (`wantConn`) enters/exits the `idleConnWait` and `connsPerHostWait` queues. For example -
```
type IdleConnWaitInInfo struct {
    ConnectMethodKey string
    EventTime time.Time
    RemainingInQueue int
}
IdleConnWaitIn(info IdleConnWaitInInfo)
```
  These hooks could take as parameters `connectMethodKey` of the queue, time of entry/exit, and the number of remaining __active__ `wantConns` in the queues after the event (an __active__ `wantConn`is one which has not been delivered a `persistConn` yet and its `context` has not exceeded its deadline. Note that this is not equal to the number of `wantConns` in the queues as some of them could have been delivered a `persistConn` in another goroutine or their context could have exceeded its deadline.) The `IdleConnWaitOut` and `ConnsPerHostWaitOut` hooks could also have an extra parameter `error` which is `nil` if the `wantConns` were removed from the queues because they were successfully delivered a `persistConn`. If not, `error` gives the reason for their removal from the queues, for example, if the request was cancelled or its context deadline expired.

2. **IdleConnIn, IdleConnOut** - `IdleConnIn` serves the same purpose as the existing `PutIdleConn` hook from the `httptrace` package. `IdleConnOut` is the related hook for when a connection is removed from the `idleConn` queue. These hooks could take as parameters `connectMethodKey` of the queue and time of entry/exit. `IdleConnOut` could take an additional parameter error which is nil if the idle connection was delivered to a `wantConn`. If not, it represents the reason it was removed from the queue, for example, being too old to be reused.

3. **ConnReused** - This hook is called after serving a request, if the connection is directly delivered to a `wantConn` waiting in `idleConnWait` instead of being added to the `idleConn` queue. Currently, there is only one hook related to connections, `PutIdleConn`, which records the latter. `ConnReused` could take as parameters `connectMethodKey` of the `persistConn` (`cacheKey` field) and time of delivery.

4. **MaxIdleConnsHit, MaxIdleConnsPerHost, MaxConnsPerHostHit** - When configured capacities of queues like `MaxIdleConns`, `MaxIdleConnsPerHost`, `MaxConnsPerHost` are hit, these hooks should be able to take a "snapshot of the state" of the HTTP Client.  They could take as parameters the counts of __active__ `wantConns` in `idleConnWait` and `connsPerHostWait` queues for each `connectMethodKey` and the length of `idleConn` queues for each `connectMethodKey`. They could also take as parameters the state of all active connections like their TCP connection state( this is an optional feature for which we may have to make external calls).  These hooks are useful for the user to understand where their server's bottlenecks or the server they are connecting to.

**Note:** I have already implemented this during an internship.