Repository: go
Issue: proposal: go/types: lazy resolution of imported objects
Labels: Proposal

cmd/compile uses an indexed export format that allows lazily resolving imported objects so that only the objects that are actually used by user code need to be loaded into memory. However, the go/types API doesn't currently support lazy import resolution, so the gcexportdata importer still needs to eagerly load all objects. (It's able to use the index to skip loading of objects that were already loaded by another importer though.)

This proposal is to add a new function to facilitate lazy import resolution:

```
func (s *Scope) InsertLazy(name string, resolve func() Object) Object
```

Semantically from the end user's point of view, this function behaves like:

```
func (s *Scope) InsertLazy(name string, resolve func() Object) Object {
    obj := resolve()
    if name != obj.Name() {
        panic("invalid resolved object name")
    }
    return s.Insert(obj)
}
```

except it's unspecified when/whether `resolve` is actually called.


Importer implementations can then use this function to create lazy objects to add to the Package.Scope without having to fully load all details right away. go/types then assumes responsibility for noticing when it has a lazy object and calling resolve to get the actual object.

For example, a partial sketch of how this might really work:

```
type lazyObject func() Object

// Unused methods to implement Object.
func (*lazyObject) Pkg() *Package { panic("unresolved object") }
// ...

func (s *Scope) InsertLazy(name string, resolve func() Object) Object {
    if alt := s.elems[name]; alt != nil {
        return alt
    }
    if s.elems == nil {
        s.elems = make(map[string]Object)
    }
    s.elems[name] = lazyObject(resolve)
    return nil
}

func (s *Scope) Lookup(name string) Object {
    obj := s.elems[name]
    if resolve, ok := obj.(lazyObject); ok {
        obj = resolve()
        if name != obj.Name() {
            panic("invalid resolved object name")
        }
        s.elems[name] = obj
        if obj.Parent() == nil {
            obj.setParent(s)
        }
    }
    return obj
}
```

Other internal code that directly accesses Scope.elems would need to be similarly updated to handle `lazyObject`s. User code should be unaffected though: the public APIs will always return fully resolved objects.

One possible concern with lazy resolution though is the `resolve` function for any unresolved objects will need to hang onto the importer resources for constructing them. Which in turn means the `Scope`, `Package`, and any other `Object`s from that same package will also keep those resources live. I would expect this is likely still an improvement over the status quo (i.e., holding onto a single flat byte slice, rather than the complete object/type graph).

Another possible future extension would be for the Scope to only keep a weak reference to the resolved object, and then re-resolve it on demand as needed. I think this would be transparent both to users and importer implementations (as long as the resolve method is safe to invoke multiple times).

/cc @griesemer @findleyr