Repository: Pythagora-io/gpt-pilot
File: core/agents/external_docs.py (Python)

Code before:
                reqs.append(client.get(url, params={"q": qs, "doc_key": docset_key}))
                ordered_keys.append(docset_key)

            try:
                results = await asyncio.gather(*reqs)
            except httpx.HTTPError:
                log.warning("Failed to fetch documentation snippets", exc_info=True)

        for k, res in zip(ordered_keys, results):
            snippets.append((k, res.json()))
        return snippets

    async def _store_docs(self, snippets: list[tuple], available_docsets: list[tuple]):
        """Store the snippets into current task data.

        Documentation snippets are stored as a list of dictionaries:
        {"key": docset-key, "desc": documentation-description, "snippets": list-of-snippets}

        """

        docsets_dict = dict(available_docsets)
        docs = []
        for docset_key, snip in snippets:
            docs.append({"key": docset_key, "desc": docsets_dict[docset_key], "snippets": snip})

        self.next_state.current_task["docs"] = docs
        self.next_state.flag_tasks_as_modified()


Review comment (bug):
I'd prefer the docs are stored directly in the project state, similar to how we handle relevant files and modified files.

The "tasks" structure is intended to capture the information for all the tasks that we need in each state, not as a free-form bag in which to put data relevant to the current state or current task. While one can argue that task instructions also fail this, we should not dig ourselves deeper in the hole by adding more stuff.

I know this adds a couple of extra minutes of work now, but if we do this we're never going to go back on it and it's going to make each new addition like this less objectionable,...


Code after:
        """
        url = urljoin(EXTERNAL_DOCUMENTATION_API, "query")
        snippets: list[tuple] = []
        async with httpx.AsyncClient(transport=httpx.AsyncHTTPTransport(retries=3)) as client:
            reqs = []
            ordered_keys = []
            for docset_key, qs in queries.items():
                reqs.append(client.get(url, params={"q": qs, "doc_key": docset_key, "num_results": 3}))
                ordered_keys.append(docset_key)

            try:
                results = await asyncio.gather(*reqs)
            except httpx.HTTPError:
                log.warning("Failed to fetch documentation snippets", exc_info=True)

        for k, res in zip(ordered_keys, results):
            snippets.append((k, res.json()))
        return snippets

    async def _store_docs(self, snippets: list[tuple], available_docsets: list[tuple]):
        """Store the snippets into current task data.

        Documentation snippets are stored as a list of dictionaries:
        {"key": docset-key, "desc": documentation-description, "snippets": list-of-snippets}

        """

        docsets_dict = dict(available_docsets)
        docs = []
        for docset_key, snip in snippets:
            docs.append({"key": docset_key, "desc": docsets_dict[docset_key], "snippets": snip})

        self.next_state.docs = docs
        self.next_state.flag_tasks_as_modified()