Prompto: An open source library for asynchronous querying of LLM endpoints (2025)

Abstract: Recent surge in Large Language Model (LLM) availability has opened exciting avenues for research. However, efficiently interacting with these models presents a significant hurdle since LLMs often reside on proprietary or self-hosted API endpoints, each requiring custom code for interaction. Conducting comparative studies between different models can therefore be time-consuming and necessitate significant engineering effort, hindering research efficiency and reproducibility. To address these challenges, we present prompto, an open source Python library which facilitates asynchronous querying of LLM endpoints enabling researchers to interact with multiple LLMs concurrently, while maximising efficiency and utilising individual rate limits. Our library empowers researchers and developers to interact with LLMs more effectively and allowing faster experimentation, data generation and evaluation. prompto is released with an introductory video (https://youtu.be/lWN9hXBOLyQ) under MIT License and is available via GitHub (https://github.com/alan-turing-institute/prompto).

## introduction the field of natural language processing is going through a massive transition since the introduction of transformer-based large language models (llms) @xcite @xcite which have demonstrated exceptional generalisation capability on a wide range of language-related tasks. while user-friendly interfaces like chatgpt, gemini and claude have made llms more accessible to the public, researchers nowadays increasingly interact with such models through programmatic interfaces and apis. la @xcite noted several reproducibility issues with this language-models-as-a-service (lmaas) paradigm @xcite , where language models are cen-trally hosted and typically provided on a subscription or pay-per-use basis (e.g., the openai api foot_2 and google's gemini api foot_3 ). to address some of them, @xcite suggested a series of best practices when evaluating llms, such as sharing your exact prompts, parameter inputs and code and always providing model outputs. in addition to lmaas, there are now also several open llms, defined here as those with broadly available model weights as in @xcite , e.g. llama (llama team, 2024), gemma @xcite , aya @xcite . these "open" models are often accompanied by a model card @xcite ) which provides instructions for executing them locally and for accessing their internals and weights. this allows users to create their own api endpoints for llms (e.g., via ollama foot_4 ) on their available hardware. a significant challenge with both lmaas and open llms is that conducting a comparative study across multiple models necessitates writing separate code to interact with each api and this obstacle further hinders already complex evaluation and reproducibility efforts. an additional problem is that apis may have different constrains (e.g., queryper-minute (qpm) rate limits), adding even more complexity to the engineering design. to address these limitations and simplify largescale comparative studies across llms, we introduce prompto, an open source python library for asynchronous querying of llm endpoints in a consistent and highly efficient manner. prompto uses asynchronous programming to efficiently interact with endpoints by allowing users to send multiple requests to different apis concurrently. this eliminates idle wait times and maximises efficiency, especially when dealing with different rate limits. in contrast, in traditional synchronous programming, a user sends a single request to an endpoint and waits for a response from the api before sending another request, repeating for each query. prompto supports a range of lmaas endpoints (e.g. ope-nai, gemini, anthropic) as well as self-hosted endpoints for querying local models (e.g. ollama, hugging face's text-generation-inference foot_5 for serving models hosted on hugging face). the codebase is easily extensible to integrate new apis and/or locally self-hosted models. for instance, we provide an example using quart foot_6 in prompto to easily set up an endpoint for inferencing local models using transformers @xcite . inspired by @xcite ; @xcite , the library promotes experiment reproducibility by facilitating the definition of all inputs/prompts within a single json lines (jsonl) or csv file. the file can encompass queries for various apis and models, enabling parallel processing for even greater efficiency gains. the library's scalability allows it to handle large-scale experiments. prompto also provides built-in functionalities for automatic evaluation of the obtained responses, such as allowing the user to apply scoring functions to model outputs, and model graded evaluation or llm-as-a-judge @xcite . in this paper, we present an overview of prompto, highlighting its modular design and flexibility. we present prompto's functionalities, design choices, technical implementation and show its advantages in comparison with alternative approaches. we accompany its release with extensive documentation and a series of jupyter notebooks as tutorials, to allow the research community to easily explore all its functionalities. in addition to the library's provided examples foot_7 , we provide an illustrative showcase of using our library and compare against a synchronous approach to query several llm endpoints in parallel in appendix a. ## related work and motivation to support transparent and reproducible evaluation of large language models, a series of evaluation frameworks have been published in recent years, such as language model evalua-tion harness (lm-eval foot_8 ) @xcite , the uk ai safety institute's inspect framework (inspect-ai foot_9 ), confident ai's deepeval (deeepeval foot_10 ), llm comparator (llm-comparator foot_11 ). nevertheless, conducting evaluation experiments on multiple large language models in order to compare their performance remains an engineering challenge, as each endpoint requires dedicated code to be written. to address this, our prompto library allows users to specify queries for various apis and models within a single jsonl file, without having to interact directly with the details of each endpoint's infrastructure. this prioritises flexibility in experiment design, simplifies the setup for researchers and especially promotes reproducibility as all model parameters and prompts can be documented within a single file. after running an experiment, we record all model responses into an output experiment file which can be further analysed or used downstream in other pipelines. furthermore, our prompto library prioritises efficiency when running experiments. when interacting with llms, a sequential and synchronous approach where requests are sent to an endpoint and then waited upon for a response is highly inefficient, especially when dealing with different rate limits or when conducting comparisons across multiple models. prompto and tackles this challenge by leveraging asynchronous programming. the ability to interact with multiple llms concurrently and process requests asynchronously makes our library particularly well-suited for initial exploration and rapid comparison between different llms. researchers can efficiently experiment with various models and gauge their performance on specific tasks. this might be particularly useful in experiments where an "expected" or "ideal" response is not known ahead of time since evaluation framework tools are often most useful for settings where there is pre-defined model response from which the performance of an llm can be scored against (e.g. automated metrics such as bleu @xcite , hellaswag @xcite or mmlu @xcite ). prompto may also be preferred in settings where the responses are intended for human evaluation, for instance in finally, prompto's applicability goes beyond beyond evaluation tasks, and can be leveraged to generate synthetic datasets from llms for model training and development @xcite , or to create datasets specifically tailored for llm distilation tasks, a technique for compressing knowledge from larger models into smaller, more efficient ones @xcite @xcite . ## library design our prompto library is an open source python package foot_12 which facilitates processing of experiments of language models stored as jsonl files. the library has commands to process an experiment file to query models and store results. a user can query multiple models asynchronously and in parallel in a single experiment file. we detail key components of setting up an experiment and core commands of the library. for a comprehensive exploration of prompto's features, we kindly refer readers to the detailed documentation at https://alan-turing-institute. github.io/prompto/ . ## conclusions and future work we present prompto, an open source library designed to facilitate researchers to efficiently query large language models which reside on proprietary or self-hosted api endpoints. using prompto is simple and all inputs/prompts can be easily defined in a single jsonl file, which can encompass prompts for various apis and models. outputs are then stored in output jsonl files allowing researchers to easily share experiment outputs promoting reproducibility. consequently, our library enables faster experimentation and evaluation. prompto is an ongoing open source project and we welcome contributions from the community to ensure support for a wide range of llm endpoints and new features. there are a number of interesting avenues for extending prompto. in the future, we hope to continue focusing on extending the evaluation pipelines discussed in section 3.5. for evaluation, integration of larger-scale nlp pipelines can be useful in order to efficiently apply more advanced scoring functions such as toxicity classifiers and other safety risk classifiers for automatic safety evaluation. furthermore, integrating prompto with data visualisation libraries can bring additional plotting features to the library to enable deeper exploration of model performance. another direction of work is to extend the rephrasing pipelines in prompto (see section 3.6). a particular area of interest is extending the pipeline for translation of prompts. this can be useful for synthetic data generation applications. the current pipeline enables the use of llms for translation which have demonstrated remarkable potential in handling multilingual machine translation (mmt) @xcite , however extending this pipeline to dedicated mmt models, such as nllb @xcite , could particularly be effective for translating to lower resourced languages. moreover, a rephrasing pipeline can also be important in evaluation settings to generate a wider set of prompts to test a model. however, further research is required in order to do this effectively. lastly, future work includes expanding our tool's accessibility by developing direct local model integration capabilities. the current implementation requires setting up an api endpoint for querying, but we envision simplifying this process by enabling direct model instantiation (e.g., using transformers @xcite ). while toolkits for deploying and serving llms, such as text-generation-inference or vllm @xcite , already simplify setting up efficient endpoints, this approach would reduce overhead for researchers who wish to query local models and utilise offline batched inference, allowing for more streamlined experimentation and rapid prototyping. large language models: empirical results and analysis. in findings of the association for computational linguistics: naacl 2024, pages 2765-2781, mexico city, mexico. association for computational linguistics.