Paper: Efficiency Pentathlon: A Standardized Benchmark for Efficiency Evaluation (ICLR-2024)

Reviewer comment (Writing):
The whole paper is not very clear. In the section 2, the reason that considering the proposed five aspects for efficiency evaluation is not described clearly.

Author response:
We believe that using multiple metrics to evaluate a model's efficiency is crucial for a comprehensive understanding of its performance and suitability for various applications. It can also help understand the tradeoff among these aspects and help make better decisions while deploying the models. Each metric provides a different perspective 
- Latency is crucial for real-time applications, such interactive systems like chatbots,  and real-time translation systems, where quick response is critical for the user’s experience.
- Throughput is important in scenarios where the model needs to process large volumes of data, e.g., batched processing of many scientific papers or news articles.
- Memory overhead can reveal whether a model might not be suitable for deployment on devices with limited memory, such as mobile phones or IoT devices.
- Energy consumption is important for justifying the model’s suitability for battery-powered devices or in situations where energy efficiency is a priority.
- Model size  affects its portability and the speed of its deployment. Smaller models are easier to distribute and can be deployed on devices with limited storage capacity.

We will highlight these motivations in the revision.

Reviewer comment (Experiments):
The experimental parts are not very sufficient. Only two tasks are selected which makes the results not very convincing.

Author response:
Our decision of experimenting with three tasks (instead of two) aims to demonstrate the Pentathlon’s versatility supporting a variety of architectures and settings.  Please refer to the general response for further details.

Reviewer comment (Presentation):
Table 1 is not very clear. What is the schema meaning in table 1, such as Acc., TP., Latency, Mem., etc. The authors should describe them in the table title.

Author response:
We chose to use abbreviations in the interest of space, which are described in the caption of Table 1. Acc.: accuracy, TP.: throughput; Mem.: memory overhead.

Reviewer comment (Evaluation):
The paper is more like a technical report, which may suit a benchmark or industry track. It would be great if the authors could provide additional scientific findings and conclusions through the evaluations. This work has provided a relatively comprehensive and mature evaluation benchmark. With more inspiring and interesting findings and conclusions based on the evaluations, the paper would be more valuable to the community.

Author response:
We agree with the reviewer that this submission fits the datasets and benchmarks track of ICLR 24, the exact track that this paper is submitted to: https://iclr.cc/Conferences/2024/CallForPapers

We believe that the value of this work lies in providing the community with a platform that can help discover interesting findings and conclusions about the efficiency of machine learning models.

Reviewer comment (Experiments):
There are many datasets designed for large language model evaluation. Using classical tasks (e.g., machine translation, mathematical reasoning, and classification) makes the experiments less convincing.

Author response:
Our decision to experiment with machine translation, classification, and math word problems aims to demonstrate the Pentathlon’s versatility supporting a variety of architectures and settings.  Please refer to the general response for further details.

Reviewer comment (Reproducibility):
Hardware Flexibility: While the authors have outlined CPU and GPU settings, it remains unclear whether the benchmark suite can easily accommodate other hardware platforms. Given the growing popularity of new platforms like Metal, ROCm, Mali, and Vulkan, it's essential to address the adaptability of Pentathlon to ensure it remains relevant and applicable to diverse hardware configurations. Moreover, certain models, such as Llama-70b, may require multiple high-end GPUs like A100 or H100 for distributed inference, highlighting the need for flexibility in hardware options.

Author response:
We appreciate the reviewer for raising the valid concern that controlling the hardware leads to less flexibility. In preliminary experiments, we found it possible to simulate mobile phone platforms by connecting to our Linux-host an Arm chip device. Some on-device applications can be simulated by running our platform on the NVIDIA Jeston module: https://developer.nvidia.com/embedded/jetson-tx2

We acknowledge the challenge of continuously supporting new hardware platforms, which is beyond the capabilities of the authors. Trying to address this, Efficiency Pentathlon is designed to be flexible, and can be used as a standalone software without our machine. It is compatible with any hardware platform that supports Linux, offering a versatile solution for users on their preferred hardware platform when centralized evaluation and controlling for the hardware are not primary concerns.

Reviewer comment (Experiments):
Software Environment Assumptions: The paper primarily focuses on a specific software environment, potentially overlooking the fact that various software stacks, such as TVM and Cutlass, may require an additional step called tuning. This tuning phase optimizes the compilation stack for the given hardware, which can significantly improve model performance. However, the tuning process itself may not always be efficient and can be time-consuming. It's crucial to consider these software-related aspects for a more comprehensive evaluation.

Author response:
Trying to maximize software flexibility is something we considered while designing the Pentathlon. Our benchmark is designed with minimal assumptions regarding the evaluated system's software environment, with the primary requirement of being able to run in a Docker container. As a quick verification, Cutlass, which the reviewer brought up, is compatible with our benchmark. Due to the limited time of the response period, the authors were not able to verify TVM at the time of this response. We plan to include this discussion in the revision.

Even when the evaluated system requires compilation, its efficiency will not be negatively affected. Specifically, the Efficiency Pentathlon begins tracking efficiency metrics only once the system has completed its compilation phase and is operational.This is implemented through a 'handshake protocol': The benchmark sends a dummy input batch to the system. It will start to measure the efficiency only after it has received the system’s outputs for this dummy batch, indicating that the model has finished its preparation stage and is ready to go.

Reviewer comment (Experiments):
Controlled Hardware vs. Cloud-Based Platforms: While the controlled hardware setting provides fairness and accuracy, it may not fully cater to researchers who heavily rely on cloud-based platforms like AWS, Azure, or Google Cloud. Many recent large language models (LLMs) are built and deployed on cloud platforms, and their efficiency and latency results may significantly differ from those obtained in a controlled environment. To make Pentathlon more applicable to a wider range of real-world scenarios, consideration could be given to extending the benchmarking to include cloud-based machines and their specific challenges.

Author response:
We fully agree with the reviewer’s great point that controlling for the hardware may not always be is not always practical or desirable in practice. This consideration is integrated into the design of our benchmark. The benchmark is developed as a standalone, open-source software, allowing users to employ it on their preferred hardware platforms, including many cloud machines that support Linux. We will emphasize this point more explicitly in the revision.

Reviewer comment (Reproducibility):
What level of effort is required to expand Pentathlon to accommodate a new hardware platform or incorporate a new model into the benchmark?

Author response:
Expanding our centralized host machine to accommodate a new hardware platform involves physically connecting  the hardware to our host machine and making necessary software adjustments. This process can be challenging and may take 6–10 hours based on the authors’ experience. For users that want to use our benchmark on their chosen hardware,  we expect the process to involve minimal effort, typically just cloning our software and installing necessary dependencies, provided their hardware is Linux-compatible.

To make it easier to incorporate new models, we provide code templates and detailed documentation to guide this process. Based on the experience of an author who was initially unfamiliar with the benchmark, it's estimated that an average researcher can adapt a model to our benchmark in under two hours. if the model is already compatible with Hugging Face APIs or is hosted on Hugging Face, much less effort is required.

Reviewer comment (Evaluation):
Beyond the BLEU score, are there additional metrics available within Pentathlon to assess model quality, such as perplexity or other relevant NLP-specific metrics?

Author response:
Efficiency Pentathlon is intentionally designed without built-in support for accuracy or quality metrics. Instead, it provides users with the models’ outputs, deferring to them conducting quality evaluations. We choose to do this for a couple of reasons. 

First, Efficiency Pentathlon is designed to support a variety of tasks, and its current version supports anything that is available on Hugging Face. Anticipating and implementing the accuracy metrics for these tasks is beyond our capabilities and the scope of this work. Second, for many tasks (e.g., MT as the reviewer mentioned), post-processing decisions can significantly affect a model's accuracy. For instance, in XX to Romanian translation, the choice to include or exclude diacritics can greatly influence the model's BLEU score. Such nuances in post-processing are critical to the overall performance evaluation. We believe that the users are better-positioned to make decisions about e.g., post processing and selecting evaluation metrics for quality. Therefore we decide to not impose specific evaluation metrics for model quality, instead leaving these decisions to the users.

Reviewer comment (Evaluation):
How can you distinguish the impact of "algorithmic innovations" from other efficiency-related factors in the Pentathlon benchmark?

Author response:
The reviewer raised a great point that algorithmic innovations can be hard to distinguish from many other factors in practice. To address this, we try to control as many potential “non-algorithmic” confounders as possible, including hardware platforms, software environment, data processing pipelines, IO frameworks, etc.

Reviewer comment (Presentation):
Figure 3 (a) and (b) are exactly the same? Is it a coincidence or mistake?

Author response:
Thanks for catching this. This was due to a LaTex typo and has been fixed.