Real-world application scenarios like modern Internet services consist of diversity of AI and non-AI modules with very long and complex execution paths . Using component or micro AI benchmarks alone can lead to error-prone conclusions. This paper proposes a scenario-distilling AI benchmarking methodology . Instead of using real-world applications, we propose the permutations of essential AI and non-AI tasks as a scenario-distilling benchmark . We consider scenario-distilling benchmarks, component and micro benchmarks as three indispensable parts of a benchmark suite. Together with seventeen industry partners, we identify nine important real-world application scenarios . We design and implement a highly extensible, configurable, and flexible benchmark framework . On the basis of the framework, we propose the guideline for building scenario-distilling benchmarks, and present two Internet service AI ones. The preliminary evaluation shows the advantage of scenario-distilling AI benchmarking against using component or micro AI benchmarks alone. The specifications, source code, testbed, and results are publicly available from the web site URL/ AIBench /index.html}.
Modern real-world application scenarios like Internet services not only consist of diversity of AI and non-AI modules with very long and complex execution paths , but also have huge code size, which raises serious benchmarking or evaluating challenges. Using AI components or micro benchmarks alone can lead to error-prone conclusions. This paper presents a scenario-distilling methodology to attack the above challenge. We formalize a real-world application scenario as a Directed Acyclic Graph-based model, and propose the rules to distill it into the permutation of essential AI and non-AI tasks as a high-level scenario benchmark specification. Together with seventeen industry partners, we extract nine typical application scenarios, and identify the primary components . We design and implement a highly extensible, configurable, and flexible benchmark framework , on the basis of which, we implement two Internet service AI scenario benchmarks as proxies to two real-world application scenarios. We claim scenario, component and micro benchmarks should be considered as three indispensable parts for evaluating. Our evaluation shows the advantage of our methodology against using component or micro AI benchmarks alone. The specifications, source code, testbed, and results are publicly available from URL/ aibench-scenario /index.html}.
R
Real-world
Modern real-world
R
modern Internet services
Internet services not only
R
. Using component or micro AI
, but also have huge code size, which raises serious benchmarking or evaluating challenges. Using AI components or micro
R
proposes
presents
R
AI benchmarking methodology . Instead of using
methodology to attack the above challenge. We formalize a
R
applications, we propose the permutations
application scenario as a Directed Acyclic Graph-based model, and propose the rules to distill it into the permutation
R
scenario-distilling benchmark . We consider scenario-distilling benchmarks, component and micro benchmarks as three indispensable parts of a benchmark suite.
high-level scenario benchmark specification.
R
identify nine important real-world application scenarios
extract nine typical application scenarios, and identify the primary components
R
. On
, on
R
the framework, we propose the guideline for building scenario-distilling benchmarks, and present
which, we implement
R
ones. The preliminary
scenario benchmarks as proxies to two real-world application scenarios. We claim scenario, component and micro benchmarks should be considered as three indispensable parts for evaluating. Our
R
scenario-distilling AI benchmarking
our methodology
D
the web site
R
AIBench
aibench-scenario