  Tests that cause spurious failures without any code changes, i.e., flaky
tests, hamper regression testing, increase maintenance costs, may shadow real
bugs, and decrease trust in tests. While the prevalence and importance of
flakiness is well established, prior research focused on Java projects, thus
raising the question of how the findings generalize. In order to provide a
better understanding of the role of flakiness in software development beyond
Java, we empirically study the prevalence, causes, and degree of flakiness
within software written in Python, one of the currently most popular
programming languages. For this, we sampled 22352 open source projects from the
popular PyPI package index, and analyzed their 876186 test cases for flakiness.
Our investigation suggests that flakiness is equally prevalent in Python as it
is in Java. The reasons, however, are different: Order dependency is a much
more dominant problem in Python, causing 59% of the 7571 flaky tests in our
dataset. Another 28% were caused by test infrastructure problems, which
represent a previously undocumented cause of flakiness. The remaining 13% can
mostly be attributed to the use of network and randomness APIs by the projects,
which is indicative of the type of software commonly written in Python. Our
data also suggests that finding flaky tests requires more runs than are often
done in the literature: A 95% confidence that a passing test case is not flaky
on average would require 170 reruns.
