  While algorithm audits are growing rapidly in commonality and public
importance, relatively little scholarly work has gone toward synthesizing prior
work and strategizing future research in the area. This systematic literature
review aims to do just that, following PRISMA guidelines in a review of over
500 English articles that yielded 62 algorithm audit studies. The studies are
synthesized and organized primarily by behavior (discrimination, distortion,
exploitation, and misjudgement), with codes also provided for domain (e.g.
search, vision, advertising, etc.), organization (e.g. Google, Facebook,
Amazon, etc.), and audit method (e.g. sock puppet, direct scrape,
crowdsourcing, etc.). The review shows how previous audit studies have exposed
public-facing algorithms exhibiting problematic behavior, such as search
algorithms culpable of distortion and advertising algorithms culpable of
discrimination. Based on the studies reviewed, it also suggests some behaviors
(e.g. discrimination on the basis of intersectional identities), domains (e.g.
advertising algorithms), methods (e.g. code auditing), and organizations (e.g.
Twitter, TikTok, LinkedIn) that call for future audit attention. The paper
concludes by offering the common ingredients of successful audits, and
discussing algorithm auditing in the context of broader research working toward
algorithmic justice.
