Abstract-Despite the recent advances in test generation, fully automatic software testing remains a dream: Ultimately, any generated test input depends on a test oracle that determines correctness, and, except for generic properties such as "the program shall not crash", such oracles require human input in one form or another. CrowdSourcing is a recently popular technique to automate computations that cannot be performed by machines, but only by humans. A problem is split into small chunks, that are then solved by a crowd of users on the Internet. In this paper we investigate whether it is possible to exploit CrowdSourcing to solve the oracle problem: We produce tasks asking users to evaluate CrowdOraclesassertions that reflect the current behavior of the program. If the crowd determines that an assertion does not match the behavior described in the code documentation, then a bug has been found. Our experiments demonstrate that CrowdOracles are a viable solution to automate the oracle problem, yet taming the crowd to get useful results is a difficult task.
To solve the oracle problem, Pastore et al. REF applied the crowdsourcing technique to generating test inputs depending on a test oracle that requires human input in one form or another.
CrowdOracles: Can the Crowd Solve the Oracle Problem?
2013 IEEE Sixth International Conference on Software Testing, Verification and Validation
2013 IEEE Sixth International Conference on Software Testing, Verification and Validation
Computer Science