  Recruiting professional programmers in sufficient numbers for research
studies can be challenging because they often cannot spare the time, or due to
their geographical distribution and potentially the cost involved. Online
platforms such as Clickworker or Qualtrics do provide options to recruit
participants with programming skill; however, misunderstandings and fraud can
be an issue. This can result in participants without programming skill taking
part in studies and surveys. If these participants are not detected, they can
cause detrimental noise in the survey data. In this paper, we develop screener
questions that are easy and quick to answer for people with programming skill
but difficult to answer correctly for those without. In order to evaluate our
questionnaire for efficacy and efficiency, we recruited several batches of
participants with and without programming skill and tested the questions. In
our batch 42% of Clickworkers stating that they have programming skill did not
meet our criteria and we would recommend filtering these from studies. We also
evaluated the questions in an adversarial setting. We conclude with a set of
recommended questions which researchers can use to recruit participants with
programming skill from online platforms.
