The idea of a quantum computer was first popularized by Richard Feynman near 40 years ago (Feynman, 1982) . The concept of a second quantum revolution expanded this idea to an array of quantum technologies (Dowling & Milburn, 2003) that has been utilized for almost two decades now. However, it has been only recently demonstrated that quantum computers could outperform even the strongest classical supercomputers on particular (albeit useless for now) tasks (Arute et al., 2019) , which was considered as a milestone and noted as the point of "quantum supremacy" (Preskill, 2011) .
These developments in the hardware realm are also supported by national initiatives in the order of billion dollars by countries like UK (Knight & Walmsley, 2019) , China (Zhang, Xu, Li, Liu, & Pan, 2019) , Russia (Fedorov et al., 2019) , and the US (Monroe, Raymer, & Taylor, 2019; ). Also on a regional level, the Quantum Flagship program of the European Union is the largest international initiative of this scale (Riedel, Kovacs, Zoller, Mlynek, & Calarco, 2019) . Most of these initiatives have strong focus on not only research and development but also in commercialization of these technologies.
This focus on commercialization and maturation of quantum technologies allowed hundreds of firms to start exploring the commercial opportunities presented by these developing set of technologies. As of 2019, there are more than 200 firms dealing in quantum technologies and more than half of them are focused on quantum computing (Z. Seskir, 2019) . It has been noted in the literature that expanding the general level of understanding on quantum technologies, especially quantum computing, is gaining importance for both societal impact (Vermaas, 2017) , and commercialization purposes in terms of developing the necessary quantum workforce (of Sciences Engineering & Medicine, 2019).
In 2016 IBM launched the world's first open-access quantum computing hardware scheme called 'IBM Quantum Experience' 1 . Initially, the system was a circuit composer with a drag and drop interface. Later in 2017, Qiskit was released by IBM as a programming language built upon Python for programmers to write quantum algorithms and run.
In this line of thinking, the QWorld initiative has been launched with the mission of establishing "an open access and public global ecosystem for quantum technologies and quantum software by the year 2025" 2 . To involve more people in the field of quantum computing, QWorld has adopted the idea of organizing introduction to quantum programming workshops, and used quantum programming as the tool to introduce quantum computing. As of September 2020, 34 workshops have been organized in 14 countries and 614 people have completed the workshops.
The material developed by QWorld to be used during the workshops is called Bronze 3 and it is intended for everyone with a background in linear algebra and programming experience in Python. Bronze introduces quantum computing as a generalization of probabilistic computing, which is an approach adopted by computer scientists (Arora & Barak, 2009; Yanofsky & Mannucci, 2008) and also popularized by Scott Aaronson in his book 'Quantum Computing since Democritus' (Aaronson, 2013) . Leaving the underlying physics behind, this approach allows teaching quantum computing phenomena through abstract and linear algebraic concepts with which the participants are already familiar. Furthermore, the task-based programming approach helps participants to learn through experience instead of the classical approach which first presents all information about the subject without leaving any gap for self-learning. Altogether, this methodology accelerates the learning process through bypassing the re-education of participants on the physical phenomena and puts off the discussions on the fundamental properties of physical reality such as determinism 1 https://www-03.ibm.com/press/us/en/pressrelease/49661.wss 2 http://qworld.lu.lv/ 3 https://gitlab.com/qkitchen/basics-of-quantum-computing vs. uncertainty, realism vs. contextuality, and so on.
Computer science oriented approach which views quantum computing as a generalization of probabilistic computing has probably been used in many lectures, and informal educational materials such as ours. Yet, no studies discussing this methodology were found in the literature. Teaching quantum computing through task-based quantum programming is growing in popularity recently and there are some reported case studies (Carrascal, del Barrio, & Botella, 2020; LaRose, 2019; Mykhailova & Svore, 2020) . However, these works are mainly concerned with singular experiences in a limited context, usually one or two semesters of undergraduate/graduate level of university level courses and they do not undertake any study to assess the effectiveness of the approach on the knowledge acquisition of the students. The main purpose of this study is to propose a methodology for introducing quantum computing and to demonstrate its effectiveness through an empirical study on a number of dimensions. To the best of our knowledge, this is the first empirical study of this size in the field to measure knowledge acquisition in quantum computing.
In this context, pre and post-tests are conducted and data is collected from 22 workshops and 317 participants. Our first research objective is to test whether there is a significant increase in the knowledge of the participants. Our next research objective is to analyze data to see whether various factors such as gender, age, department and educational background have an impact on the learning process and to see whether the undertaken approach is suitable for introducing quantum computing to a large audience.
The rest of the paper is organized as follows. In Section 2, we review some background information about teaching quantum computing and programming. In Section 3 we describe the material used at workshops in detail. In Section 4, we explain our methods for assessment and in Section 5, we present our results. We discuss the main findings in Section 5.4 and in Section 7 we discuss the threats to validity. We conclude by Section 8.
A recent study in which 21 companies from the quantum industry are interviewed reveals that coding is the most valued skill according to these employers (Fox, Zwickl, & Lewandowski, 2020) . The study further suggests that an introductory level quantum course focusing either on the hardware or software track to increase quantum awareness would appeal to a variety of students from different science majors. A similar point is made by Peterssen (2020) , which emphasizes the need for training of engineers and programmers as specialists in Quantum Programming and Software to prepare for the quantum workforce. Another study on quantum engineering conducted with 26 experts in the field emphasizes the need for increasing 'quantum awareness' for the quantum workforce of the future (Gerke, Mller, Bitzenbauer, Ubben, & Weber, 2020) , while arguing that this is mostly necessary for non-physicists. Anticipating these rapid developments, in 2017, a diverse group of researchers from academia and industry published a document titled "Quantum Software Manifesto" (Ambainis et al., 2017) . This document emphasizes "educating more quantum programmers," among other crucial points. The literature also demonstrates that quantum computation is still more closely related to quantum information theory concepts than experimental fields on a conceptual level of mapping for quantum technologies related academic topics, but the academic literature is overwhelmingly rooted in physics (Z. C. Seskir & Aydinoglu, 2019) . During the Bronze workshops, participants get acquainted with quantum programming paradigms, which also serves to this necessity. This also allows Bronze to reach a larger audience as it does not require any prior knowledge on quantum mechanics and anyone with basic programming skills can attempt.
Traditional quantum computing curricula do not incorporate quantum programming as it is a fairly new area. Nevertheless, number of such courses and educative events is increasing as quantum industry is emerging. In 2019, Microsoft prepared an introductory quantum computing course which focused at implementation of quantum algorithms by Microsoft's quantum programming language Q#. Following the first course offered in University of Washington, Microsoft paired with 10 more institutions around the world (Microsoft, n.d.) . Likewise, IBM has prepared the Qiskit Textbook which is a collection of Jupyter Notebooks to introduce quantum computing through Qiskit programming languages (Asfaw, 2020) . Besides the traditional topics covered in quantum computing courses, it includes topics like error mitigation and benchmarking which are peculiar to IBM quantum computers. The textbook can be used for self-study or by the educators to integrate quantum programming to existing quantum computing curricula.
As courses and events related to quantum computing grow in numbers, the studies discussing experiences in teaching quantum computing are appearing as well. A report about the Microsoft course offered in University of Washington is prepared by Mykhailova and Svore (2020) , based on the observations of the authors during the course, the performance of the students and the feedback forms completed by 14 students out of 35 who completed the course. Overall, the study puts forward that software oriented approach is a viable way to introduce students to quantum computing. Hughes, Isaacson, Perry, Sun, and Turner (2020) discuss the survey results of 45 participants who took the course Quantum Computing as a High School Module prepared (Perry, Sun, Hughes, Isaacson, & Turner, 2019) for high school students between the age 15 and 18. The students have been successful at acquiring the new concepts and the course was able to generate a baseline knowledge on quantum computing. LaRose (2019) discusses experiences from a 1-week quantum computing course relying on quantum programming and aimed at students with no physics background. The survey results show that the students got the base for learning more on their own and most of them were interested in learning more. Carrascal et al. (2020) presents a methodology based on quantum programming for introducing undergraduate students with no physics background to quantum computing and shares the survey results which supports that of LaRose (2019).
In this section we will go through the details of the Bronze material.
Quantum computing courses are usually offered in physics and computer science departments at the graduate level. Most of the courses begin with teaching the concepts from quantum physics and focus on the theoretical aspects, which make quantum computing a relatively hard subject to grasp. There are several studies reporting student and teacher experience in learning and teaching quantum physics (Ireson, 2000; Krijtenburg-Lewerissa, Pol, Brinkman, & Van Joolingen, 2017; Mermin, 2003; Müller & Wiesner, 2002; Singh, 2001) . In (Akarsu, 2010) , the three main problems in teaching quantum physics related to mathematical difficulties as identified by the professors are listed as
• Students not being able to connect formal mathematical training with mathematical tools for quantum physics • Students struggling with the new mathematical notation used in quantum physics • Students having problems with the mathematical formulations of quantum physics Apart from mathematical difficulties, another problem stems from abstract physical concepts as quantum physics is less intuitive when compared to other physics courses.
Bronze is designed to teach quantum computing from a linear algebra and computer science perspective, by presenting quantum computing as a generalization of probabilistic and classical computing and abstracting the physical concepts through linear algebra. This is discussed elegantly in (Aaronson, 2013) through the following quote:
Quantum mechanics is what you would inevitably come up with if you started from probability theory, and then said, lets try to generalize it so that the numbers we used to call probabilities can be negative numbers. As such, the theory could have been invented by mathematicians in the nineteenth century without any input from experiment. It wasnt, but it could have been.
This approach has several advantages over the traditional approach as discussed next. In traditional quantum computing courses, the concept of superposition is introduced following its historical roots, either through photon polarization or spin of an electron. For the photon polarization case, the concept of light as an EM-wave with propagation at a certain direction, and polarization on an orthogonal direction needs to be understood. This requires a diversion into the theory of light and how waves propagate, which are not obvious for participants without a background covering at least one semester of modern physics. Similarly, introducing superposition using spin of an electron is discussed in physics course through Stern-Gerlach experiments, which have no further immediate conceptual use in quantum computing. Instead of relying on these, we focus on superposition as a property of an unit vector in a Hilbert space, which is a particular vector space where vectors have a constant L2 norm of one.
Quantum logic gates are introduced as rotation and reflection operations on this vector space, which are represented as unitary matrices. So, for anyone with a background in linear algebra, a quantum state is vector, and quantum gates are unitary matrices acting upon this state, preserving its norm. Similarly, entangled systems are just vectors in a higher dimension that cannot be represented as tensor products of single qubit states, and two-qubit gates are 4x4 unitary matrices. Measurement, which is the only quantum operation that does not preserve the norm of the state vector, is described as the projection operation on this vector to two orthogonal planes, where the associated probabilities are given as the appropriate functions depending on the angle between the state vector and these planes. This procedure, in fact, does not require any introduction of physical phenomena. Concepts of superposition and entanglement can entirely be explained using algebraic concepts, leaving the route taken through physics rather cumbersome for learners who will not work on hardware related topics. Additionally, this approach does not restrict further elaboration on the correspondence between these mathematical constructs and physical phenomena, but leaves it optional.
There are similar materials following the same approach such as The Quantum Quest, which was used in a 4-week online course for high school students by QuSoft and the University of Amsterdam (Ozols & Walter, 2018) . The course comes with an online drag and drop quantum simulator in which students can design quantum circuits.
Another feature of Bronze is that it does not involve any complex numbers and it is restricted real numbers. This choice facilitates the learning process, by reducing the level of difficulty the students face. It is still possible to discuss a wide range of subjects only focusing at real numbers as it will be discussed soon.
Bronze material consists of around 30 Jupyter notebooks and it uses Qiskit framework of IBM as the quantum programming language. Among the other alternatives such as Cirq, PyQuil, Q#, and ProjectQ, Qiskit comes forward as IBM Quantum Experience is the first platform to offer publicly available quantum computers and reached a large community. It uses Python as the primary programming language which also makes it more accessible. Bearing these in mind, Qiskit is chosen as the programming language of the Bronze tutorial.
The notebooks in Bronze can be split into 3 categories. There are auxiliary notebooks that should be completed by the participants before coming to the workshop, reviewing basic linear algebra and basics of Python programming. There are reference notebooks on Python and Qiskit as well, to guide the participants whenever needed.
The section reviewing basic linear algebra is titled Basic Math and it provides information on vectors, dot product, matrices and tensor product. Numpy library is not used for linear algebraic operations as a pedagogical strategy, forcing participants to perform matrix-vector multiplications by hand to let then grasp the evolution of states through linear operators. Python review consists of basic structures such as variables, loops, conditionals, and lists. Reference notebooks contain all sufficient information about Qiskit and Python in general.
At the core of Bronze lie around 30 notebooks which cover the following topics: Each notebook consists of some theoretical background about the subject, programming, and conceptual tasks, accompanied by a notebook in which the solutions for the tasks can be found. Some of the first tasks involve only Python programming and the Qiskit library is introduced gradually as new quantum computing concepts come forward. Most of the conceptual tasks involve mathematical derivations which should be completed on paper. Overall, the focus is on practical concepts and quantum programming instead of theoretical proofs. Aimed to serve as introductory material for a 3-day long workshop, some traditional topics such as Deutsch and Bernstein-Vazirani algorithms are omitted.
Next, we will talk about each section in more detail. This is the first section of the material introducing the basics of classical systems. The section starts with classical bits and concentrates on probabilistic bits and states. Vector representation for probabilistic states and their evolution through operators by vector-matrix multiplication is given through a game of coin flips. A notation for probabilistic states is also introduced which builds the foundations of braket notation. The tasks in this section do not involve quantum programming. An example Task from the probabilistic bit notebook is given in Figure 1 . Some other tasks include random probabilistic state generation and simulation of a coin-flipping game and there are also some tasks to be completed on paper such as finding the tensor product of two states. Before moving onto the basics of quantum systems, a notebook describing how to run quantum circuits in Qiskit is provided.
Then comes the section about basics of quantum systems. The section starts with an interferometer setup which is given in Figure 2 . It is presented as a quantum coin-flipping experiment, the quantum analog of coin-flipping previously introduced in the basics of the classical systems section. The following notebooks contain tasks that ask for the implementation of the experimental setups Qiskit. The section also contains notebooks on quantum states, quantum operators, and superposition and measurement.
As the scope of Bronze is restricted to real numbers, the quantum states are elegantly represented in the 2D plane. Tasks in the section include drawing random quantum states in the 2D plane using Python. There is a section on quantum operators with realvalued entries; reflections and rotations involving both programming and visualization tasks. Examples are given in Figure 3 . The next introduced subject is multiple qubits and entanglement. Superdense coding and quantum teleportation protocols are continually discussed alongside with entanglement. Tasks in the section ask for the implementation of both protocols. There also some tasks in which mathematical derivations should be performed such as verification of the superdense coding protocol.
The last section of Bronze is about Grover's search and almost all previous content builds the foundations for this section. The first notebook aims to introduce amplitude amplification idea through an "Inversion about the mean" game whose outputs can be seen in Figure 4 , where the idea of query and inversion phases are applied on a list of elements. One qubit representation of Grover's Search and implementation of query and inversion phases with phase kickback are the next topics that are covered. 
This study aimed to investigate the effectiveness of teaching the basics of quantum computing through quantum programming by assessing the Bronze material based workshops through increase in knowledge of key concepts of participants. One group pre/post-test design has been used for the study. The study consists of pre/post-test results of 22 two/three day workshops organized by QWorld using the Bronze material in 10 countries between May of 2019 and March of 2020, and was finalized due to the global COVID-19 outbreak (detailed information on workshops can be found in Table  2 ). Out of 430 participants who have completed the workshops, 317 participants filled both pre and post-tests.
Participants were provided with ID numbers for them to mark their pre/post-test to allow individual comparison while allowing anonymity. The organizers did not record these participant IDs. An informed consent form was provided with both tests, and it is also verbally noted that participation is voluntary. Ratio of the participants who filled both tests out of the total set of participants is 74%. The form was generated using Google Forms. Gender, age, education level, and departments (current or graduated) were asked as demographic data, and only the department information was a required section to submit the test. Hence any participant that might not want to provide their demographic data in terms of gender, age, or education level was respected.
The questions focused on the circuit based quantum computing model since the educational material runs on Qiskit, the quantum programming language developed by IBM (Asfaw, 2020) , which is developed based on the circuit based model. There are seven questions, one on the base programming language that Qiskit runs on (which is Python), and six on the concepts of quantum computing such as quantum logic gates, qubits, teleportation, superdense coding, and Grovers algorithm. The list of questions can be found below on table 1 (the full questionnaire and the points for each question can be found in the Appendix section). Table 1 .: Graded questions on the pre/post-test.
Answer Type What are the two fundamental quantum phenomena that differentiates quantum computing from classical computing?
What is the programming language that Qiskit runs on?
Multiple Choice Match the quantum logic gates with their respective matrix representations (leave empty if you dont know the subject).
Which of the following elements are not necessary for quantum teleportation? Select all that applies.
Which quantum resource is used for superdense coding?
Multiple Choice What is the common property of probabilistic bits and qubits?
Multiple Choice What is Grovers algorithm used for?
Multiple Choice
The same questions were asked on both of the pre/post-test. The demographic data is collected with the pre-test. In the post-test, we also asked the participants attendance levels and feedback on the workshop to determine whether satisfaction levels have any correlations with the level of increase in knowledge acquisition.
In this section, the main findings of the analyses performed on the pre/post-test data are provided. These findings cover the overall results, mean, gain score, and the normalized gain score for the entire set. Additionally, comparison with respect to gender, age, education level, and educational background (department) are given. The information on the location and dates of the workshops, the number of participants who have completed the workshops, and the number of participants that filled both test can be found on Table 2 .
For each participant, knowledge acquisition is measured by the gain score, which is the difference between the pre/post-test scores. Overall, the mean test score of all participants has increased from 32.32 to 71.09 with a gain score of 38.77. A plot showing the pre/post-test scores of all participants is given in Figure 5 .
Normalized gain score is introduced by (Hake, 1998) as a measure of change to assess the knowledge of the students at the beginning and at the end of a physics course when the same test is used. According to Hake (1998) , normalized gain score (n gain ) is calculated as,
where µ pre and µ post are the average pre/post-test scores respectively. Normalized gain score for all participants is calculated as 0.57. A table summarizing statistics about overall scores is given in Table 3 . Normality of the gain scores is tested with D'Agostino Pearson test as the sample size is large (> 300) (D'Agostino, Belanger, & D'Agostino Jr, 1990 ) and the test resulted in a p-score of 0.110 > 0.05 showing that the gain scores are distributed normal. Visual inspection of the Q-Q plot given which can be found in Figure 6 also supports test results.
We used paired t-test for hypothesis testing and the test result reveals that there is a significant difference between the pre/post-test scores. The results are summarized in Table 4 . We have conducted Shapiro-Wilk normality test for the gain scores acquired at each workshop and used paired t-test or Wilcoxon signed rank test for hypothesis testing depending on the normality of the data. Hypothesis testing yielded that there was a significant increase in the test scores of the participants in all workshops. n gain scores of the participants vary from 0.33 to 0.83. The results are summarized in Table 5 and plotted in Figure 7 4 .
Among the 317 participants who both filled pre/post-test. gender data is available for 310 of them. Summary of the results based on gender data is available in Table 6 .
Gain scores are distributed normal both for male and female participants and two sample t test resulted in a p-value of 0.403 (statistics = -0.837) proving that there is no significant difference between male and female participants in knowledge acquisition. Box-plot for the pre/post-test scores grouped into male and female participants is given in Figure 8 .
Age information is available for 308 participants. To assess whether there is any significant difference in the gain scores depending on the age, participants are grouped into 6 different age groups and the results are summarized in Table 7 . Sample sizes for the age groups 45-54 and 55-60 are too small to make meaningful inferences. For the remaining age groups, data suggests that the most effective learning is realized for the age group 13-17, taking into account both the gain scores and normalized gain.
With age as a grouping variable, not all gain scores are distributed normally according to Shapiro-Wilk normality test. Consequently, we applied Kruskal Wallis test which resulted in a p-value of 0.024 (statistics=12.896), exhibiting a significant dif- ference in the gain scores between the age groups. Box plot of pre/post-test scores grouped into age groups is given in Figure 9 .
We grouped participants into four categories based on their education level by taking into account the highest degree they are pursuing or have obtained: high school, bachelor, masters, and doctorates. Number of participants belonging to each category and summary of the test scores is given in Table 8 When the gain scores are compared, it is observed that the average gain score decreases as the education level increases. For each group, Shapiro-Wilk test proved that the distributions of the gain scores are normal. One-way ANOVA test shows that there is a significant difference between the groups with a p-value of 0.016 (statistics=3.51). When we also take into consideration the pre-test scores, we observe that high school students are still the most effective learners while the n gain scores of the other three groups are very close to each other.
Considering the participants who are not high school students and those who provided information about their current or graduated departments, we have categorized them into five: computer science, physics, engineering, science, and social science. Engineering category consists of the students with backgrounds in electrical and electronical engineering and other engineering departments. Those who do not study physics, computer science and engineering but a science related department is categorized as science. Results are summarized in Table 9 .
Even though the participants with the highest gain score are those with computer science background, those with physics background have the highest n gain score. Shapiro-Wilk test is conducted for the gain scores of each group which reveals that the gain scores are distributed normal. One way ANOVA test resulted in a p-value of 0.811 (statistics=0.397) suggesting that there is no significant difference between the groups.
Post-test consisted of 10 statements related to participant satisfaction measured on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Responses of the 283 participants for each statement is visualized in Figure 10 .
Mean satisfaction scores are ranging from 4.17 to 4.83 and the details can be found in Table 10 . Pearson Correlation Coefficient between the gain scores and satisfaction scores for each participant proves that no correlation exists between the satisfaction level of the participants and their gain scores and details are omitted here.
Depending on our findings presented in the previous section, we can discuss several points regarding the effectiveness of the Bronze material based workshops in teaching quantum computing concepts through quantum programming.
The material has been effective in terms of increasing the knowledge level of participants significantly. Overall n gain score of the participants falls into the region of medium gain. Considering each workshop separately, we see that n gain scores vary in a range between medium and high gain and there is a significant increase in the gain scores in all workshops. We can conclude that the success of the workshops heavily depends on the material as it is the only unchanging factor in 22 workshops organized in 10 countries with different educators and mentors.
The gender gap in STEM is a global problem at all levels of education and in the workforce, and there are various projects developed in Europe trying to close the gap including ones in the area of physics and programming (García-Holgado, Verdugo-Castro, González, Sánchez-Gómez, & García-Peñalvo, 2020) . Even though some workshops are aimed only at women and women are prioritized when the number of applications exceeds the quota, the number of women participants who completed the workshops is approximately one-third of men. There is a significant difference 5 in the pre-test scores of female and male participants reflecting the gap while no significant difference in the gain scores between the two groups was evident.
Overall, the results suggest that hands-on programming experience in learning quantum computing has been beneficial for both male and female participants, and contributes towards closing the gap.
There are several reported experiences in teaching quantum computing to high school students (Hughes et al., 2020; Ozols & Walter, 2018) . Both the analysis conducted on age groups and educational background further supports the importance of reaching high school students. Participants between 13-17 is the group with the highest mean gain score and n gain (excluding the groups with less than 10 participants) and high school students have the highest mean gain score and n gain compared to other groups. We can safely conclude that for high school students with a basic knowledge of programming and linear algebra, Bronze material based workshops are an effective way of teaching quantum computing. The results also indicate that the groups with less prior knowledge on the topic are the ones that benefited the most, which is an expected finding since Bronze is designed as introductory material.
As the questions on the test are about concepts in quantum computing and the material teaches new concepts through programming, participants with physics background have the highest mean pre-test scores and those with CS background have the highest mean gain scores as expected. No significant difference in the gain scores was found between the participants with different backgrounds and this was Further research can be conducted with novel survey design with questions aimed at distinguishing different types of backgrounds (i.e. physics and CS) contribution to learning. Such an analysis might provide insight to developing materials in accordance with the background of participants. Since the Bronze material targets the general audience, such a method was not considered.
There were 10 statements about the satisfaction of the participants which were answered in the Likert scale between 1 and 5 and the averages were above 4.5 for each statement except statement 2. Statement 2 is about whether the participants had enough time to complete the tasks on their own and had an average score of 4.17. This is understandable as the material is quite dense for 3 days and sometimes participants struggle completing the tasks.
Having discussed the findings of the study, let us now consider the threats to the validity of the research. Validity can be divided into four categories (Wohlin, Höst, & Henningsson, 2003) : internal validity, external validity, conclusion validity, construct validity. Next, we will be exploring each one in more detail.
Internal validity is concerned with outside factors that can affect the dependent variable. In single group pre/post design, some of the most common threats to internal validity are listed as history, maturity, instrumentation, statistical regression and attrition (Price & Murnan, 2004) . Workshops were organized either on two or three consecutive days and continued the whole day, not leaving place for any other intervention that might result in history and maturation threat. Instrumentation threat neither poses a challenge, as the performance evaluation metric given in the Appendix section relies on objective measures. We have also considered the normalized gain scores together with the gain scores to overcome "You can only go up from here" phenomenon. Attrition, which is also known as mortality, takes place when some of the participants drop out of the experiment. In our setting, out of 430 participants who completed the workshops, 421 participants answered the pre-test, 317 participants answered both tests, and we have discarded the test results of the participants who only answered one of the tests. When we compare the mean score of the pre-tests with and without discarding the ones who did not answer post-test, the mean scores are 32.32 and 31.11 respectively, not yielding any implication about the pre knowledge of the participants who did not answer post-test.
External validity refers to the generalizability of the results when the same experiment is run with different people, at different times and settings. Even though environmental factors may pose a threat to external validity, we have analyzed data which we have collected from 22 workshops organized in 10 countries by different educators and mentors. The repetitive nature of our data is evidence of the generalizability of our experiment results. A possible threat might be volunteer bias, which stems from the fact that filling the pre/post-test surveys was not compulsory.
Previous studies evaluating volunteer bias have reported that volunteers differ from the rest of the sample in various means (Rosnow & Rosenthal, 1976) , and one might argue that the participants who did not choose to answer are the ones with lower success rates.
Construct validity encompasses the extent to which the experiment and the performance measure captures the construct to be assessed. In this study, we aimed to measure knowledge acquisition, which according to Bloom's Taxonomy of Educational Objectives is the most basic level (Bloom, Englehart, Furst, Hill, & Krathwohl, 1956) . Design of the test questions can be a threat to construct validity. Further attention can be paid to assess the reliability of the test questions in measuring knowledge by using various statistical tests Wells and Wollack (2003) and to improve the test quality in general.
Conclusion validity is concerned with the ability to draw correct conclusions based on the outcomes of the experiment. For hypothesis testing to decide whether there was a significant difference, we first conducted normality tests and then applied paired t-test or Wilcoxon singed rank test depending on the normality of the data. For the remaining post-hoc analysis, we again conducted normality test followed by two sample test, Kruskal Wallis and one-way ANOVA depending on the structure of the data.
The purpose of the current study was to share a methodology in introducing quantum computing to a wider audience by on-site events through task-based quantum programming and computer-science oriented approach and investigate its effectiveness based on the conducted pre-test post-test surveys.
As the field of quantum computing emerges rapidly, and the need for a quantum workforce for industry grows urgent (Ambainis et al., 2017; of Sciences Engineering & Medicine, 2019) , expanding beyond the roots in physics (Z. C. Seskir & Aydinoglu, 2019) is becoming a necessity for educational purposes. Studies focusing on doing this through quantum programming started appearing in the literature recently (Carrascal et al., 2020; LaRose, 2019; Mykhailova & Svore, 2020) , with promising results. However, all are from single study cases, mainly from undergraduate/graduate courses in universities. We have expanded this to a study covering 22 workshops in 10 countries with 317 participants from diverse backgrounds. The research method was single group pre/post-test design where participants answered the same test before and after participating in the workshop. Some demographic data including gender, age, education level, and the department was collected as well as the satisfaction level of the participants. Our analysis has showed that there was a significant increase in the overall test scores of the participants. Another result is that the material is most beneficial for the participants who are beginners in the subject, which is in accordance with the intention of the material. We can conclude that quantum programming through adopting an understanding of quantum computing as a generalized probability theory is an effective tool to introduce quantum computing to an audience from different backgrounds with no prior knowledge in quantum mechanics.
Due to the COVID-19 crisis it has become clear that alternative methods to onsite hands-on training workshops are required for continuing and expanding access to these educational means on quantum computing. A successful example of such an event is the Qiskit Global Summer School 6 , which was planned to be held for 200 participants but due to going online it hosted around 4000 participants from over 100 countries. Assessing the effectiveness of such online events, and using the insight obtained from these assessments to improve the educational process is also important. Quantum computing is a relatively new field, quantum programming is even newer. Therefore, it is safe to assume that we still have much to learn on how to best adjust the introductory level materials for different audiences, mediums, and methods of teaching. Exploring such aspects can yield valuable outcomes that can be utilized in the path forward to the quantum era.
The pre-test (titled as the "Participation Form") contains the following section at the top.
The information below is being collected to enable researchers to estimate disparities in the effectiveness of the workshop you are currently attending. Participation is based on voluntariness. The information you are providing will be only used by researchers in a possible academic publication and only be shared with workshop organizers.
Thank you for your participation.
The section is followed by the items
• Please enter your participation ID no. (required)
• Gender • Age • Write your current department(s) (or the ones that you've graduated most recently) (required)
The education level is obtained via a table of the form Table A1 . For the analysis, we accepted the highest ongoing or graduated level as the "Education" parameter for the participant. 