Toloka
Toloka is a crowdsourcing platform and microtasking project launched by Yandex in 2014 to quickly markup large amounts of data, which are then used for machine learning and improving search algorithms. The proposed tasks are usually very simple and do not require any special training from the performer. Most of the tasks are designed to improve algorithms that are used by modern technologies spanning self-driving vehicles, smart web searches, advanced voice assistants and e-commerce. For the correct completion of each task, the performer receives a small reward. About Toloka. Origin of the platform's name. A used  to be a form of mutual assistance among villagers of Russia, Ukraine, Belarus, Estonia, Latvia, and Lithuania. It was organized in villages to perform urgent work requiring a large number of workers, such as harvesting, logging, building houses, etc. Sometimes a toloka was used for community works (building churches, schools, roads, etc.). The idea of joint useful work for a common result is consistent with the principle of crowdsourcing. Types of tasks and scope of results. Improving search algorithms. Data labeling helps to improve search quality and effectively tune result ranking algorithms. For example, in August 2017, Yandex introduced a  new search algorithm called "Korolyov". It's a neural network in creation of which, to train neural networks, the estimates of «tolokers» were used. In more than two years of service, the tolokers have put about two billion ratings, allowed developers to significantly improve the algorithm. Machine learning. To "teach" a machine to solve a particular problem, it is necessary to give it a huge number of examples, both positive and negative. Among other tasks, users of the Toloka service can receive ones to describe objects defined by a computer in an image. For example, the photo shows objects that  artificial intelligence was able to recognize, but in order to use this data in work, precise information is needed about what kind of objects they are. Artificial intelligence can be trained on the mass of such recognitions provided by humans, who, in turn, will receive encouragement for their work. In tasks of another type, a context of the dialogue is given and a scale is proposed by which it is necessary to assess whether a chatbot's answer in this context is appropriate, interesting, and so on. Another group of tasks in Toloka is checking automatically received drafts of dictionary  entries for Yandex.Translate. Audit and marketing research. Checking the quality of the online store, delivery service, writing reviews about products and services. For example, in 2017, Toloka users took part in checking the quality of the product return service on . Such experiments allow to control the quality of the service and identify weaknesses, over which work will be carried out in the future to improve and eliminate the identified problems. In another project, users were asked to write a review for a product from popular categories and provide it with their own photos. Choosing designs and names for new  products and services. Service users also answer questions related to the choice of the most successful name and design for a new product or service, personal or consumer preferences of users. In one of these polls, tolokers chose a name for Yandex's voice assistant. The purpose of the assignment was to determine the character traits of a girl with one name or another. For the survey, they took both the necessary characteristics and completely inappropriate ones, and began to ask: if a girl has such a name, is she rather kind or aggressive? The name "Alice" won the poll  by a wide margin. Payment and money withdrawal. Completion of tasks in Toloka are paid for - the corresponding amount is indicated next to each task. The amount they can earn is indicated next to the task. It is possible to withdraw funds using several payment systems: YooMoney, PayPal, Skrill, Papara, Payoneer. Also, money can be withdrawn to the mobile phone account of Russian operators MTS, MegaFon, Beeline, Tele2. At the stage of debugging processes on the part of Toloka, withdrawal of money could take up to 30 days, at present - from several minutes to several days. Toloka mobile apps. To  solve problems that require work "in the field" (for example, go to the address and check the relevance of information about the organization), there is a mobile version of Toloka. The service has two app versions - for Android and iOS. Users. Toloka users, also known as performers or tolokers, are people who earn money by completing system testing and improvement tasks on the Toloka crowdsourcing platform. There are currently about 9 million registered Toloka users from Russia, Ukraine, Belarus, Kazakhstan, Uzbekistan, Turkey, France, Indonesia, India and other countries, who have completed a total of more than 2 billion evaluations. In  2018, more than a million people participated in Toloka projects. Most performers are young people under 35 (usually engineering students or mothers on maternity leave). Performers mainly see Toloka as an additional source of income, but many of them note that they like to do meaningful work and clean up the internet. Requesters. All tasks in Toloka are placed by requesters. The main requesters are various Yandex teams. The main uses of Toloka within Yandex projects are data collection and processing for machine learning, speech technology, computer vision, smart search algorithms, and other projects, as well as content moderation (conversations  in Yandex Maps), field tasks (updating information about businesses and organizations in Yandex Business Directory), optimization of internal business processes (searching for posts on social media, labeling mentions of Yandex by tone as positive or negative). Since at least hundreds or even thousands of tolokers are simultaneously working on the problem, the results are ready in the worst case the next day. In April 2019, Toloka launched an affiliate program, within the framework of which it will redirect external customers to partner companies (or individual performers) to prepare relevant projects and complete tasks. Toloka Research. Toloka is both a tool  for solving applied problems and a platform for scientific research in various subject areas. In May 2019, the service's team started publishing datasets for non-commercial and academic purposes to support the scientific community and attract researchers to Toloka. Such datasets are addressed to researchers in different directions like linguistics, computer vision, testing of result aggregation models, and chatbot training. Toloka research has been showcased at a range of conferences, including the Conference on Neural Information Processing Systems (NeurIPS), the International Conference on Machine Learning (ICML) and the International Conference on Very Large Data Bases (VLDB).