Prediction of Video Game Development Problems Based on Postmortems using Different Word Embedding Techniques (2021)

Abstract: The interactive entertainment industry is being actively involved with the development, marketing and sale of video games in the past decade. The increasing interest in video games has led to an increase in video game development techniques and methods. It has emerged as an immensely large sector, and now it has grown to be larger than the movie and music industry combined. The postmortem of a game outlines and analyzes the game’s history, team goals, what went right, and what went wrong with the game. Despite its significance, there is little understanding related to the challenges encountered by the programmers. Postmortems are not properly maintained and are informally written, leading to a lack of trustworthiness. In this study, we perform a systematic analysis on different problems faced in the video game development. The need for automation and ML techniques arises because it could help game developers easily identify the exact problem from the description, and hence be able to easily find a solution. This work could also help developers in identifying frequent mistakes that could be avoided, and will provide researchers a beginning point to further consider game development in context of software engineering.

## introduction the video game industry is engaged in the process of development, promotion, and selling video games. it includes several occupation disciplines and employs a huge number of individuals across the globe. the business has developed from focused markets to the mainstream in recent years. despite being an extremely competitive market where knowledge is the principle weapon, absence of information regarding processes and techniques used in game development makes it hard to understand the game development process. due to this, developers often find it difficult to avoid commonly occurring issues and learn from past faults. the motive behind this work is to classify the dataset based on the quote into types of problems so that future developers could easily recognize the type of problem they are facing and find solution accordingly. however, there are 3 main challenges in this process: to overcome the 3 challenges, a technical analysis and comparative study between the performance of 7 word embedding, 3 feature selection, and 5 classification techniques have been conducted. 5 classification techniques namely, k-nearest neighbours (knn), support vector classifier (svc), naive bayes classifier (nbc), decision tree (dt) and random forest (rf) have been used. finally, the performance of the word embedding and classification techniques are compared. the results obtained from the original data is also compared to the results obtained from smote data. to compare the results, accuracy, f-measure, and area under the curve (auc) are used. box plots are drawn based on auc values since accuracy is not a good measure when the data suffers from class imbalance. finally, rank sum test and friedman's test are used to test the hypotheses. ## related work this section details about studies available in the broad domain of game development. @xcite prepared a grounded dataset obtained from post-mortems of video games which details about various software engineering problems during development. an iterative method has been used to create the dataset. 1035 problems were extracted from more than 200 post-mortems spanning over 20 years . the problems are divided into 3 problem groups (production, business, management) which are further divided into 20 different types. this work utilizes the above stated dataset to understand issues encountered by developers during the process of video-game development and further we foster a model for distinguishing the problem group based on description. game industry problems: an extensive analysis of the gray literature @xcite tries to analyse and develop a state of the problems of the gaming industry, their evolution and root causes which would help researchers and practitioners to work towards addressing and solving these problems. it was observed that the industry suffers from similar proportions of management and production related problems. over the years as the industry became more mainstream, management related problems decreased only to give space to business related problems. technical and design related problems have also decreased over the years. team related problems increased over the last decade, and marketing problems had the biggest increase over the past 23 years. finally, it was concluded that people (and not technology) were the root cause of most problems. callele et al. @xcite analysed the various factors which led to the success or failure of a video game. using the game developer magazine, they analysed 50 post-mortems and investigated how requirements engineering could be applied to game development. "what went right" and "what went wrong" aspects were grouped into 5 categories: (1) pre-production issues (2) internal, and management related problems (3) external problems (4) technological problems (5) scheduler related problems. finally, it was concluded that the transition from preproduction to production plays a crucial role in deciding the fate (success/failure) of the developed video game. 20 post-mortems were taken from the gamasutra website and were analysed by petrillo et al. @xcite the most common game development problems were identified and they were compared with traditional software-engineering problems. it was concluded that (1) management (and not technical) related issues contribute most to the video-game development problems (2) problems faced in video-game development and traditional software development are very similar and, (3) the most common problems are related to scope, feature creep, and cutting features. washburn et al. @xcite analysed 155 game development post-mortems for what went right and wrong. various characteristics of game development have been identified, linked with positive and negative experiences and a set of best practices, pitfalls for game development have been distilled. design aspects cover all situations and decisions that were made that are external to the direct team and development process. production issues relate to scheduling and work prioritizing issues. other aspects include art, programming and testing issues. ## study design this section enlists the information about different design plans utilized in this work. ## research methodology the motive behind this work is to do a technical analysis and comparison between performance of 7 word embedding and 5 classification techniques on game development problems. the algorithm tries to classify the dataset based on the quote into types of problem so that future developers could easily recognize the type of problem they are facing and find solution accordingly. firstly, 7 word embedding techniques are applied on text available in the dataset to obtain feature vectors. next, data imbalance problem was dealt with using smote technique which adds more minority test cases and balances data. dimensionality reduction and feature selection was done using pca, lda and anova. finally, 5 classification techniques were used to predict class of the test data using embedded vectors. the performance of various word embedding and classification techniques are compared. the results obtained from original data are also compared to that obtained from smote data using accuracy, f-measure and auc. box plots are drawn based on the auc values since accuracy is not a good measure when the data suffers from class imbalance @xcite . rank sum test and friedman's test are used to test the hypotheses. framework of the proposed work is depicted in figure 1 . ## empirical results and analysis in this work, 7 word embedding, 1 sampling, 3 feature selection, and 5 ml classifiers were applied to develop models which predict the group of game development problem. each word-embedding is applied on the chosen dataset and its effectiveness is evaluated using classifiers. the predicted values for each of the word embedding and classification techniques have been tabulated in • word2vec embedding gives the best results. ## comparative analysis in this section, we compare the performance of models built using 7 word embedding, smote, 3 feature selection, & 5 classification techniques. descriptive statistics, box-plots, & significant tests have been used to compare developed models. ## conclusion a postmortem is a summarization procedure used to analyse the various positive and negative aspects of the game development project. it aids developers in drawing meaningful conclusions and helps them learn from past successes and failures. however, given the various responsibilities of a video game developer, it is not surprising that they hardly take time to conduct and prepare project postmortems. moreover, the lack of formal structure leads to a lack of trust worthiness. in this work, a data set of the different problems in game development has been taken and studied. 7 word embedding techniques have been applied on the data set and it can be observed that word2vec gives the best results and these results are significantly different from skipgram, cbow and fasttext. as far as the classification techniques are concerned, knn, svc, and rf produce significantly better results. rf gives the best value in friedman's rank sum test. however, knn may also be considered since it takes the lowest computation time amongst knn, svc, rf whilst still producing similar results. lda is the best suitable feature selection technique here, followed by anova. finally, it is important to note that the data obtained after smote gives significantly better yield than the original data, and hence, accounting for class imbalance is crucial. this work, thus provides a way to classify the dataset based on the quote into types of problems. this could help future developers easily recognize the type of problem they are facing and find suitable solutions.