Telegram ML Competition 2023: TonRadar Solution
Tags: telegram, machinelearning, python, azure
Published: 2023-10-22 23:23:03

This document explains how **TonRadar Team** developed a solution for the **[ML Competition 2023](https://contest.com/docs/ML-Competition-2023)** organized by Telegram. 

> The goal of the competition is to create a library that can detect the programming language of a given source code snippet. 

The complete source code and documentation are available on our GitHub repository: [github.com/tonradar/contest-tglang](^1^).

## Task Breakdown
We divided our work into three teams to win this contest. Each team had a specific role and responsibility:
 - **Crawler Team:** This team was responsible for crawling source codes from GitHub and preparing an appropriate dataset. They ensured that the dataset had at least 1000 samples for each language.
 - **Model Team:** This team was responsible for training a model using the prepared dataset and optimizing it to reach the maximum accuracy. They experimented with different architectures and hyperparameters to find the best model.
 - **LibTgLang Team:** This team was responsible for integrating the trained model as a `.so` library that can be referenced in C. They also made sure that the library could predict the language in less than 10ms.

## Source Code
You can find the resulting source code for each team:
 - **[crawler](https://github.com/tonradar/contest-tglang/tree/main/src/crawler):** Crawling code files from GitHub (C#)
 - **[model](https://github.com/tonradar/contest-tglang/tree/main/src/model):** Training the model (Python, AzureML)
 - **[libtglang](https://github.com/tonradar/contest-tglang/tree/main/src/libtglang):** Building the final *.so* file (C)

## Language Selection
We analyzed the list of languages provided by the competition and decided to exclude some of them for the following reasons:
 - Some languages are not widely used or not even real languages (e.g., *1S_ENTERPRISE*, *BISON*).
 - Some languages are very similar to other languages and can confuse the model (e.g., *Delphi~Pascal* and *VISUAL_BASIC~VBSCRIPT*).
 - Some languages have very few or no source code files available on GitHub (e.g., *HACK*).

 Here is the list of **excluded** languages:
 - **1S_ENTERPRISE**
 - **BISON**
 - **HACK**
 - **DELPHI** (will map to PASCAL)
 - **VBSCRIPT** (will map to VISUAL_BASIC)
  
We also chose to use `.aspx` files instead of `.asp` files for the `ASP` language, as `.aspx` is the newer and more popular framework.

## Data Collection
We used GitHub as our main source of data for training our model. We developed a project called **TgLang.CodeCrawler** that can crawl GitHub and download 1000 source code files for each language. The project is written in C# and consists of two parts:
  - **TgLang.CodeCrawler**: A console application that crawls GitHub and saves the files in a folder named after the language.
  - **TgLang.CodeCrawler.Test**: A set of unit tests and integration tests to ensure the correctness and reliability of the crawler.

## Model Training
We used Python and AzureML to train our model using the collected dataset. We chose a deep neural network architecture that can learn from the syntactic and semantic features of the source code. We saved our model in ONNX format so that we could use it in a C program to create the final library.

We trained the model using AzureML and here is the result:
![Model Metrics](https://github.com/tonradar/contest-tglang/assets/5070766/bc169a40-ab77-480c-9997-9c1e9ba0c0fd)

During training, we observed that some languages caused confusion using the *Confusion Matrix*:
![Confusion Matrix](https://github.com/tonradar/contest-tglang/assets/5070766/eb391d21-f784-4fe9-82c2-08f411e63c0f)

As you can see in this confusion matrix, some blue columns show a high possibility of confusing most languages with the language of that column. 

These **confusing languages** are:
 - JSON
 - REGEX
 - CSV

A common pattern in these languages is that they can be used within other languages. For example, a regex pattern or a JSON value can be used in any language.

## Final Library
The final library is written in C and conforms to the specifications of the competition. It loads the ONNX model and uses it to predict the programming language of an input string. It returns the related enum of the predicted language as output.

## Accuracy and Performance
We achieved an accuracy of **87.14%** (Weighted Accuracy of **99.29%**) and our function is answering in an average of **9ms**, which is amazing!