| # Title: Code Optimization Using Character-Level Transformers: A Feasibility Study | |
| # Experiment description: ['1. Collect or create a dataset of code snippets and their optimized versions. Focus on Python code and include pairs of unoptimized and optimized code snippets that demonstrate performance improvements.', '2. Preprocess the code snippets to fit the character-level format required by the model. This includes tokenizing the code into characters and converting them into numerical IDs.', '3. Modify the training script to include a new loss function that incorporates the optimization task. Use a sequence-to-sequence (seq2seq) setup where the model is trained to generate optimized code snippets from unoptimized inputs.', '4. Train the model on the code optimization dataset. Use a validation set to monitor the performance and avoid overfitting. Implement early stopping based on validation loss.', "5. Evaluate the model's ability to generate optimized code by comparing the generated code snippets with the optimized versions in the dataset. Use metrics such as syntax correctness, performance improvements (e.g., execution time, memory usage), and readability to assess the quality of the optimized code.", "6. Analyze the results to determine the feasibility and accuracy of the model for code optimization. Identify any patterns or insights that can be drawn from the model's performance.", '7. Compare the performance of the character-level model with a word-level model on the same task to highlight any advantages or disadvantages. Use the same evaluation metrics for a fair comparison.'] | |
| ## Run 0: Baseline | |
| Results: {'shakespeare_char': {'final_train_loss_mean': 0.8173830509185791, 'best_val_loss_mean': 1.4637625217437744, 'total_train_time_mean': 92.05195260047913, 'avg_inference_tokens_per_second_mean': 697.3658396135052}} | |
| Description: Baseline results. | |