Initial

README.md

@@ -335,4 +335,7 @@ $ cat ./Scripts/Exp/Perf/Fig10.csv
 
 ## 8. Experiment Customization
 
-Users can run this experiment in different software environments, but they must ensure that PyTorch version is compatible with the CUDA version in those software environments. The experiment can also be conducted in different hardware environments, but adjustments to the batch size for fine-tuning and inference are necessary based on the available GPU memory. We have fixed the random seed and parameters in the provided scripts to ensure consistent code generation accuracy within the same hardware and software environment. However, when the experiment is executed in different hardware or software environments, the accuracy may experience some fluctuations.
+Users can run this experiment in different software environments, but they must ensure that PyTorch version is compatible with the CUDA version in those software environments. The experiment can also be conducted in different hardware environments, but adjustments to the batch size for fine-tuning and inference are necessary based on the available GPU memory. We have fixed the random seed and parameters in the provided scripts to ensure consistent code generation accuracy within the same hardware and software environment. However, if the model is re-fine-tuned under different hardware or software environments, the accuracy of the newly fine-tuned model may exhibit slight variations.
+
+
+**We further conducted code generation tests on a machine with an Nvidia A100 GPU (80GB memory). With a consistent software environment, the experimental results demonstrated a reduction in the time overhead of the code generation process (Fig. 7), as the previous setup with 8 V100 GPUs incurred higher time overhead due to the need for synchronization between multiple GPUs. However, the code accuracy remained unchanged (Fig. 8, Table. 2, Fig. 9, Table. 3). This confirms that our experiment can also be executed across different hardware environments.**

Scripts/UnixCoder/run_one_model.py

@@ -359,9 +359,6 @@ def vega_train_main():
 
     # print arguments
     args = parser.parse_args()
-    # set log
-    logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s -   %(message)s',
-                        datefmt='%m/%d/%Y %H:%M:%S', level=logging.INFO)
     # set device
     device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
     args.n_gpu = torch.cuda.device_count()
@@ -395,7 +392,6 @@ def vega_train_main():
                     beam_size=args.beam_size, max_length=args.max_target_length,
                     sos_id=tokenizer.convert_tokens_to_ids(["<mask0>"])[0], eos_id=tokenizer.sep_token_id)
 
-    logger.info("Training/evaluation parameters %s", args)
     model.to(args.device)
 
     if args.n_gpu > 1:
@@ -642,7 +638,7 @@ def vega_train_main():
         checkpoint_prefix = 'checkpoint-best-acc/pytorch_model.bin'
         output_dir = os.path.join(args.output_dir, checkpoint_prefix)
         model_to_load = model.module if hasattr(model, 'module') else model
-        model_to_load.load_state_dict(torch.load(output_dir))
+        model_to_load.load_state_dict(torch.load(output_dir), strict=False)
 
         eval_examples_all = read_examples(args.test_filename, args.do_function_test)
         eval_examples = read_examples_no_bracket(args.test_filename, args.do_function_test)