2022-05-04 13:11:52 INFO Running runs: [] 2022-05-04 13:11:53 INFO Agent received command: run 2022-05-04 13:11:53 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 8 greater_is_better: True hidden_dropout: 0.18004101365999406 language: fr.en learning_rate: 0.0002757119755681108 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-04 13:11:53 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=8 --greater_is_better=True --hidden_dropout=0.18004101365999406 --language=fr.en --learning_rate=0.0002757119755681108 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-04 13:11:58 INFO Running runs: ['qk3ze7ok'] 2022-05-04 13:12:13 INFO Running runs: [] 2022-05-04 13:12:13 INFO Agent received command: run 2022-05-04 13:12:13 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 8 greater_is_better: True hidden_dropout: 0.04999238095195753 language: fr.en learning_rate: 0.0007702133913256148 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-04 13:12:13 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=8 --greater_is_better=True --hidden_dropout=0.04999238095195753 --language=fr.en --learning_rate=0.0007702133913256148 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-04 13:12:18 INFO Running runs: ['o7jpar4x'] 2022-05-04 13:30:33 INFO Running runs: [] 2022-05-04 13:30:33 INFO Agent received command: run 2022-05-04 13:30:33 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 8 greater_is_better: True hidden_dropout: 0.035938233699532036 language: fr.en learning_rate: 0.0003284999261672522 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-04 13:30:33 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=8 --greater_is_better=True --hidden_dropout=0.035938233699532036 --language=fr.en --learning_rate=0.0003284999261672522 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-04 13:30:38 INFO Running runs: ['1tmxz74i'] 2022-05-05 09:24:03 ERROR 500 response executing GraphQL. 2022-05-05 09:24:03 ERROR {"errors":[{"message":"context deadline exceeded"}]} 2022-05-05 09:25:36 ERROR 500 response executing GraphQL. 2022-05-05 09:25:36 ERROR {"errors":[{"message":"context deadline exceeded"}]} 2022-05-05 16:32:16 INFO Cleaning up finished run: 1tmxz74i 2022-05-05 16:32:16 INFO Agent received command: run 2022-05-05 16:32:16 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 8 greater_is_better: True hidden_dropout: 0.18004101365999406 language: fr.en learning_rate: 0.0002757119755681108 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-05 16:32:16 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=8 --greater_is_better=True --hidden_dropout=0.18004101365999406 --language=fr.en --learning_rate=0.0002757119755681108 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-05 16:32:22 INFO Running runs: ['urfp82ib'] 2022-05-05 16:41:31 INFO Cleaning up finished run: urfp82ib 2022-05-05 16:41:33 INFO Agent received command: run 2022-05-05 16:41:33 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 8 greater_is_better: True hidden_dropout: 0.04999238095195753 language: fr.en learning_rate: 0.0007702133913256148 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-05 16:41:33 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=8 --greater_is_better=True --hidden_dropout=0.04999238095195753 --language=fr.en --learning_rate=0.0007702133913256148 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-05 16:41:38 INFO Running runs: ['1zwo1c2h'] 2022-05-05 16:44:52 INFO Cleaning up finished run: 1zwo1c2h 2022-05-05 16:44:53 INFO Agent received command: run 2022-05-05 16:44:53 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 8 greater_is_better: True hidden_dropout: 0.1043496520848404 language: fr.en learning_rate: 0.00023215434357723729 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-05 16:44:53 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=8 --greater_is_better=True --hidden_dropout=0.1043496520848404 --language=fr.en --learning_rate=0.00023215434357723729 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-05 16:44:58 INFO Running runs: ['0sgg5024'] 2022-05-05 16:48:03 INFO Cleaning up finished run: 0sgg5024 2022-05-05 16:48:04 INFO Agent received command: run 2022-05-05 16:48:04 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 8 greater_is_better: True hidden_dropout: 0.16627274455555233 language: fr.en learning_rate: 0.00022154311254852488 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-05 16:48:04 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=8 --greater_is_better=True --hidden_dropout=0.16627274455555233 --language=fr.en --learning_rate=0.00022154311254852488 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-05 16:48:09 INFO Running runs: ['lr7oonrp'] 2022-05-05 16:51:25 ERROR Detected 5 failed runs in a row, shutting down. 2022-05-05 16:51:25 INFO To change this value set WANDB_AGENT_MAX_INITIAL_FAILURES=val 2022-05-05 17:29:38 INFO Running runs: [] 2022-05-05 17:29:38 INFO Agent received command: run 2022-05-05 17:29:38 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 16 greater_is_better: True hidden_dropout: 0.2 language: fr.en learning_rate: 0.0007057712331944904 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-05 17:29:38 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=16 --greater_is_better=True --hidden_dropout=0.2 --language=fr.en --learning_rate=0.0007057712331944904 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-05 17:29:43 INFO Running runs: ['rwbnpkt2'] 2022-05-05 17:29:50 INFO Running runs: [] 2022-05-05 17:29:51 INFO Agent received command: run 2022-05-05 17:29:51 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 16 greater_is_better: True hidden_dropout: 0.2 language: fr.en learning_rate: 0.0005587128574267087 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-05 17:29:51 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=16 --greater_is_better=True --hidden_dropout=0.2 --language=fr.en --learning_rate=0.0005587128574267087 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-05 17:29:56 INFO Running runs: ['ydgnpqx2'] 2022-05-05 17:30:19 INFO Cleaning up finished run: rwbnpkt2 2022-05-05 17:30:19 INFO Agent received command: run 2022-05-05 17:30:19 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 16 greater_is_better: True hidden_dropout: 0.2 language: fr.en learning_rate: 0.0003851276453057612 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-05 17:30:19 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=16 --greater_is_better=True --hidden_dropout=0.2 --language=fr.en --learning_rate=0.0003851276453057612 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-05 17:30:24 INFO Running runs: ['xbi4p92m'] 2022-05-05 17:31:15 INFO Running runs: [] 2022-05-05 17:31:16 INFO Agent received command: run 2022-05-05 17:31:16 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 16 greater_is_better: True hidden_dropout: 0.2 language: fr.en learning_rate: 0.0003287457929573604 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-05 17:31:16 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=16 --greater_is_better=True --hidden_dropout=0.2 --language=fr.en --learning_rate=0.0003287457929573604 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-05 17:31:21 INFO Running runs: ['ykkm1chu'] 2022-05-05 17:36:50 INFO Running runs: [] 2022-05-05 17:36:50 INFO Agent received command: run 2022-05-05 17:36:50 INFO Agent starting run with config: eval_split_name: test eval_steps: 500 evaluation_strategy: steps generation_max_length: 40 generation_num_beams: 1 gradient_accumulation_steps: 16 greater_is_better: True hidden_dropout: 0.2 language: fr.en learning_rate: 3e-05 logging_steps: 1 max_duration_in_seconds: 20 metric_for_best_model: bleu model_name_or_path: ./ num_train_epochs: 3 output_dir: ./ per_device_eval_batch_size: 8 per_device_train_batch_size: 8 save_steps: 500 task: covost2 warmup_steps: 500 2022-05-05 17:36:50 INFO About to run command: python3 run_xtreme_s.py --overwrite_output_dir --freeze_feature_encoder --gradient_checkpointing --predict_with_generate --fp16 --group_by_length --do_train --do_eval --load_best_model_at_end --push_to_hub --use_auth_token --eval_split_name=test --eval_steps=500 --evaluation_strategy=steps --generation_max_length=40 --generation_num_beams=1 --gradient_accumulation_steps=16 --greater_is_better=True --hidden_dropout=0.2 --language=fr.en --learning_rate=3e-05 --logging_steps=1 --max_duration_in_seconds=20 --metric_for_best_model=bleu --model_name_or_path=./ --num_train_epochs=3 --output_dir=./ --per_device_eval_batch_size=8 --per_device_train_batch_size=8 --save_steps=500 --task=covost2 --warmup_steps=500 2022-05-05 17:36:55 INFO Running runs: ['b097rk18'] 2022-05-06 01:57:18 ERROR 500 response executing GraphQL. 2022-05-06 01:57:18 ERROR {"error":"invalid connection"} 2022-05-06 01:58:05 ERROR 500 response executing GraphQL. 2022-05-06 01:58:05 ERROR {"error":"driver: bad connection"} 2022-05-06 07:06:10 ERROR 500 response executing GraphQL. 2022-05-06 07:06:10 ERROR {"error":"driver: bad connection"} 2022-05-06 07:06:21 ERROR 500 response executing GraphQL. 2022-05-06 07:06:21 ERROR {"error":"driver: bad connection"} 2022-05-06 07:06:32 ERROR 500 response executing GraphQL. 2022-05-06 07:06:32 ERROR {"error":"driver: bad connection"} 2022-05-06 07:06:32 ERROR Retry attempt failed: Traceback (most recent call last): File "/home/sanchit_huggingface_co/gcp/lib/python3.9/site-packages/wandb/sdk/lib/retry.py", line 102, in __call__ result = self._call_fn(*args, **kwargs) File "/home/sanchit_huggingface_co/gcp/lib/python3.9/site-packages/wandb/sdk/internal/internal_api.py", line 146, in execute six.reraise(*sys.exc_info()) File "/home/sanchit_huggingface_co/gcp/lib/python3.9/site-packages/six.py", line 719, in reraise raise value File "/home/sanchit_huggingface_co/gcp/lib/python3.9/site-packages/wandb/sdk/internal/internal_api.py", line 140, in execute return self.client.execute(*args, **kwargs) File "/home/sanchit_huggingface_co/gcp/lib/python3.9/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute result = self._get_result(document, *args, **kwargs) File "/home/sanchit_huggingface_co/gcp/lib/python3.9/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in _get_result return self.transport.execute(document, *args, **kwargs) File "/home/sanchit_huggingface_co/gcp/lib/python3.9/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 39, in execute request.raise_for_status() File "/home/sanchit_huggingface_co/gcp/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/graphql 2022-05-06 09:15:13 ERROR 500 response executing GraphQL. 2022-05-06 09:15:13 ERROR {"errors":[{"message":"context deadline exceeded"}]}