init

Optimus/.gitignore

@@ -0,0 +1,8 @@
+data/datasets/glue_data/glue_data
+data/datasets/glue_data/train.tx
+data/datasets/glue_data/cached_lm_gpt_bert_256_train.jsont
+code/runs
+output/*
+code/pytorch_transformers/__pycache__/*
+code/examples/big_ae/modules/encoders/__pycache__/*
+

Optimus/README.md

@@ -0,0 +1,121 @@
+# Optimus: the first pre-trained Big VAE language model <img src="doc/figs/logo_optimus.png" width="100" align="right">  
+ 
+This repository contains source code necessary to reproduce the results presented in the EMNLP 2020 paper [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://arxiv.org/abs/2004.04092).
+
+
+|<img src="doc/figs/optimus_scheme.png" width="350"> | <img src="doc/figs/headfig_optimus.png" width="800"> 
+|-------------------------|:-------------------------:|
+| The network architecture of Optimus: encoder for representation learning and decoder for generation | Sentences are organized and manipulated in a pre-trained compact and smooth latent space 
+
+
+For more on this project, see the [Microsoft Research Blog post](https://www.microsoft.com/en-us/research/blog/a-deep-generative-model-trifecta-three-advances-that-work-towards-harnessing-large-scale-power/).
+
+
+## News
+
+May 21, 2020: Releasing a [`demo`](http://40.71.23.172:8899/) for latent space manipulation, including sentence interpolation and analogy. Check out the [`website`](http://40.71.23.172:8899/).
+
+May 20, 2020: The latent space manipulation code is cleaned and released. See instructions at [`optimius_for_snli.md`](doc/optimius_for_snli.md).
+
+May 13, 2020: The fine-tuning code for langauge modeling is released. See instructions  at [`optimus_finetune_language_models.md`](doc/optimus_finetune_language_models.md)
+
+## Contents
+There are four steps to use this codebase to reproduce the results in the paper.
+
+1. [Dependencies](#dependencies)
+2. [Prepare datasets](#prepare-datasets)
+3. [Model training](#Model-training)
+    1. Pre-training on setences in Wikipedia
+    2. Languange Modeling
+    3. Guided Language Generation
+    4. Low-resource Language Understanding
+4. [Collect and plot results](#collect-and-plot-results)
+
+
+## Dependencies
+
+Pull docker from Docker Hub at: `chunyl/pytorch-transformers:v2`. Please see the instruction at [`doc/env.md`](doc/env.md)
+
+The project is organized into the following structures, with ensential files & folders visualized.  `output` saves the models checkpoints.
+```
+├── Optimus
+   └── code
+       ├── examples
+           ├── big_ae
+               ├── modules
+                   ├── vae.py
+                   └── ...
+               ├── run_lm_vae_pretraining_phdist_beta.py
+               ├── run_lm_vae_training.py
+               └── ...
+	   ├── pytorch_transformers
+               ├── modeling_bert.py
+               ├── modeling_gpt2.py
+               └── ...
+       ├── scripts
+           ├── scripts_docker
+	   ├── scripts_local
+	   ├── scripts_philly
+   └── data
+       └── datasets
+           ├── wikipedia_json_64_filtered
+               └── ...
+	   ├── snli_data
+           └── ...
+   └── output
+       ├── pretrain
+       ├── LM
+       └── ...       
+```
+
+## Prepare Datasets
+
+Please download or preparation the data via following the instructions at [`data/download_datasets.md`](data/download_datasets.md). 
+
+## Model Training
+
+**1. Pre-training on setences in Wikipedia**
+
+We pre-trained our models on Philly (a Microsoft internal compute cluster), the code is specialized for multi-node multi-GPU compute on this platform. The pre-training main python is [`run_lm_vae_pretraining_phdist_beta.py`](code/examples/big_ae/run_lm_vae_pretraining_phdist_beta.py). You may need to adjust the distributed training scripts. 
+
+**2. Languange Modeling**
+
+To have a fair comparison with existing VAE languange models, we consider a model with latent dimension 32. The pre-trained model is fine-tuned on four commonly datasets for one epoch. Please see the details at [`doc/optimus_finetune_language_models.md`](doc/optimus_finetune_language_models.md)
+
+**3. Guided Language Generation**
+
+
+**Latent Space Manipulation** To ensure good performance, we consider a model with latent dimension 768. The pre-trained model is fine-tuned on SNLI dataset, where sentences show related patterns. Please see the details at 
+Please see the details at [`doc/optimius_for_snli.md`](doc/optimius_for_snli.md)
+
+**4. Low-resource Language Understanding**
+
+## Collect and Plot Results
+
+Once the networks are trained and the results are saved, we extracted key results using Python script. The results can be plotted using the included IPython notebook `plots/main_plots.ipynb`.
+Start the IPython Notebook server:
+
+```
+$ cd plots
+$ ipython notebook
+```
+
+Select the `main_plots.ipynb` notebook and execute the included
+code. Note that without modification, we have copyed our extracted results into the notebook, and script will output figures in the paper. If you've run your own training and wish to plot results, you'll have to organize your results in the same format instead.
+
+
+## Questions?
+
+Please drop me ([Chunyuan](http://chunyuan.li/)) a line if you have any questions.
+
+
+```
+@inproceedings{li2020_Optimus,
+  title={Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space},
+  author={Li, Chunyuan and Gao, Xiang and Li, Yuan and Li, Xiujun and Peng, Baolin and Zhang, Yizhe and Gao, Jianfeng},
+  booktitle={EMNLP},
+  year={2020}
+}
+```
+
+

Optimus/code/README.md

@@ -0,0 +1,41 @@
+## Set up Environment
+
+Pull docker from Docker Hub at: chunyl/pytorch-transformers:v2
+
+Edit the project path to the absolute path on your computer by changing the "SCRIPTPATH" in [run_docker.sh](./scripts/scripts_docker/run_docker.sh)
+
+In this directory ("code"), and run docker
+
+    sh scripts/scripts_docker/run_docker.sh
+    
+    
+
+  
+## Fine-tune Language Models
+
+    sh scripts/scripts_local/run_ft_lm_vae_optimus.sh
+    
+    
+The main training script is [`run_lm_vae_training.py`](./examples/big_ae/run_lm_vae_training.py) and conducts the fine-tuning loop, taking the following options (among others) as arguments:
+
+- `--checkpoint_dir`: the folder that the pre-trained Optimus is saved.
+- `--gloabl_step_eval`: it specifies the checkpoint (the steps that Optimus is trained).
+- `--train_data_file` and `--eval_data_file`: the path for training and testing datasets for the downstream fine-tuning.
+- `--dataset`: the dataset for fine-tuning. such as `Penn`
+- `--num_train_epochs`: number of training epochs (type=int); default 1.
+- `--dim_target_kl`:   the hyper-paramter used in dimension-wise thresholding used in fine-tuning(type=float); default 0.5.
+- `--beta`:   the maximum beta value used in cyclical annealing schedule used in fine-tuning(type=float); default 1.0.
+- `--ratio_zero`:   the proportion of beta=0 in one period for fine-tuning(type=float); default 0.5
+- `--ratio_increase`:  the proportion of beta that increases from 0 to the maximum value in one period in cyclical annealing schedule used in fine-tuning(type=float); default 0.25.
+
+
+For more options, please see [`run_lm_vae_training.py`](./examples/big_ae/run_lm_vae_training.py) and  see the examples we provided in [`run_ft_lm_vae_optimus.sh`](./scripts/scripts_local/run_ft_lm_vae_optimus.sh), or [more running scripts we used to run the code on a cluster](./scripts/scripts_philly).
+
+
+## Play with the latent space
+
+    sh scripts/scripts_local/eval_optimus_latent_space.sh
+    
+The main training script is [`run_latent_generation.py`](./examples/big_ae/run_latent_generation.py) and evaluates the various ways to generate text conditioned on latent vectors, taking the following options (among others) as arguments:
+
+- `--play_mode`:  The current scripts supports two ways to play with the pre-trained VAE models: [`reconstrction`, `interpolation`]

Optimus/code/examples/README.md

@@ -0,0 +1,392 @@
+# Examples
+
+In this section a few examples are put together. All of these examples work for several models, making use of the very
+similar API between the different models.
+
+| Section                    | Description                                                                                                                                                |
+|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Language Model fine-tuning](#language-model-fine-tuning) | Fine-tuning the library models for language modeling on a text dataset. Causal language modeling for GPT/GPT-2, masked language modeling for BERT/RoBERTa. |
+| [Language Generation](#language-generation) | Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL and XLNet.                                         |
+| [GLUE](#glue) | Examples running BERT/XLM/XLNet/RoBERTa on the 9 GLUE tasks. Examples feature distributed training as well as half-precision.                              |
+| [SQuAD](#squad) | Using BERT for question answering, examples with distributed training.                                                                                  |
+| [Multiple Choice](#multiple choice) | Examples running BERT/XLNet/RoBERTa on the SWAG/RACE/ARC tasks. 
+
+## Language model fine-tuning
+
+Based on the script [`run_lm_finetuning.py`](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_lm_finetuning.py).
+
+Fine-tuning the library models for language modeling on a text dataset for GPT, GPT-2, BERT and RoBERTa (DistilBERT 
+to be added soon). GPT and GPT-2 are fine-tuned using a causal language modeling (CLM) loss while BERT and RoBERTa 
+are fine-tuned using a masked language modeling (MLM) loss.
+
+Before running the following example, you should get a file that contains text on which the language model will be
+fine-tuned. A good example of such text is the [WikiText-2 dataset](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/).
+
+We will refer to two different files: `$TRAIN_FILE`, which contains text for training, and `$TEST_FILE`, which contains
+text that will be used for evaluation.
+
+### GPT-2/GPT and causal language modeling
+
+The following example fine-tunes GPT-2 on WikiText-2. We're using the raw WikiText-2 (no tokens were replaced before
+the tokenization). The loss here is that of causal language modeling.
+
+```bash
+export TRAIN_FILE=/path/to/dataset/wiki.train.raw
+export TEST_FILE=/path/to/dataset/wiki.test.raw
+
+python run_lm_finetuning.py \
+    --output_dir=output \
+    --model_type=gpt2 \
+    --model_name_or_path=gpt2 \
+    --do_train \
+    --train_data_file=$TRAIN_FILE \
+    --do_eval \
+    --eval_data_file=$TEST_FILE
+```
+
+This takes about half an hour to train on a single K80 GPU and about one minute for the evaluation to run. It reaches
+a score of ~20 perplexity once fine-tuned on the dataset.
+
+### RoBERTa/BERT and masked language modeling
+
+The following example fine-tunes RoBERTa on WikiText-2. Here too, we're using the raw WikiText-2. The loss is different
+as BERT/RoBERTa have a bidirectional mechanism; we're therefore using the same loss that was used during their
+pre-training: masked language modeling. 
+
+In accordance to the RoBERTa paper, we use dynamic masking rather than static masking. The model may, therefore, converge
+slightly slower (over-fitting takes more epochs).
+
+We use the `--mlm` flag so that the script may change its loss function.
+
+```bash
+export TRAIN_FILE=/path/to/dataset/wiki.train.raw
+export TEST_FILE=/path/to/dataset/wiki.test.raw
+
+python run_lm_finetuning.py \
+    --output_dir=output \
+    --model_type=roberta \
+    --model_name_or_path=roberta-base \
+    --do_train \
+    --train_data_file=$TRAIN_FILE \
+    --do_eval \
+    --eval_data_file=$TEST_FILE \
+    --mlm
+```
+
+## Language generation
+
+Based on the script [`run_generation.py`](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_generation.py).
+
+Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL and XLNet.
+A similar script is used for our official demo [Write With Transfomer](https://transformer.huggingface.co), where you
+can try out the different models available in the library.
+
+Example usage:
+
+```bash
+python run_generation.py \
+    --model_type=gpt2 \
+    --model_name_or_path=gpt2
+```
+
+## GLUE
+
+Based on the script [`run_glue.py`](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py).
+
+Fine-tuning the library models for sequence classification on the GLUE benchmark: [General Language Understanding 
+Evaluation](https://gluebenchmark.com/). This script can fine-tune the following models: BERT, XLM, XLNet and RoBERTa. 
+
+GLUE is made up of a total of 9 different tasks. We get the following results on the dev set of the benchmark with an
+uncased  BERT base model (the checkpoint `bert-base-uncased`). All experiments ran on 8 V100 GPUs with a total train
+batch size of 24. Some of these tasks have a small dataset and training can lead to high variance in the results
+between different runs. We report the median on 5 runs (with different seeds) for each of the metrics.
+
+| Task  | Metric                       | Result      |
+|-------|------------------------------|-------------|
+| CoLA  | Matthew's corr               | 48.87       |
+| SST-2 | Accuracy                     | 91.74       |
+| MRPC  | F1/Accuracy                  | 90.70/86.27 |
+| STS-B | Person/Spearman corr.        | 91.39/91.04 |
+| QQP   | Accuracy/F1                  | 90.79/87.66 |
+| MNLI  | Matched acc./Mismatched acc. | 83.70/84.83 |
+| QNLI  | Accuracy                     | 89.31       |
+| RTE   | Accuracy                     | 71.43       |
+| WNLI  | Accuracy                     | 43.66       |
+
+Some of these results are significantly different from the ones reported on the test set
+of GLUE benchmark on the website. For QQP and WNLI, please refer to [FAQ #12](https://gluebenchmark.com/faq) on the webite.
+
+Before running anyone of these GLUE tasks you should download the
+[GLUE data](https://gluebenchmark.com/tasks) by running
+[this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)
+and unpack it to some directory `$GLUE_DIR`.
+
+```bash
+export GLUE_DIR=/path/to/glue
+export TASK_NAME=MRPC
+
+python run_glue.py \
+  --model_type bert \
+  --model_name_or_path bert-base-cased \
+  --task_name $TASK_NAME \
+  --do_train \
+  --do_eval \
+  --do_lower_case \
+  --data_dir $GLUE_DIR/$TASK_NAME \
+  --max_seq_length 128 \
+  --per_gpu_train_batch_size 32 \
+  --learning_rate 2e-5 \
+  --num_train_epochs 3.0 \
+  --output_dir /tmp/$TASK_NAME/
+```
+
+where task name can be one of CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, WNLI.
+
+The dev set results will be present within the text file `eval_results.txt` in the specified output_dir. 
+In case of MNLI, since there are two separate dev sets (matched and mismatched), there will be a separate 
+output folder called `/tmp/MNLI-MM/` in addition to `/tmp/MNLI/`.
+
+The code has not been tested with half-precision training with apex on any GLUE task apart from MRPC, MNLI, 
+CoLA, SST-2. The following section provides details on how to run half-precision training with MRPC. With that being 
+said, there shouldn’t be any issues in running half-precision training with the remaining GLUE tasks as well, 
+since the data processor for each task inherits from the base class DataProcessor.
+
+### MRPC
+
+#### Fine-tuning example
+
+The following examples fine-tune BERT on the Microsoft Research Paraphrase Corpus (MRPC) corpus and runs in less 
+than 10 minutes on a single K-80 and in 27 seconds (!) on single tesla V100 16GB with apex installed.
+
+Before running anyone of these GLUE tasks you should download the
+[GLUE data](https://gluebenchmark.com/tasks) by running
+[this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)
+and unpack it to some directory `$GLUE_DIR`.
+
+```bash
+export GLUE_DIR=/path/to/glue
+
+python run_glue.py \
+  --model_type bert \
+  --model_name_or_path bert-base-cased \
+  --task_name MRPC \
+  --do_train \
+  --do_eval \
+  --do_lower_case \
+  --data_dir $GLUE_DIR/MRPC/ \
+  --max_seq_length 128 \
+  --per_gpu_train_batch_size 32 \
+  --learning_rate 2e-5 \
+  --num_train_epochs 3.0 \
+  --output_dir /tmp/mrpc_output/
+```
+
+Our test ran on a few seeds with [the original implementation hyper-
+parameters](https://github.com/google-research/bert#sentence-and-sentence-pair-classification-tasks) gave evaluation 
+results between 84% and 88%.
+
+#### Using Apex and mixed-precision
+
+Using Apex and 16 bit precision, the fine-tuning on MRPC only takes 27 seconds. First install 
+[apex](https://github.com/NVIDIA/apex), then run the following example:
+
+```bash
+export GLUE_DIR=/path/to/glue
+
+python run_glue.py \
+  --model_type bert \
+  --model_name_or_path bert-base-cased \
+  --task_name MRPC \
+  --do_train \
+  --do_eval \
+  --do_lower_case \
+  --data_dir $GLUE_DIR/MRPC/ \
+  --max_seq_length 128 \
+  --per_gpu_train_batch_size 32 \
+  --learning_rate 2e-5 \
+  --num_train_epochs 3.0 \
+  --output_dir /tmp/mrpc_output/ \
+  --fp16
+```
+
+#### Distributed training
+
+Here is an example using distributed training on 8 V100 GPUs. The model used is the BERT whole-word-masking and it
+reaches F1 > 92 on MRPC.
+
+```bash
+export GLUE_DIR=/path/to/glue
+
+python -m torch.distributed.launch \
+    --nproc_per_node 8 run_glue.py \
+    --model_type bert \
+    --model_name_or_path bert-base-cased \
+    --task_name MRPC \
+    --do_train \
+    --do_eval \
+    --do_lower_case \
+    --data_dir $GLUE_DIR/MRPC/ \
+    --max_seq_length 128 \
+    --per_gpu_train_batch_size 8 \
+    --learning_rate 2e-5 \
+    --num_train_epochs 3.0 \
+    --output_dir /tmp/mrpc_output/
+```
+
+Training with these hyper-parameters gave us the following results:
+
+```bash
+acc = 0.8823529411764706
+acc_and_f1 = 0.901702786377709
+eval_loss = 0.3418912578906332
+f1 = 0.9210526315789473
+global_step = 174
+loss = 0.07231863956341798
+```
+
+### MNLI
+
+The following example uses the BERT-large, uncased, whole-word-masking model and fine-tunes it on the MNLI task.
+
+```bash
+export GLUE_DIR=/path/to/glue
+
+python -m torch.distributed.launch \
+    --nproc_per_node 8 run_glue.py \
+    --model_type bert \
+    --model_name_or_path bert-base-cased \
+    --task_name mnli \
+    --do_train \
+    --do_eval \
+    --do_lower_case \
+    --data_dir $GLUE_DIR/MNLI/ \
+    --max_seq_length 128 \
+    --per_gpu_train_batch_size 8 \
+    --learning_rate 2e-5 \
+    --num_train_epochs 3.0 \
+    --output_dir output_dir \
+```
+
+The results  are the following:
+
+```bash
+***** Eval results *****
+  acc = 0.8679706601466992
+  eval_loss = 0.4911287787382479
+  global_step = 18408
+  loss = 0.04755385363816904
+
+***** Eval results *****
+  acc = 0.8747965825874695
+  eval_loss = 0.45516540421714036
+  global_step = 18408
+  loss = 0.04755385363816904
+```
+
+##Multiple Choice
+
+Based on the script [`run_multiple_choice.py`]().
+
+#### Fine-tuning on SWAG
+Download [swag](https://github.com/rowanz/swagaf/tree/master/data) data
+
+```
+#training on 4 tesla V100(16GB) GPUS
+export SWAG_DIR=/path/to/swag_data_dir
+python ./examples/single_model_scripts/run_multiple_choice.py \
+--model_type roberta \
+--task_name swag \
+--model_name_or_path roberta-base \
+--do_train \
+--do_eval \
+--do_lower_case \
+--data_dir $SWAG_DIR \
+--learning_rate 5e-5 \
+--num_train_epochs 3 \
+--max_seq_length 80 \
+--output_dir models_bert/swag_base \
+--per_gpu_eval_batch_size=16 \
+--per_gpu_train_batch_size=16 \
+--gradient_accumulation_steps 2 \
+--overwrite_output
+```
+Training with the defined hyper-parameters yields the following results:
+```
+***** Eval results *****
+eval_acc = 0.8338998300509847
+eval_loss = 0.44457291918821606
+```
+
+## SQuAD
+
+Based on the script [`run_squad.py`](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_squad.py).
+
+#### Fine-tuning on SQuAD
+
+This example code fine-tunes BERT on the SQuAD dataset. It runs in 24 min (with BERT-base) or 68 min (with BERT-large) 
+on a single tesla V100 16GB. The data for SQuAD can be downloaded with the following links and should be saved in a 
+$SQUAD_DIR directory.
+
+* [train-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)
+* [dev-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json)
+* [evaluate-v1.1.py](https://github.com/allenai/bi-att-flow/blob/master/squad/evaluate-v1.1.py)
+
+```bash
+export SQUAD_DIR=/path/to/SQUAD
+
+python run_squad.py \
+  --model_type bert \
+  --model_name_or_path bert-base-cased \
+  --do_train \
+  --do_eval \
+  --do_lower_case \
+  --train_file $SQUAD_DIR/train-v1.1.json \
+  --predict_file $SQUAD_DIR/dev-v1.1.json \
+  --per_gpu_train_batch_size 12 \
+  --learning_rate 3e-5 \
+  --num_train_epochs 2.0 \
+  --max_seq_length 384 \
+  --doc_stride 128 \
+  --output_dir /tmp/debug_squad/
+```
+
+Training with the previously defined hyper-parameters yields the following results:
+
+```bash
+f1 = 88.52
+exact_match = 81.22
+```
+
+#### Distributed training
+
+
+Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD:
+
+```bash
+python -m torch.distributed.launch --nproc_per_node=8 run_squad.py \
+    --model_type bert \
+    --model_name_or_path bert-base-cased \
+    --do_train \
+    --do_eval \
+    --do_lower_case \
+    --train_file $SQUAD_DIR/train-v1.1.json \
+    --predict_file $SQUAD_DIR/dev-v1.1.json \
+    --learning_rate 3e-5 \
+    --num_train_epochs 2 \
+    --max_seq_length 384 \
+    --doc_stride 128 \
+    --output_dir ../models/wwm_uncased_finetuned_squad/ \
+    --per_gpu_train_batch_size 24 \
+    --gradient_accumulation_steps 12
+```
+
+Training with the previously defined hyper-parameters yields the following results:
+
+```bash
+f1 = 93.15
+exact_match = 86.91
+```
+
+This fine-tuneds model is available as a checkpoint under the reference
+`bert-large-uncased-whole-word-masking-finetuned-squad`.
+

Optimus/code/examples/big_ae/debug_data.py

@@ -0,0 +1,6 @@
+import torch
+import os
+
+output_dir = "../output/philly_rr1_vae_wikipedia_pretraining_2nd_file"
+
+data = torch.load(os.path.join(output_dir, 'batch_debug_6621.pt')

Optimus/code/examples/big_ae/eval_dialog_multi_response.py

@@ -0,0 +1,378 @@
+import numpy as np
+import torch
+import torch.nn.functional as F
+from nltk.translate.bleu_score import sentence_bleu
+from nltk.translate.bleu_score import SmoothingFunction
+from sklearn.metrics.pairwise import cosine_similarity as cosine
+from collections import Counter
+import os, pickle, pdb
+
+class Metrics:
+    # based on https://raw.githubusercontent.com/guxd/DialogWAE/29f206af05bfe5fe28fec4448e208310a7c9258d/experiments/metrics.py
+    
+    def __init__(self, path_word2vec='../data/datasets/dailydialog_data/glove.twitter.27B.200d.txt'):
+        """
+        :param word2vec - a numpy array of word2vec with shape [vocab_size x emb_size]
+        """
+        super(Metrics, self).__init__()
+        self.load_word2vec(path_word2vec)
+        #self.word2vec = dict()
+
+    def load_word2vec(self, path_word2vec):
+        path_pkl = path_word2vec + '.pkl'
+        if os.path.exists(path_pkl):
+            print('loading word2vec from '+path_pkl)
+            self.word2vec = pickle.load(open(path_pkl, 'rb'))
+        else:
+            self.word2vec = dict()
+            for i, line in enumerate(open(path_word2vec, encoding='utf-8')):
+                ss = line.strip('\n').split() 
+                self.word2vec[ss[0]] = [float(v) for v in ss[1:]]
+                if i % 1e4 == 0:
+                    print('processed %ik word2vec'%(i/1e3))
+            print('dumping word2vec to '+path_pkl)
+            pickle.dump(self.word2vec, open(path_pkl, 'wb'))
+        self.embed_dim = len(list(self.word2vec.values())[0])
+        print('loaded %i word2vec of dim %i'%(len(self.word2vec), self.embed_dim))
+
+    def embedding(self, seqs): 
+        # note: different from original implementation
+        batch_size, seqlen = seqs.shape
+        embs = np.zeros([batch_size, seqlen, self.embed_dim])
+        for i in range(batch_size):
+            for j in range(seqlen):
+                w = seqs[i,j] 
+                if w != '' and w in self.word2vec:
+                    embs[i, j, :] = self.word2vec[w]
+        return embs
+
+    
+    def extrema(self, embs, lens): # embs: [batch_size x seq_len x emb_size]  lens: [batch_size]
+        """
+        computes the value of every single dimension in the word vectors which has the greatest
+        difference from zero.
+        :param seq: sequence
+        :param seqlen: length of sequence
+        """
+        # Find minimum and maximum value for every dimension in predictions
+        batch_size, seq_len, emb_size = embs.shape
+        max_mask = np.zeros((batch_size, seq_len, emb_size), dtype=np.int)
+        for i,length in enumerate(lens):
+            max_mask[i,:length,:]=1
+        min_mask = 1-max_mask
+        seq_max = (embs*max_mask).max(1) # [batch_sz x emb_sz]
+        seq_min = (embs+min_mask).min(1)
+        # Find the maximum absolute value in min and max data
+        comp_mask = seq_max >= np.abs(seq_min)# [batch_sz x emb_sz]
+        # Add vectors for finding final sequence representation for predictions
+        extrema_emb = seq_max* comp_mask + seq_min* np.logical_not(comp_mask)
+        return extrema_emb
+    
+    def mean(self, embs, lens):
+        batch_size, seq_len, emb_size=embs.shape
+        mask = np.zeros((batch_size, seq_len, emb_size), dtype=np.int)
+        for i,length in enumerate(lens):
+            mask[i,:length,:]=1
+        return (embs*mask).sum(1)/(mask.sum(1)+1e-8)
+
+    def sim_bleu(self, hyps, ref):
+        """
+        :param ref - a list of tokens of the reference
+        :param hyps - a list of tokens of the hypothesis
+    
+        :return maxbleu - recall bleu
+        :return avgbleu - precision bleu
+        """
+        scores = []
+        for hyp in hyps:
+            try:
+                scores.append(sentence_bleu([ref], hyp, smoothing_function=SmoothingFunction().method7,
+                                        weights=[1./3, 1./3, 1./3]))
+            except:
+                scores.append(0.0)
+        return np.max(scores), np.mean(scores)
+
+
+    def sim_bow(self, pred, pred_lens, ref, ref_lens):
+        """
+        :param pred - ndarray [batch_size x seqlen]
+        :param pred_lens - list of integers
+        :param ref - ndarray [batch_size x seqlen]
+        """
+        # look up word embeddings for prediction and reference
+        emb_pred = self.embedding(pred) # [batch_sz x seqlen1 x emb_sz]
+        emb_ref = self.embedding(ref) # [batch_sz x seqlen2 x emb_sz]
+        
+        ext_emb_pred=self.extrema(emb_pred, pred_lens)
+        ext_emb_ref=self.extrema(emb_ref, ref_lens)
+        bow_extrema=cosine(ext_emb_pred, ext_emb_ref) # [batch_sz_pred x batch_sz_ref]
+        
+        avg_emb_pred = self.mean(emb_pred, pred_lens) # Calculate mean over seq
+        avg_emb_ref = self.mean(emb_ref, ref_lens) 
+        bow_avg = cosine(avg_emb_pred, avg_emb_ref) # [batch_sz_pred x batch_sz_ref]
+
+        
+        batch_pred, seqlen_pred, emb_size=emb_pred.shape
+        batch_ref, seqlen_ref, emb_size=emb_ref.shape
+        cos_sim = cosine(emb_pred.reshape((-1, emb_size)), emb_ref.reshape((-1, emb_size))) # [(batch_sz*seqlen1)x(batch_sz*seqlen2)]
+        cos_sim = cos_sim.reshape((batch_pred, seqlen_pred, batch_ref, seqlen_ref))
+        # Find words with max cosine similarity
+        max12 = cos_sim.max(1).mean(2) # max over seqlen_pred
+        max21 = cos_sim.max(3).mean(1) # max over seqlen_ref
+        bow_greedy=(max12+max21)/2 # [batch_pred x batch_ref(1)]
+        return np.max(bow_extrema), np.max(bow_avg), np.max(bow_greedy)
+    
+    def div_distinct(self, seqs, seq_lens):
+        """
+        distinct-1 distinct-2 metrics for diversity measure proposed 
+        by Li et al. "A Diversity-Promoting Objective Function for Neural Conversation Models"
+        we counted numbers of distinct unigrams and bigrams in the generated responses 
+        and divide the numbers by total number of unigrams and bigrams. 
+        The two metrics measure how informative and diverse the generated responses are. 
+        High numbers and high ratios mean that there is much content in the generated responses, 
+        and high numbers further indicate that the generated responses are long
+        """
+        batch_size = seqs.shape[0]
+        intra_dist1, intra_dist2=np.zeros(batch_size), np.zeros(batch_size)
+        
+        n_unigrams, n_bigrams, n_unigrams_total , n_bigrams_total = 0. ,0., 0., 0.
+        unigrams_all, bigrams_all = Counter(), Counter()
+        for b in range(batch_size):
+            unigrams= Counter([tuple(seqs[b,i:i+1]) for i in range(seq_lens[b])])
+            bigrams = Counter([tuple(seqs[b,i:i+2]) for i in range(seq_lens[b]-1)])
+            intra_dist1[b]=(len(unigrams.items())+1e-12)/(seq_lens[b]+1e-5)
+            intra_dist2[b]=(len(bigrams.items())+1e-12)/(max(0, seq_lens[b]-1)+1e-5)
+            
+            unigrams_all.update([tuple(seqs[b,i:i+1]) for i in range(seq_lens[b])])
+            bigrams_all.update([tuple(seqs[b,i:i+2]) for i in range(seq_lens[b]-1)])
+            n_unigrams_total += seq_lens[b]
+            n_bigrams_total += max(0, seq_lens[b]-1)
+
+        inter_dist1 = (len(unigrams_all.items())+1e-12)/(n_unigrams_total+1e-5)
+        inter_dist2 = (len(bigrams_all.items())+1e-12)/(n_bigrams_total+1e-5)
+        return intra_dist1, intra_dist2, inter_dist1, inter_dist2
+
+import pdb
+
+def eval_multi_ref(path, path_multi_ref=None):
+    """
+    based on: https://github.com/guxd/DialogWAE/blob/29f206af05bfe5fe28fec4448e208310a7c9258d/sample.py
+    path:   each line is '\t'.join([src, ref, hyp])
+    path_multi_ref:   each line is '\t'.join([src, hyp])
+    the order of unique src appeared in `path_multi_ref` should be the same as that in `path`
+    """
+    metrics = Metrics()
+    d_ref = dict()
+    d_hyp = dict()
+    src2ix = dict()
+    ix2src = dict()
+    ix = 0
+    for line in open(path, encoding='utf-8'):
+        line = line.strip('\n').strip()
+        if len(line) == 0:
+            continue
+        
+        # pdb.set_trace()
+        src, ref, hyp = line.split('\t')
+        #src, ref = line.split('\t'); hyp = ref
+        src = src.replace(' EOS ',' [SEP] ').strip()
+        ref = ref.strip().split()
+        hyp = hyp.strip().split()
+        if src not in d_ref:
+            d_ref[src] = ref
+            d_hyp[src] = [hyp]
+            src2ix[src] = ix
+            ix2src[ix] = src
+            ix += 1
+        else:
+            d_hyp[src].append(hyp)
+    print('loaded %i src-ref-hyp tuples'%(len(d_ref)))
+        
+    def chr_only(s):
+        ret = ''
+        for c in s:
+            if c.isalpha():
+                ret += c
+        return ret
+
+    if path_multi_ref is not None:
+        set_src4multiref = set()
+        ix = -1
+        d_multi_ref = dict()
+        for line in open(path_multi_ref, encoding='utf-8'):
+            line = line.strip('\n').strip()
+            if len(line) == 0:
+                continue
+            src4multiref, ref = line.split('\t')[:2]
+            src4multiref = src4multiref.replace(' EOS ', ' ').replace(' [SEP] ',' ').strip()
+            ref = ref.strip().split()
+            if src4multiref not in set_src4multiref:
+                set_src4multiref.add(src4multiref)
+                ix += 1
+                src = ix2src[ix]
+                id_hyp = chr_only(src)
+                id_multiref = chr_only(src4multiref)
+                if id_multiref != id_hyp:
+                    print('[ERROR] cannot match src4multiref and src4hyp')
+                    print('src4multiref:', src4multiref)
+                    print('src4hyp:', ix2src[ix])
+                    # pdb.set_trace()
+                    raise ValueError
+                d_multi_ref[src] = [ref]
+            else:
+                d_multi_ref[src].append(ref)
+            
+        n_ref = [len(d_multi_ref[k]) for k in d_multi_ref]
+        print('loaded %i src with multi-ref, avg n_ref = %.3f'%(len(d_multi_ref), np.mean(n_ref)))
+
+    n_miss = 0
+    for src in d_ref:
+        if src not in d_multi_ref:
+            n_miss += 1
+            print('[WARNING] cannot find multiref for src: '+src)
+            d_multi_ref[src] = [d_ref[src]]
+    if n_miss > 5:
+        raise ValueError
+    
+    n = len(d_ref)
+    print(path)
+    print('n_src\t%i'%n)
+
+    avg_lens = 0
+    maxbleu = 0
+    avgbleu = 0
+    intra_dist1, intra_dist2, inter_dist1, inter_dist2 = 0,0,0,0
+    bow_extrema, bow_avg, bow_greedy = 0,0,0
+    for src in d_ref:
+
+        # BLEU ----
+
+        if path_multi_ref is None:
+            m, a = metrics.sim_bleu(d_hyp[src], d_ref[src])
+        else:
+            n_ref = len(d_multi_ref[src])
+            m, a = 0, 0
+            for ref in d_multi_ref[src]:
+                _m, _a = metrics.sim_bleu(d_hyp[src], ref)
+                m += _m
+                a += _a
+            m /= n_ref
+            a /= n_ref
+
+        maxbleu += m
+        avgbleu += a
+
+        # diversity ----
+
+        seq_len = [len(hyp) for hyp in d_hyp[src]]
+        max_len = max(seq_len)
+        seqs = []
+        for hyp in d_hyp[src]:
+            padded = hyp + [''] * (max_len - len(hyp))
+            seqs.append(np.reshape(padded, [1, -1]))
+        seqs = np.concatenate(seqs, axis=0)
+        intra1, intra2, inter1, inter2 = metrics.div_distinct(seqs, seq_len)
+        intra_dist1 += np.mean(intra1)
+        intra_dist2 += np.mean(intra2)
+        inter_dist1 += inter1
+        inter_dist2 += inter2
+
+        avg_lens += np.mean(seq_len)
+        
+        # BOW ----
+
+        def calc_bow(ref):
+            n_hyp = len(d_hyp[src])
+            seqs_ref = np.concatenate([np.reshape(ref, [1,-1])] * n_hyp, axis=0)
+            seq_len_ref = [len(ref)] * n_hyp
+            return metrics.sim_bow(seqs, seq_len, seqs_ref, seq_len_ref)
+
+        if path_multi_ref is None:
+            extrema, avg, greedy = calc_bow(d_ref[src])
+        else:
+            extrema, avg, greedy = 0, 0, 0
+            for ref in d_multi_ref[src]:
+                e, a, g = calc_bow(ref)
+                extrema += e
+                avg += a
+                greedy += g
+            extrema /= n_ref
+            avg /= n_ref
+            greedy /= n_ref
+
+        bow_extrema += extrema
+        bow_avg += avg
+        bow_greedy += greedy
+
+    recall_bleu = maxbleu/n
+    prec_bleu = avgbleu/n
+    f1 = 2*(prec_bleu*recall_bleu) / (prec_bleu+recall_bleu+10e-12)
+    
+    print('BLEU')
+    print('  R\t%.3f'%recall_bleu)
+    print('  P\t%.3f'%prec_bleu)
+    print('  F1\t%.3f'%f1)
+    print('BOW')
+    print('  A\t%.3f'%(bow_avg/n))
+    print('  E\t%.3f'%(bow_extrema/n))
+    print('  G\t%.3f'%(bow_greedy/n))
+    print('intra_dist')
+    print('  1\t%.3f'%(intra_dist1/n))
+    print('  2\t%.3f'%(intra_dist2/n))
+    print('inter_dist')
+    print('  1\t%.3f'%(inter_dist1/n))
+    print('  2\t%.3f'%(inter_dist2/n))
+    print('avg_L\t%.1f'%(avg_lens/n))
+
+    results = {
+        "BLEU_R": recall_bleu, "BLEU_P": prec_bleu, "BLEU_F1": f1, "BOW_A": bow_avg/n, "BOW_E": bow_extrema/n, "BOW_G": bow_greedy/n, "intra_dist1": intra_dist1/n, "intra_dist2": intra_dist2/n, "inter_dist1": inter_dist1/n, "inter_dist2": inter_dist2/n, "avg_L": avg_lens/n
+    }
+
+    return results
+
+
+def create_rand_baseline():
+    path = 'data/datasets/dailydialog_data/test.txt'
+    srcs = []
+    refs = []
+    for line in open(path, encoding='utf-8'):
+        src, ref = line.strip('\n').split('\t')
+        srcs.append(src.strip())
+        refs.append(ref.strip())
+    
+    hyps = set()
+    path = 'data/datasets/dailydialog_data/train.txt'
+    for line in open(path, encoding='utf-8'):
+        _, ref = line.strip('\n').split('\t')
+        hyps.add(ref)
+        if len(hyps) == len(srcs) *10:
+            print('collected training ref')
+            break
+    
+    hyps = list(hyps)
+    lines = []
+    j = 0
+    for i in range(len(srcs)):
+        lines += ['\t'.join([srcs[i], refs[i], hyp]) for hyp in hyps[j:j+10]]
+        j = j + 10
+    with open('out/rand.tsv', 'w', encoding='utf-8') as f:
+        f.write('\n'.join(lines))
+
+
+def create_human_baseline():
+    path = 'data/datasets/dailydialog_data/test.txt'
+    lines = []
+    for line in open(path, encoding='utf-8'):
+        src, ref = line.strip('\n').split('\t')
+        src = src.strip()
+        ref = ref.strip()
+        lines.append('\t'.join([src, ref, ref]))
+        
+    with open('out/human.tsv', 'w', encoding='utf-8') as f:
+        f.write('\n'.join(lines))
+
+
+if __name__ == "__main__":
+    path = 'D:/data/switchboard/test.txt.1ref'
+    path_multi_ref = 'D:/data/switchboard/test.txt'
+    eval_multi_ref(path_multi_ref, path)

Optimus/code/examples/big_ae/eval_dialog_response.py

@@ -0,0 +1,295 @@
+import numpy as np
+import torch
+import torch.nn.functional as F
+from nltk.translate.bleu_score import sentence_bleu
+from nltk.translate.bleu_score import SmoothingFunction
+from sklearn.metrics.pairwise import cosine_similarity as cosine
+from collections import Counter
+import os, pickle
+
+class Metrics:
+    # based on https://raw.githubusercontent.com/guxd/DialogWAE/29f206af05bfe5fe28fec4448e208310a7c9258d/experiments/metrics.py
+    
+    def __init__(self, path_word2vec='../data/datasets/dailydialog_data/glove.twitter.27B.200d.txt'):
+        """
+        :param word2vec - a numpy array of word2vec with shape [vocab_size x emb_size]
+        """
+        self.path_word2vec = path_word2vec
+        super(Metrics, self).__init__()
+        self.load_word2vec(path_word2vec)
+
+    def load_word2vec(self, path_word2vec):
+        path_pkl = path_word2vec + '.pkl'
+        if os.path.exists(path_pkl):
+            print('loading word2vec from '+path_pkl)
+            self.word2vec = pickle.load(open(path_pkl, 'rb'))
+        else:
+            self.word2vec = dict()
+            for i, line in enumerate(open(path_word2vec, encoding='utf-8')):
+                ss = line.strip('\n').split() 
+                self.word2vec[ss[0]] = [float(v) for v in ss[1:]]
+                if i % 1e4 == 0:
+                    print('processed %ik word2vec'%(i/1e3))
+            print('dumping word2vec to '+path_pkl)
+            pickle.dump(self.word2vec, open(path_pkl, 'wb'))
+        # pdb.set_trace()
+        self.embed_dim = len(self.word2vec["."]) # len(self.word2vec.values()[0])
+        print('loaded %i word2vec of dim %i'%(len(self.word2vec), self.embed_dim))
+
+    def embedding(self, seqs): 
+        # note: different from original implementation
+        batch_size, seqlen = seqs.shape
+        embs = np.zeros([batch_size, seqlen, self.embed_dim])
+        for i in range(batch_size):
+            for j in range(seqlen):
+                w = seqs[i,j] 
+                if w != '' and w in self.word2vec:
+                    embs[i, j, :] = self.word2vec[w]
+        return embs
+
+    
+    def extrema(self, embs, lens): # embs: [batch_size x seq_len x emb_size]  lens: [batch_size]
+        """
+        computes the value of every single dimension in the word vectors which has the greatest
+        difference from zero.
+        :param seq: sequence
+        :param seqlen: length of sequence
+        """
+        # Find minimum and maximum value for every dimension in predictions
+        batch_size, seq_len, emb_size = embs.shape
+        max_mask = np.zeros((batch_size, seq_len, emb_size), dtype=np.int)
+        for i,length in enumerate(lens):
+            max_mask[i,:length,:]=1
+        min_mask = 1-max_mask
+        seq_max = (embs*max_mask).max(1) # [batch_sz x emb_sz]
+        seq_min = (embs+min_mask).min(1)
+        # Find the maximum absolute value in min and max data
+        comp_mask = seq_max >= np.abs(seq_min)# [batch_sz x emb_sz]
+        # Add vectors for finding final sequence representation for predictions
+        extrema_emb = seq_max* comp_mask + seq_min* np.logical_not(comp_mask)
+        return extrema_emb
+    
+    def mean(self, embs, lens):
+        batch_size, seq_len, emb_size=embs.shape
+        mask = np.zeros((batch_size, seq_len, emb_size), dtype=np.int)
+        for i,length in enumerate(lens):
+            mask[i,:length,:]=1
+        return (embs*mask).sum(1)/(mask.sum(1)+1e-8)
+
+    def sim_bleu(self, hyps, ref):
+        """
+        :param ref - a list of tokens of the reference
+        :param hyps - a list of tokens of the hypothesis
+    
+        :return maxbleu - recall bleu
+        :return avgbleu - precision bleu
+        """
+        scores = []
+        for hyp in hyps:
+            try:
+                scores.append(sentence_bleu([ref], hyp, smoothing_function=SmoothingFunction().method7,
+                                        weights=[1./3, 1./3, 1./3]))
+            except:
+                scores.append(0.0)
+        return np.max(scores), np.mean(scores)
+
+
+    def sim_bow(self, pred, pred_lens, ref, ref_lens):
+        """
+        :param pred - ndarray [batch_size x seqlen]
+        :param pred_lens - list of integers
+        :param ref - ndarray [batch_size x seqlen]
+        """
+        # look up word embeddings for prediction and reference
+        emb_pred = self.embedding(pred) # [batch_sz x seqlen1 x emb_sz]
+        emb_ref = self.embedding(ref) # [batch_sz x seqlen2 x emb_sz]
+        
+        ext_emb_pred=self.extrema(emb_pred, pred_lens)
+        ext_emb_ref=self.extrema(emb_ref, ref_lens)
+        bow_extrema=cosine(ext_emb_pred, ext_emb_ref) # [batch_sz_pred x batch_sz_ref]
+        
+        avg_emb_pred = self.mean(emb_pred, pred_lens) # Calculate mean over seq
+        avg_emb_ref = self.mean(emb_ref, ref_lens) 
+        bow_avg = cosine(avg_emb_pred, avg_emb_ref) # [batch_sz_pred x batch_sz_ref]
+
+        
+        batch_pred, seqlen_pred, emb_size=emb_pred.shape
+        batch_ref, seqlen_ref, emb_size=emb_ref.shape
+        cos_sim = cosine(emb_pred.reshape((-1, emb_size)), emb_ref.reshape((-1, emb_size))) # [(batch_sz*seqlen1)x(batch_sz*seqlen2)]
+        cos_sim = cos_sim.reshape((batch_pred, seqlen_pred, batch_ref, seqlen_ref))
+        # Find words with max cosine similarity
+        max12 = cos_sim.max(1).mean(2) # max over seqlen_pred
+        max21 = cos_sim.max(3).mean(1) # max over seqlen_ref
+        bow_greedy=(max12+max21)/2 # [batch_pred x batch_ref(1)]
+        return np.max(bow_extrema), np.max(bow_avg), np.max(bow_greedy)
+    
+    def div_distinct(self, seqs, seq_lens):
+        """
+        distinct-1 distinct-2 metrics for diversity measure proposed 
+        by Li et al. "A Diversity-Promoting Objective Function for Neural Conversation Models"
+        we counted numbers of distinct unigrams and bigrams in the generated responses 
+        and divide the numbers by total number of unigrams and bigrams. 
+        The two metrics measure how informative and diverse the generated responses are. 
+        High numbers and high ratios mean that there is much content in the generated responses, 
+        and high numbers further indicate that the generated responses are long
+        """
+        batch_size = seqs.shape[0]
+        intra_dist1, intra_dist2=np.zeros(batch_size), np.zeros(batch_size)
+        
+        n_unigrams, n_bigrams, n_unigrams_total , n_bigrams_total = 0. ,0., 0., 0.
+        unigrams_all, bigrams_all = Counter(), Counter()
+        for b in range(batch_size):
+            unigrams= Counter([tuple(seqs[b,i:i+1]) for i in range(seq_lens[b])])
+            bigrams = Counter([tuple(seqs[b,i:i+2]) for i in range(seq_lens[b]-1)])
+            intra_dist1[b]=(len(unigrams.items())+1e-12)/(seq_lens[b]+1e-5)
+            intra_dist2[b]=(len(bigrams.items())+1e-12)/(max(0, seq_lens[b]-1)+1e-5)
+            
+            unigrams_all.update([tuple(seqs[b,i:i+1]) for i in range(seq_lens[b])])
+            bigrams_all.update([tuple(seqs[b,i:i+2]) for i in range(seq_lens[b]-1)])
+            n_unigrams_total += seq_lens[b]
+            n_bigrams_total += max(0, seq_lens[b]-1)
+
+        inter_dist1 = (len(unigrams_all.items())+1e-12)/(n_unigrams_total+1e-5)
+        inter_dist2 = (len(bigrams_all.items())+1e-12)/(n_bigrams_total+1e-5)
+        return intra_dist1, intra_dist2, inter_dist1, inter_dist2
+
+import pdb
+
+def eval_dialog_response(generated_text_file_path):
+    """
+    based on: https://github.com/guxd/DialogWAE/blob/29f206af05bfe5fe28fec4448e208310a7c9258d/sample.py
+    quoted from the DialogWAE paper: https://arxiv.org/pdf/1805.12352.pdf
+    * "For each test context, we sample 10 responses from the models and compute their BLEU scores"
+    * "We use Glove vectors" "For each test context, we report the maximum BOW embedding score among the 10 sampled responses."
+    * "intra-dist as the average of distinct values within each sampled response"
+    " "inter-dist as the distinct value among all sampled responses."
+    """
+    metrics = Metrics()
+    d_ref = dict()
+    d_hyp = dict()
+    for line in open(generated_text_file_path, encoding='utf-8'):
+        line = line.strip('\n').strip()
+        if len(line) == 0:
+            continue
+        src, ref, hyp = line.split('\t')
+        src = src.strip()
+        ref = ref.strip().split()
+        hyp = hyp.strip().split()
+        if src not in d_ref:
+            d_ref[src] = ref
+            d_hyp[src] = [hyp]
+        else:
+            d_hyp[src].append(hyp)
+    
+    n = len(d_ref)
+    print(generated_text_file_path)
+    print('n_src\t%i'%n)
+
+    avg_lens = 0
+    maxbleu = 0
+    avgbleu = 0
+    intra_dist1, intra_dist2, inter_dist1, inter_dist2 = 0,0,0,0
+    bow_extrema, bow_avg, bow_greedy = 0,0,0
+    for src in d_ref:
+        m, a = metrics.sim_bleu(d_hyp[src], d_ref[src])
+        maxbleu += m
+        avgbleu += a
+        
+        seq_len = [len(hyp) for hyp in d_hyp[src]]
+        max_len = max(seq_len)
+        seqs = []
+        for hyp in d_hyp[src]:
+            padded = hyp + [''] * (max_len - len(hyp))
+            seqs.append(np.reshape(padded, [1, -1]))
+        seqs = np.concatenate(seqs, axis=0)
+        intra1, intra2, inter1, inter2 = metrics.div_distinct(seqs, seq_len)
+        intra_dist1 += np.mean(intra1)
+        intra_dist2 += np.mean(intra2)
+        inter_dist1 += inter1
+        inter_dist2 += inter2
+        
+        n_hyp = len(d_hyp[src])
+        seqs_ref = np.concatenate([np.reshape(d_ref[src], [1,-1])] * n_hyp, axis=0)
+        seq_len_ref = [len(d_ref[src])] * n_hyp
+        if metrics.word2vec is not None:
+            extrema, avg, greedy = metrics.sim_bow(seqs, seq_len, seqs_ref, seq_len_ref)
+            bow_extrema += extrema
+            bow_avg += avg
+            bow_greedy += greedy
+
+        avg_lens += np.mean(seq_len)
+
+    recall_bleu = maxbleu/n
+    prec_bleu = avgbleu/n
+    f1 = 2*(prec_bleu*recall_bleu) / (prec_bleu+recall_bleu+10e-12)
+    
+    print('BLEU')
+    print('  R\t%.3f'%recall_bleu)
+    print('  P\t%.3f'%prec_bleu)
+    print('  F1\t%.3f'%f1)
+    print('BOW')
+    print('  A\t%.3f'%(bow_avg/n))
+    print('  E\t%.3f'%(bow_extrema/n))
+    print('  G\t%.3f'%(bow_greedy/n))
+    print('intra_dist')
+    print('  1\t%.3f'%(intra_dist1/n))
+    print('  2\t%.3f'%(intra_dist2/n))
+    print('inter_dist')
+    print('  1\t%.3f'%(inter_dist1/n))
+    print('  2\t%.3f'%(inter_dist2/n))
+    print('avg_L\t%.1f'%(avg_lens/n))
+
+    results = {
+        "BLEU_R": recall_bleu, "BLEU_P": prec_bleu, "BLEU_F1": f1, "BOW_A": bow_avg/n, "BOW_E": bow_extrema/n, "BOW_G": bow_greedy/n, "intra_dist1": intra_dist1/n, "intra_dist2": intra_dist2/n, "inter_dist1": inter_dist1/n, "inter_dist2": inter_dist2/n, "avg_L": avg_lens/n
+    }
+
+    return results
+
+
+
+def create_rand_baseline():
+    path = 'data/datasets/dailydialog_data/test.txt'
+    srcs = []
+    refs = []
+    for line in open(path, encoding='utf-8'):
+        src, ref = line.strip('\n').split('\t')
+        srcs.append(src.strip())
+        refs.append(ref.strip())
+    
+    hyps = set()
+    path = 'data/datasets/dailydialog_data/train.txt'
+    for line in open(path, encoding='utf-8'):
+        _, ref = line.strip('\n').split('\t')
+        hyps.add(ref)
+        if len(hyps) == len(srcs) *10:
+            print('collected training ref')
+            break
+    
+    hyps = list(hyps)
+    lines = []
+    j = 0
+    for i in range(len(srcs)):
+        lines += ['\t'.join([srcs[i], refs[i], hyp]) for hyp in hyps[j:j+10]]
+        j = j + 10
+    with open('out/rand.tsv', 'w', encoding='utf-8') as f:
+        f.write('\n'.join(lines))
+
+
+def create_human_baseline():
+    path = 'data/datasets/dailydialog_data/test.txt'
+    lines = []
+    for line in open(path, encoding='utf-8'):
+        src, ref = line.strip('\n').split('\t')
+        src = src.strip()
+        ref = ref.strip()
+        lines.append('\t'.join([src, ref, ref]))
+        
+    with open('out/human.tsv', 'w', encoding='utf-8') as f:
+        f.write('\n'.join(lines))
+
+
+if __name__ == "__main__":
+    #create_rand_baseline()
+    #create_human_baseline()
+    eval_dialog_response('out/eval_text_generation_results (1).txt')
+    #eval('out/rand.tsv')