File size: 2,936 Bytes
52ca9c9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
## Download the filtered Conceptual Captions, SBU, LAION datasets

### Pre-training datasets download:
We use the filtered synthetic captions prepared by BLIP. For more details about the dataset, please refer to [BLIP](https://github.com/salesforce/BLIP).

It requires ~2.3T to store LAION and CC3M+CC12M+SBU datasets

Image source | Filtered synthetic caption by ViT-L
--- | :---:
CC3M+CC12M+SBU | <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/datasets/ccs_synthetic_filtered_large.json">Download</a>
LAION115M |  <a href="https://storage.googleapis.com/sfr-vision-language-research/BLIP/datasets/laion_synthetic_filtered_large.json">Download</a>

This will download two json files 
```
ccs_synthetic_filtered_large.json
laion_synthetic_filtered_large.json
```

## prepare the data step-by-step


### setup the dataset folder and move the annotation file to the data storage folder
```
export MINIGPT4_DATASET=/YOUR/PATH/FOR/LARGE/DATASET/
mkdir ${MINIGPT4_DATASET}/cc_sbu
mkdir ${MINIGPT4_DATASET}/laion
mv ccs_synthetic_filtered_large.json ${MINIGPT4_DATASET}/cc_sbu
mv laion_synthetic_filtered_large.json ${MINIGPT4_DATASET}/laion
```

### Convert the scripts to data storate folder
```
cp convert_cc_sbu.py ${MINIGPT4_DATASET}/cc_sbu
cp download_cc_sbu.sh ${MINIGPT4_DATASET}/cc_sbu
cp convert_laion.py ${MINIGPT4_DATASET}/laion
cp download_laion.sh ${MINIGPT4_DATASET}/laion
```


### Convert the laion and cc_sbu annotation file format to be img2dataset format
```
cd ${MINIGPT4_DATASET}/cc_sbu
python convert_cc_sbu.py

cd ${MINIGPT4_DATASET}/laion
python convert_laion.py
```

### Download the datasets with img2dataset
```
cd ${MINIGPT4_DATASET}/cc_sbu
sh download_cc_sbu.sh
cd ${MINIGPT4_DATASET}/laion
sh download_laion.sh
```


The final dataset structure

```
.
β”œβ”€β”€ ${MINIGPT4_DATASET}
β”‚   β”œβ”€β”€ cc_sbu
β”‚       β”œβ”€β”€ convert_cc_sbu.py
β”‚       β”œβ”€β”€ download_cc_sbu.sh
β”‚       β”œβ”€β”€ ccs_synthetic_filtered_large.json
β”‚       β”œβ”€β”€ ccs_synthetic_filtered_large.tsv
β”‚       └── cc_sbu_dataset
β”‚           β”œβ”€β”€ 00000.tar
β”‚           β”œβ”€β”€ 00000.parquet
β”‚           ...
β”‚   β”œβ”€β”€ laion
β”‚       β”œβ”€β”€ convert_laion.py
β”‚       β”œβ”€β”€ download_laion.sh
β”‚       β”œβ”€β”€ laion_synthetic_filtered_large.json
β”‚       β”œβ”€β”€ laion_synthetic_filtered_large.tsv
β”‚       └── laion_dataset
β”‚           β”œβ”€β”€ 00000.tar
β”‚           β”œβ”€β”€ 00000.parquet
β”‚           ...
...   
```


## Set up the dataset configuration files

Then, set up the LAION dataset loading path in 
[here](../minigpt4/configs/datasets/laion/defaults.yaml#L5) at Line 5 as 
${MINIGPT4_DATASET}/laion/laion_dataset/{00000..10488}.tar

and the Conceptual Captoin and SBU datasets loading path in 
[here](../minigpt4/configs/datasets/cc_sbu/defaults.yaml#L5) at Line 5 as 
${MINIGPT4_DATASET}/cc_sbu/cc_sbu_dataset/{00000..01255}.tar