File size: 19,399 Bytes
570c07f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c79e854
570c07f
c79e854
570c07f
 
 
c79e854
570c07f
ecea61b
570c07f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ecea61b
570c07f
 
 
ecea61b
570c07f
 
 
ecea61b
570c07f
ecea61b
570c07f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ecea61b
570c07f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
#!/usr/bin/env python
# coding: utf-8

# # How to Build a Reverse Video Search Engine
# 
# This notebook illustrates how to build a reverse-video-search engine from scratch using [Milvus](https://milvus.io/) and [Towhee](https://towhee.io/).
# 
# **What is Reverse Video Search?**
# 
# Reverse video search is similar like [reverse image search](https://en.wikipedia.org/wiki/Reverse_image_search). In simple words, it takes a video as input to search for similar videos. As we know that video-related tasks are harder to tackle, video models normally do not achieve as high scores as other types of models. However, there are increasing demands in AI applications in video. Reverse video search can effectively discover related videos and improve other applications.
# 
# **What are Milvus & Towhee?**
# 
# - Milvus is the most advanced open-source vector database built for AI applications and supports nearest neighbor embedding search across tens of millions of entries.
# - Towhee is a framework that provides ETL for unstructured data using SoTA machine learning models.
# 
# We will go through the procedure of building a reverse-video-search engine and evaluate its performance.

# ## Preparation
# 
# ### Install packages
# 
# Make sure you have installed required python packages:
# 
# | package |
# | -- |
# | towhee |
# | towhee.models |
# | pillow |
# | ipython |
# | gradio |

# In[1]:


#! python -m pip install -q towhee towhee.models pillow ipython gradio

# ### Prepare data
# 
# This tutorial will use a small data extracted from [Kinetics400](https://www.deepmind.com/open-source/kinetics). You can download the subset from [Github](https://github.com/towhee-io/examples/releases/download/data/reverse_video_search.zip). 
# 
# The data is organized as follows:
# - **train:** candidate videos, 20 classes, 10 videos per class (200 in total)
# - **test:** query videos, same 20 classes as train data, 1 video per class (20 in total)
# - **reverse_video_search.csv:** a csv file containing an ***id***, ***path***, and ***label*** for each video in train data
# 
# Let's take a quick look:

# In[1]:

import  time
from zipfile import ZipFile

with ZipFile('reverse_video_search.zip', 'r') as zips:
    # printing all the contents of the zip file
    # extracting all the files
    print('Extracting all the files now...')
    zips.extractall()
    print('Done!')
    
# In[2]:

import pandas as pd
import time

df = pd.read_csv('./reverse_video_search.csv')
df.head(3)


# For later steps to easier get videos & measure results, we build some helpful functions in advance:
# - **ground_truth:** get ground-truth video ids for the query video by its path

# In[3]:


id_video = df.set_index('id')['path'].to_dict()
label_ids = {}
for label in set(df['label']):
    label_ids[label] = list(df[df['label']==label].id)
    

def ground_truth(path):
    label = path.split('/')[-2]
    return label_ids[label]


# ### Start Milvus
# 
# Before getting started with the engine, we also need to get ready with Milvus. Please make sure that you have started a [Milvus service](https://milvus.io/docs/install_standalone-docker.md). This notebook uses [milvus 2.2.10](https://milvus.io/docs/v2.2.x/install_standalone-docker.md) and [pymilvus 2.2.11](https://milvus.io/docs/release_notes.md#2210).

# In[ ]:


#! python -m pip install -q pymilvus==2.2.11


# Here we prepare a function to work with a Milvus collection with the following parameters:
# - [L2 distance metric](https://milvus.io/docs/metric.md#Euclidean-distance-L2)
# - [IVF_FLAT index](https://milvus.io/docs/index.md#IVF_FLAT).

# In[4]:


from milvus import default_server
from pymilvus import connections, utility
default_server.start()
time.sleep(60)

# In[5]:
connections.connect(host='127.0.0.1', port=default_server.listen_port)
time.sleep(60)
# In[6]:
default_server.listen_port
# In[7]:
time.sleep(10)
print(utility.get_server_version())
time.sleep(10)

# In[10]:
from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection, utility

#connections.connect(host='localhost', port='19530')
connections.connect(host='127.0.0.1', port='19530')

def create_milvus_collection(collection_name, dim):

    if utility.has_collection(collection_name):
        utility.drop_collection(collection_name)
    
    fields = [
    FieldSchema(name='id', dtype=DataType.INT64, descrition='ids', is_primary=True, auto_id=False),
    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, descrition='embedding vectors', dim=dim)
    ]
    schema = CollectionSchema(fields=fields, description='reverse video search')
    collection = Collection(name=collection_name, schema=schema)

    # create IVF_FLAT index for collection.
    index_params = {
        'metric_type':'L2',
        'index_type':"IVF_FLAT",
        'params':{"nlist": 400}
    }
    collection.create_index(field_name="embedding", index_params=index_params)
    return collection

collection = create_milvus_collection('x3d_m', 2048)


# In[11]:


time.sleep(10)


# ## Build Engine
# 
# Now we are ready to build a reverse-video-search engine. The basic idea behind reverse video search is to represent each video with an embedding and then perform similarity search by comparing vector distances.
# 
# As mentioned at the beginning, we use deep learning networks provided by Towhee to extract features and generate embeddings. Milvus is used for vector storage and similarity search.
# 
# <img src='reverse_video_search.png' alt='reverse_video_search_engine' width=700px/>

# ### Load Video Embeddings into Milvus
# 
# We first generate embeddings for videos with [X3D model](https://arxiv.org/abs/2004.04730) and then insert video embeddings into Milvus. Towhee provides a [method-chaining style API](https://towhee.readthedocs.io/en/main/index.html) so that users can assemble a data processing pipeline with operators. 

# In[12]:


from towhee import pipe, ops
from towhee.datacollection import DataCollection

def read_csv(csv_file):
    import csv
    with open(csv_file, 'r', encoding='utf-8-sig') as f:
        data = csv.DictReader(f)
        for line in data:
            yield line['id'], line['path'], line['label']
            

insert_pipe = (
    pipe.input('csv_path')
        .flat_map('csv_path', ('id', 'path', 'label'), read_csv)
        .map('id', 'id', lambda x: int(x))
        .map('path', 'frames', ops.video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 16}))
        .map('frames', ('labels', 'scores', 'features'), ops.action_classification.pytorchvideo(model_name='x3d_m', skip_preprocess=True))
        .map(('id', 'features'), 'insert_res', ops.ann_insert.milvus_client(host='127.0.0.1', port='19530', collection_name='x3d_m'))
        .output()
)

insert_pipe('reverse_video_search.csv')
print('Total number of inserted data is {}.'.format(collection.num_entities))


# In[13]:

time.sleep(60)
print('Total number of inserted data is {}.'.format(collection.num_entities))


# #### Pipeline Explanation
# 
# Here are some details for each line of the assemble pipeline:
# 
# - `flat_map('csv_path', ('id', 'path', 'label'), read_csv)`: read tabular data from csv file
# 
# - `map('id', 'id', lambda x: int(x))`: for each row from the data, convert data type of the column id to int
# 
# - `map('path', 'frames', ops.video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 16}))`: an embeded Towhee operator reading video as frames with specified sample method and number of samples. [learn more](https://towhee.io/video-decode/ffmpeg)
# 
# - `map('frames', ('labels', 'scores', 'features'), ops.action_classification.pytorchvideo(model_name='x3d_m', skip_preprocess=True))`: an embeded Towhee operator applying specified model to video frames, which can be used to generate video embedding. [learn more](https://towhee.io/action-classification/pytorchvideo)
# 
# - `map(('id', 'features'), 'insert_res', ops.ann_insert.milvus_client(host='127.0.0.1', port='19530', collection_name='x3d_m'))`: insert video embedding into Milvus collection

# ### Query Similar Videos from Milvus
# 
# Now all embeddings of candidate videos have been inserted into Milvus collection, we can query embeddings across the collection for nearest neighbors.
# 
# To get query embeddings, we should go through same pre-insert steps for each input video. Because Milvus returns video ids and vector distances, we use the `id_video` dictionary to get corresponding video paths based on ids.

# In[7]:


time.sleep(60)


# In[14]:


collection.load()
time.sleep(60)
query_path = './test/eating_carrots/ty4UQlowp0c.mp4'

query_pipe = (
    pipe.input('path')
        .map('path', 'frames', ops.video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 16}))
        .map('frames', ('labels', 'scores', 'features'), ops.action_classification.pytorchvideo(model_name='x3d_m', skip_preprocess=True))
        .map('features', 'result', ops.ann_search.milvus_client(host='127.0.0.1', port='19530', collection_name='x3d_m', limit=10))  
        .map('result', 'candidates', lambda x: [id_video[i[0]] for i in x])
        .output('path', 'candidates')
)

res = DataCollection(query_pipe(query_path))
res.show()


# To display in the notebook, we convert videos to gifs. The code below first loads each video from its path and then gets full video frames with the embeded Towhee operator `.video_decode.ffmpeg()`. Finally converted gifs are saved under the directory *tmp_dir*. The section below is just help to show a search example.

# In[15]:


import os
from IPython import display
from PIL import Image

tmp_dir = './tmp'
os.makedirs(tmp_dir, exist_ok=True)

def video_to_gif(video_path):
    gif_path = os.path.join(tmp_dir, video_path.split('/')[-1][:-4] + '.gif')
    p = (
        pipe.input('path')
            .map('path', 'frames', ops.video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 16}))
            .output('frames')
    )
    frames = p(video_path).get()[0]
    imgs = [Image.fromarray(frame) for frame in frames]
    imgs[0].save(fp=gif_path, format='GIF', append_images=imgs[1:], save_all=True, loop=0)
    return gif_path

html = 'Query video "{}": <br/>'.format(query_path.split('/')[-2])
query_gif = video_to_gif(query_path)
html_line = '<img src="{}"> <br/>'.format(query_gif)
html +=  html_line
html += 'Top 3 search results: <br/>'

for path in res[0]['candidates'][:3]:
    gif_path = video_to_gif(path)
    html_line = '<img src="{}" style="display:inline;margin:1px"/>'.format(gif_path)
    html +=  html_line
display.HTML(html)


# ### Evaluation
# 
# We have just built a reverse video search engine. But how's its performance? We can evaluate the search engine against the ground truths.
# 
# In this section, we'll measure the performance with 2 metrics - mHR and mAP:
# 
# - **mHR (recall@K):**
#     - Mean Hit Ratio describes how many actual relevant results are returned out of all ground truths.
#     - Since Milvus return results with topK, we can also call this metric *recall@K*, where K is the count of searched results. When returned results are as many as ground truths, the hit ratio is equivalent to accuracy and we can take it as *accuracy@K* as well.
#     - For example, there are 100 archery videos in the collection. Then querying the engine with another archery video returns 70 archery videos out of 80 results. In this case, the number of ground truths is 100 and hitted (correct) results are 70. So the hit ratio is 70/100.
# 
# - **mAP:**
#     - Average precision describes whether all of the relevant results are ranked higher than irrelevant results.

# In[16]:


import glob

def mean_hit_ratio(actual, predicted):
    ratios = []
    for act, pre in zip(actual, predicted):
        hit_num = len(set(act) & set(pre))
        ratios.append(hit_num / len(act))
    return sum(ratios) / len(ratios)

def mean_average_precision(actual, predicted):
    aps = []
    for act, pre in zip(actual, predicted):
        precisions = []
        hit = 0
        for idx, i in enumerate(pre):
            if i in act:
                hit += 1
            precisions.append(hit / (idx + 1))
        aps.append(sum(precisions) / len(precisions))
    
    return sum(aps) / len(aps)

eval_pipe = (
    pipe.input('path')
        .flat_map('path', 'path', lambda x: glob.glob(x))
        .map('path', 'frames', ops.video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 16}))
        .map('frames', ('labels', 'scores', 'features'), ops.action_classification.pytorchvideo(model_name='x3d_m', skip_preprocess=True))
        .map('features', 'result', ops.ann_search.milvus_client(host='127.0.0.1', port='19530', collection_name='x3d_m', limit=10))  
        .map('result', 'predict', lambda x: [i[0] for i in x])
        .map('path', 'ground_truth', ground_truth)
        .window_all(('ground_truth', 'predict'), 'mhr', mean_hit_ratio)
        .window_all(('ground_truth', 'predict'), 'map', mean_average_precision)
        .output('mhr', 'map')
)

res = DataCollection(eval_pipe('./test/*/*.mp4'))
res.show()


# ## Optimization
# 
# We can see from above evaluation report, the current performance is not satisfactory. What can we do to improve the search engine? Of course we can fine-tune deep learning network with our own train data. Using more types of embeddings or filters by video tags/description/captions and audio can definitely enhance the search engine as well. But in this tutorial, I will just recommend some very simple options to make improvements.
# 
# ### Normalize embeddings
# 
# A quick optimization is normalizing all embeddings. Then the L2 distance will be equivalent to cosine similarity, which measures the similarity between two vectors using the angle between them, which ignores the magnitude of the vectors. We use the `ops.towhee.np_normalize` provided by Towhee to simply normalize all embeddings.

# In[17]:


collection = create_milvus_collection('x3d_m_norm', 2048)

insert_pipe = (
    pipe.input('csv_path')
        .flat_map('csv_path', ('id', 'path', 'label'), read_csv)
        .map('id', 'id', lambda x: int(x))
        .map('path', 'frames', ops.video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 16}))
        .map('frames', ('labels', 'scores', 'features'), ops.action_classification.pytorchvideo(model_name='x3d_m', skip_preprocess=True))
        .map('features', 'features', ops.towhee.np_normalize())
        .map(('id', 'features'), 'insert_res', ops.ann_insert.milvus_client(host='127.0.0.1', port='19530', collection_name='x3d_m_norm'))
        .output()
)

insert_pipe('reverse_video_search.csv')

collection.load()
eval_pipe = (
    pipe.input('path')
        .flat_map('path', 'path', lambda x: glob.glob(x))
        .map('path', 'frames', ops.video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 16}))
        .map('frames', ('labels', 'scores', 'features'), ops.action_classification.pytorchvideo(model_name='x3d_m', skip_preprocess=True))
        .map('features', 'features', ops.towhee.np_normalize())
        .map('features', 'result', ops.ann_search.milvus_client(host='127.0.0.1', port='19530', collection_name='x3d_m_norm', limit=10))  
        .map('result', 'predict', lambda x: [i[0] for i in x])
        .map('path', 'ground_truth', ground_truth)
        .window_all(('ground_truth', 'predict'), 'mhr', mean_hit_ratio)
        .window_all(('ground_truth', 'predict'), 'map', mean_average_precision)
        .output('mhr', 'map')
)

res = DataCollection(eval_pipe('./test/*/*.mp4'))
res.show()


# With vector normalization, we have increased the mHR to 0.66 and mAP to about 0.74, which look better now.

# ### Change model
# 
# There are more video models using different networks. Normally a more complicated or larger model will show better results while cost more. You can always try more models to tradeoff among accuracy, latency, and resource usage. Here I show the performance for the reverse video search engine using a SOTA model with [multiscale vision transformer](https://arxiv.org/abs/2104.11227) as backbone.

# In[18]:


collection = create_milvus_collection('mvit_base', 768)

insert_pipe = (
    pipe.input('csv_path')
        .flat_map('csv_path', ('id', 'path', 'label'), read_csv)
        .map('id', 'id', lambda x: int(x))
        .map('path', 'frames', ops.video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 32}))
        .map('frames', ('labels', 'scores', 'features'), ops.action_classification.pytorchvideo(model_name='mvit_base_32x3', skip_preprocess=True))
        .map('features', 'features', ops.towhee.np_normalize())
        .map(('id', 'features'), 'insert_res', ops.ann_insert.milvus_client(host='127.0.0.1', port='19530', collection_name='mvit_base'))
        .output()
)

insert_pipe('reverse_video_search.csv')

collection.load()
eval_pipe = (
    pipe.input('path')
        .flat_map('path', 'path', lambda x: glob.glob(x))
        .map('path', 'frames', ops.video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 32}))
        .map('frames', ('labels', 'scores', 'features'), ops.action_classification.pytorchvideo(model_name='mvit_base_32x3', skip_preprocess=True))
        .map('features', 'features', ops.towhee.np_normalize())
        .map('features', 'result', ops.ann_search.milvus_client(host='127.0.0.1', port='19530', collection_name='mvit_base', limit=10))  
        .map('result', 'predict', lambda x: [i[0] for i in x])
        .map('path', 'ground_truth', ground_truth)
        .window_all(('ground_truth', 'predict'), 'mhr', mean_hit_ratio)
        .window_all(('ground_truth', 'predict'), 'map', mean_average_precision)
        .output('mhr', 'map')
)

res = DataCollection(eval_pipe('./test/*/*.mp4'))
res.show()


# Switching to MVIT model increases the mHR to 0.79 and mAP to 0.83, which are much better than X3D model. However, both insert and search time have increased. It's time for you to make trade-off between latency and accuracy. You're always encouraged to play around with this tutorial.

# ## Release a Showcase
# 
# We've learnt how to build a reverse video search engine. Now it's time to add some interface and release a showcase.

# In[19]:


import gradio

video_search_pipe = (
    pipe.input('path')
        .flat_map('path', 'path', lambda x: glob.glob(x))
        .map('path', 'frames', ops.video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 32}))
        .map('frames', ('labels', 'scores', 'features'), ops.action_classification.pytorchvideo(model_name='mvit_base_32x3', skip_preprocess=True))
        .map('features', 'features', ops.towhee.np_normalize())
        .map('features', 'result', ops.ann_search.milvus_client(host='127.0.0.1', port='19530', collection_name='mvit_base', limit=3)) 
        .map('result', 'predict', lambda x: [id_video[i[0]] for i in x])
        .output('predict')
)


def video_search_function(video):
    return video_search_pipe(video).to_list()[0][0]

interface = gradio.Interface(video_search_function, 
                             inputs=gradio.Video(source='upload'),
                             outputs=[gradio.Video(format='mp4') for _ in range(3)]
                            )

interface.launch()


# In[ ]: