thumbnail
Browse files
README.md
CHANGED
@@ -44,59 +44,27 @@ virtualenv --python=python3 ~/.envs/.my_env
|
|
44 |
source ~/.envs/.my_env/bin/activate
|
45 |
cd shift/
|
46 |
pip install -r requirements.txt
|
47 |
-
```
|
48 |
-
|
49 |
-
Start API
|
50 |
|
51 |
-
```
|
52 |
CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=./hf_home CUDA_VISIBLE_DEVICES=2 python api.py
|
53 |
```
|
54 |
|
55 |
-
The following need `api.py` to be already running `on a tmux session`.
|
56 |
-
|
57 |
## Landscape 2 Soundscape
|
58 |
|
|
|
|
|
59 |
```python
|
60 |
# TTS & soundscape - overlay to .mp4
|
61 |
python landscape2soundscape.py
|
62 |
```
|
63 |
|
64 |
-
#
|
65 |
-
|
66 |
-
Substitute Native voice via TTS
|
67 |
|
68 |
-
[![Native voice ANBPR video](assets/native_video_thumb.png)](https://www.youtube.com/watch?v=tmo2UbKYAqc)
|
69 |
|
70 |
-
|
71 |
-
|
72 |
-
Same video where Native voice is replaced with English TTS voice with similar emotion
|
73 |
|
74 |
|
75 |
[![Same video w. Native voice replaced with English TTS](assets/tts_video_thumb.png)](https://www.youtube.com/watch?v=geI1Vqn4QpY)
|
76 |
|
77 |
-
|
78 |
-
<details>
|
79 |
-
<summary>
|
80 |
-
|
81 |
-
Video dubbing from subtitles `.srt`
|
82 |
-
|
83 |
-
</summary>
|
84 |
-
|
85 |
-
## Video Dubbing
|
86 |
-
|
87 |
-
[![Review demo SHIFT](assets/review_demo_thumb.png)](https://www.youtube.com/watch?v=bpt7rOBENcQ)
|
88 |
-
|
89 |
-
Generate dubbed video:
|
90 |
-
|
91 |
-
|
92 |
-
```python
|
93 |
-
python tts.py --text assets/head_of_fortuna_en.srt --video assets/head_of_fortuna.mp4
|
94 |
-
|
95 |
-
```
|
96 |
-
|
97 |
-
|
98 |
-
</details>
|
99 |
-
|
100 |
## Joint Application of D3.1 & D3.2
|
101 |
|
102 |
<a href="https://youtu.be/wWC8DpOKVvQ" rel="Subtitles to Video">![Foo4](assets/caption_to_video_thumb.png)</a>
|
@@ -128,6 +96,8 @@ For SHIFT demo / Collaboration with [SMB](https://www.smb.museum/home/)
|
|
128 |
|
129 |
[![02](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____02_Constable_AI555_001.jpg)](https://youtu.be/2YjxAPkdXIc)
|
130 |
|
|
|
|
|
131 |
[![10](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____10_Boecklin_967648_NG2-80_001_rsz.jpg)](https://www.youtube.com/watch?v=Y8QyYUgLaCg)
|
132 |
|
133 |
|
|
|
44 |
source ~/.envs/.my_env/bin/activate
|
45 |
cd shift/
|
46 |
pip install -r requirements.txt
|
|
|
|
|
|
|
47 |
|
|
|
48 |
CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=./hf_home CUDA_VISIBLE_DEVICES=2 python api.py
|
49 |
```
|
50 |
|
|
|
|
|
51 |
## Landscape 2 Soundscape
|
52 |
|
53 |
+
The following needs `api.py` to be already running `on a tmux session`.
|
54 |
+
|
55 |
```python
|
56 |
# TTS & soundscape - overlay to .mp4
|
57 |
python landscape2soundscape.py
|
58 |
```
|
59 |
|
60 |
+
# Videos / Examples
|
|
|
|
|
61 |
|
|
|
62 |
|
63 |
+
Video where Native voice is replaced with English TTS voice
|
|
|
|
|
64 |
|
65 |
|
66 |
[![Same video w. Native voice replaced with English TTS](assets/tts_video_thumb.png)](https://www.youtube.com/watch?v=geI1Vqn4QpY)
|
67 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
68 |
## Joint Application of D3.1 & D3.2
|
69 |
|
70 |
<a href="https://youtu.be/wWC8DpOKVvQ" rel="Subtitles to Video">![Foo4](assets/caption_to_video_thumb.png)</a>
|
|
|
96 |
|
97 |
[![02](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____02_Constable_AI555_001.jpg)](https://youtu.be/2YjxAPkdXIc)
|
98 |
|
99 |
+
[![03](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____03_Schinkel_WS200-002.jpg)](https://youtu.be/BhMh02knkco)
|
100 |
+
|
101 |
[![10](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____10_Boecklin_967648_NG2-80_001_rsz.jpg)](https://www.youtube.com/watch?v=Y8QyYUgLaCg)
|
102 |
|
103 |
|
landscape2soundscape.py
CHANGED
@@ -62,7 +62,7 @@ DESCRIPTIONS = [
|
|
62 |
'03_Schinkel_WS200-002.jpg',
|
63 |
'03_Schinkel_WS200-002.txt',
|
64 |
'Arriving at the valley on galloping horses',
|
65 |
-
'
|
66 |
'en_US/hifi-tts_low#6097',
|
67 |
],
|
68 |
#
|
@@ -146,7 +146,7 @@ SILENT_VIDEO = '_silent_video.mp4'
|
|
146 |
# SILENT CLIP
|
147 |
|
148 |
|
149 |
-
for img, text, scene, title, voice in DESCRIPTIONS[2:
|
150 |
|
151 |
|
152 |
|
|
|
62 |
'03_Schinkel_WS200-002.jpg',
|
63 |
'03_Schinkel_WS200-002.txt',
|
64 |
'Arriving at the valley on galloping horses',
|
65 |
+
'K. Schinkel Gotische Kirche Auf Einem Felsen 1815',
|
66 |
'en_US/hifi-tts_low#6097',
|
67 |
],
|
68 |
#
|
|
|
146 |
# SILENT CLIP
|
147 |
|
148 |
|
149 |
+
for img, text, scene, title, voice in DESCRIPTIONS[2:4]:
|
150 |
|
151 |
|
152 |
|
uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____03_Schinkel_WS200-002.jpg
ADDED