File size: 4,955 Bytes
528df8b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
RVC ํ›ˆ๋ จ์— ๋Œ€ํ•œ ์„ค๋ช…๊ณผ ํŒ๋“ค
======================================
๋ณธ ํŒ์—์„œ๋Š” ์–ด๋–ป๊ฒŒ ๋ฐ์ดํ„ฐ ํ›ˆ๋ จ์ด ์ด๋ฃจ์–ด์ง€๊ณ  ์žˆ๋Š”์ง€ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

# ํ›ˆ๋ จ์˜ ํ๋ฆ„
GUI์˜ ํ›ˆ๋ จ ํƒญ์˜ ๋‹จ๊ณ„๋ฅผ ๋”ฐ๋ผ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

## step1
์‹คํ—˜ ์ด๋ฆ„์„ ์ง€์ •ํ•ฉ๋‹ˆ๋‹ค. ๋˜ํ•œ, ๋ชจ๋ธ์ด ํ”ผ์น˜(์†Œ๋ฆฌ์˜ ๋†’๋‚ฎ์ด)๋ฅผ ๊ณ ๋ คํ•ด์•ผ ํ•˜๋Š”์ง€ ์—ฌ๋ถ€๋ฅผ ์—ฌ๊ธฐ์—์„œ ์„ค์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค..
๊ฐ ์‹คํ—˜์„ ์œ„ํ•œ ๋ฐ์ดํ„ฐ๋Š” `/logs/experiment name/`์— ๋ฐฐ์น˜๋ฉ๋‹ˆ๋‹ค..

## step2a
์Œ์„ฑ ํŒŒ์ผ์„ ๋ถˆ๋Ÿฌ์˜ค๊ณ  ์ „์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค.

### ์Œ์„ฑ ํŒŒ์ผ ๋ถˆ๋Ÿฌ์˜ค๊ธฐ
์Œ์„ฑ ํŒŒ์ผ์ด ์žˆ๋Š” ํด๋”๋ฅผ ์ง€์ •ํ•˜๋ฉด ํ•ด๋‹น ํด๋”์— ์žˆ๋Š” ์Œ์„ฑ ํŒŒ์ผ์ด ์ž๋™์œผ๋กœ ๊ฐ€์ ธ์™€์ง‘๋‹ˆ๋‹ค.
์˜ˆ๋ฅผ ๋“ค์–ด `C:Users\hoge\voices`๋ฅผ ์ง€์ •ํ•˜๋ฉด `C:Users\hoge\voices\voice.mp3`๊ฐ€ ์ฝํžˆ์ง€๋งŒ `C:Users\hoge\voices\dir\voice.mp3`๋Š” ์ฝํžˆ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

์Œ์„ฑ ๋กœ๋“œ์—๋Š” ๋‚ด๋ถ€์ ์œผ๋กœ ffmpeg๋ฅผ ์ด์šฉํ•˜๊ณ  ์žˆ์œผ๋ฏ€๋กœ, ffmpeg๋กœ ๋Œ€์‘ํ•˜๊ณ  ์žˆ๋Š” ํ™•์žฅ์ž๋ผ๋ฉด ์ž๋™์ ์œผ๋กœ ์ฝํž™๋‹ˆ๋‹ค.
ffmpeg์—์„œ int16์œผ๋กœ ๋ณ€ํ™˜ํ•œ ํ›„ float32๋กœ ๋ณ€ํ™˜ํ•˜๊ณ  -1๊ณผ 1 ์‚ฌ์ด์— ์ •๊ทœํ™”๋ฉ๋‹ˆ๋‹ค.

### ์žก์Œ ์ œ๊ฑฐ
์Œ์„ฑ ํŒŒ์ผ์— ๋Œ€ํ•ด scipy์˜ filtfilt๋ฅผ ์ด์šฉํ•˜์—ฌ ์žก์Œ์„ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค.

### ์Œ์„ฑ ๋ถ„ํ• 
์ž…๋ ฅํ•œ ์Œ์„ฑ ํŒŒ์ผ์€ ๋จผ์ € ์ผ์ • ๊ธฐ๊ฐ„(max_sil_kept=5์ดˆ?)๋ณด๋‹ค ๊ธธ๊ฒŒ ๋ฌด์Œ์ด ์ง€์†๋˜๋Š” ๋ถ€๋ถ„์„ ๊ฐ์ง€ํ•˜์—ฌ ์Œ์„ฑ์„ ๋ถ„ํ• ํ•ฉ๋‹ˆ๋‹ค.๋ฌด์Œ์œผ๋กœ ์Œ์„ฑ์„ ๋ถ„ํ• ํ•œ ํ›„์—๋Š” 0.3์ดˆ์˜ overlap์„ ํฌํ•จํ•˜์—ฌ 4์ดˆ๋งˆ๋‹ค ์Œ์„ฑ์„ ๋ถ„ํ• ํ•ฉ๋‹ˆ๋‹ค.4์ดˆ ์ด๋‚ด์— ๊ตฌ๋ถ„๋œ ์Œ์„ฑ์€ ์Œ๋Ÿ‰์˜ ์ •๊ทœํ™”๋ฅผ ์‹ค์‹œํ•œ ํ›„ wav ํŒŒ์ผ์„ `/logs/์‹คํ—˜๋ช…/0_gt_wavs`๋กœ, ๊ฑฐ๊ธฐ์—์„œ 16k์˜ ์ƒ˜ํ”Œ๋ง ๋ ˆ์ดํŠธ๋กœ ๋ณ€ํ™˜ํ•ด `/logs/์‹คํ—˜๋ช…/1_16k_wavs`์— wav ํŒŒ์ผ๋กœ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.

## step2b
### ํ”ผ์น˜ ์ถ”์ถœ
wav ํŒŒ์ผ์—์„œ ํ”ผ์น˜(์†Œ๋ฆฌ์˜ ๋†’๋‚ฎ์ด) ์ •๋ณด๋ฅผ ์ถ”์ถœํ•ฉ๋‹ˆ๋‹ค. parselmouth๋‚˜ pyworld์— ๋‚ด์žฅ๋˜์–ด ์žˆ๋Š” ๋ฉ”์„œ๋“œ์œผ๋กœ ํ”ผ์น˜ ์ •๋ณด(=f0)๋ฅผ ์ถ”์ถœํ•ด, `/logs/์‹คํ—˜๋ช…/2a_f0`์— ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. ๊ทธ ํ›„ ํ”ผ์น˜ ์ •๋ณด๋ฅผ ๋กœ๊ทธ๋กœ ๋ณ€ํ™˜ํ•˜์—ฌ 1~255 ์ •์ˆ˜๋กœ ๋ณ€ํ™˜ํ•˜๊ณ  `/logs/์‹คํ—˜๋ช…/2b-f0nsf`์— ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.

### feature_print ์ถ”์ถœ
HuBERT๋ฅผ ์ด์šฉํ•˜์—ฌ wav ํŒŒ์ผ์„ ๋ฏธ๋ฆฌ embedding์œผ๋กœ ๋ณ€ํ™˜ํ•ฉ๋‹ˆ๋‹ค. `/logs/์‹คํ—˜๋ช…/1_16k_wavs`์— ์ €์žฅํ•œ wav ํŒŒ์ผ์„ ์ฝ๊ณ  HuBERT์—์„œ wav ํŒŒ์ผ์„ 256์ฐจ์› feature๋“ค๋กœ ๋ณ€ํ™˜ํ•œ ํ›„ npy ํ˜•์‹์œผ๋กœ `/logs/์‹คํ—˜๋ช…/3_feature256`์— ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.

## step3
๋ชจ๋ธ์˜ ํ›ˆ๋ จ์„ ์ง„ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

### ์ดˆ๋ณด์ž์šฉ ์šฉ์–ด ํ•ด์„ค
์‹ฌ์ธตํ•™์Šต(๋”ฅ๋Ÿฌ๋‹)์—์„œ๋Š” ๋ฐ์ดํ„ฐ์…‹์„ ๋ถ„ํ• ํ•˜์—ฌ ์กฐ๊ธˆ์”ฉ ํ•™์Šต์„ ์ง„ํ–‰ํ•ฉ๋‹ˆ๋‹ค.ํ•œ ๋ฒˆ์˜ ๋ชจ๋ธ ์—…๋ฐ์ดํŠธ(step) ๋‹จ๊ณ„ ๋‹น batch_size๊ฐœ์˜ ๋ฐ์ดํ„ฐ๋ฅผ ํƒ์ƒ‰ํ•˜์—ฌ ์˜ˆ์ธก๊ณผ ์˜ค์ฐจ๋ฅผ ์ˆ˜์ •ํ•ฉ๋‹ˆ๋‹ค. ๋ฐ์ดํ„ฐ์…‹ ์ „๋ถ€์— ๋Œ€ํ•ด ์ด ์ž‘์—…์„ ํ•œ ๋ฒˆ ์ˆ˜ํ–‰ํ•˜๋Š” ์ด๋ฅผ ํ•˜๋‚˜์˜ epoch๋ผ๊ณ  ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค.

๋”ฐ๋ผ์„œ ํ•™์Šต ์‹œ๊ฐ„์€ ๋‹จ๊ณ„๋‹น ํ•™์Šต ์‹œ๊ฐ„ x (๋ฐ์ดํ„ฐ์…‹ ๋‚ด ๋ฐ์ดํ„ฐ์˜ ์ˆ˜ / batch size) x epoch ์ˆ˜๊ฐ€ ์†Œ์š”๋ฉ๋‹ˆ๋‹ค. ์ผ๋ฐ˜์ ์œผ๋กœ batch size๊ฐ€ ํด์ˆ˜๋ก ํ•™์Šต์ด ์•ˆ์ •์ ์ด๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. (step๋‹น ํ•™์Šต ์‹œ๊ฐ„ รท batch size)๋Š” ์ž‘์•„์ง€์ง€๋งŒ GPU ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ๋” ๋งŽ์ด ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. GPU RAM์€ nvidia-smi ๋ช…๋ น์–ด๋ฅผ ํ†ตํ•ด ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์‹คํ–‰ ํ™˜๊ฒฝ์— ๋”ฐ๋ผ ๋ฐฐ์น˜ ํฌ๊ธฐ๋ฅผ ์ตœ๋Œ€ํ•œ ๋Š˜๋ฆฌ๋ฉด ์งง์€ ์‹œ๊ฐ„ ๋‚ด์— ํ•™์Šต์ด ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค.

### ์‚ฌ์ „ ํ•™์Šต๋œ ๋ชจ๋ธ ์ง€์ •
RVC๋Š” ์ ์€ ๋ฐ์ดํ„ฐ์…‹์œผ๋กœ๋„ ํ›ˆ๋ จ์ด ๊ฐ€๋Šฅํ•˜๋„๋ก ์‚ฌ์ „ ํ›ˆ๋ จ๋œ ๊ฐ€์ค‘์น˜์—์„œ ๋ชจ๋ธ ํ›ˆ๋ จ์„ ์‹œ์ž‘ํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ์ ์œผ๋กœ `rvc-location/pretrained/f0G40k.pth` ๋ฐ `rvc-location/pretrained/f0D40k.pth`๋ฅผ ๋ถˆ๋Ÿฌ์˜ต๋‹ˆ๋‹ค. ํ•™์Šต์„ ํ•  ์‹œ์—, ๋ชจ๋ธ ํŒŒ๋ผ๋ฏธํ„ฐ๋Š” ๊ฐ save_every_epoch๋ณ„๋กœ `logs/experiment name/G_{}.pth` ์™€ `logs/experiment name/D_{}.pth`๋กœ ์ €์žฅ์ด ๋˜๋Š”๋ฐ, ์ด ๊ฒฝ๋กœ๋ฅผ ์ง€์ •ํ•จ์œผ๋กœ์จ ํ•™์Šต์„ ์žฌ๊ฐœํ•˜๊ฑฐ๋‚˜, ๋‹ค๋ฅธ ์‹คํ—˜์—์„œ ํ•™์Šตํ•œ ๋ชจ๋ธ์˜ ๊ฐ€์ค‘์น˜์—์„œ ํ•™์Šต์„ ์‹œ์ž‘ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

### index์˜ ํ•™์Šต
RVC์—์„œ๋Š” ํ•™์Šต์‹œ์— ์‚ฌ์šฉ๋œ HuBERT์˜ feature๊ฐ’์„ ์ €์žฅํ•˜๊ณ , ์ถ”๋ก  ์‹œ์—๋Š” ํ•™์Šต ์‹œ ์‚ฌ์šฉํ•œ feature๊ฐ’๊ณผ ์œ ์‚ฌํ•œ feature ๊ฐ’์„ ํƒ์ƒ‰ํ•ด ์ถ”๋ก ์„ ์ง„ํ–‰ํ•ฉ๋‹ˆ๋‹ค. ์ด ํƒ์ƒ‰์„ ๊ณ ์†์œผ๋กœ ์ˆ˜ํ–‰ํ•˜๊ธฐ ์œ„ํ•ด ์‚ฌ์ „์— index์„ ํ•™์Šตํ•˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค.
Index ํ•™์Šต์—๋Š” ๊ทผ์‚ฌ ๊ทผ์ ‘ ํƒ์ƒ‰๋ฒ• ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์ธ Faiss๋ฅผ ์‚ฌ์šฉํ•˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. `/logs/์‹คํ—˜๋ช…/3_feature256`์˜ feature๊ฐ’์„ ๋ถˆ๋Ÿฌ์™€, ์ด๋ฅผ ๋ชจ๋‘ ๊ฒฐํ•ฉ์‹œํ‚จ feature๊ฐ’์„ `/logs/์‹คํ—˜๋ช…/total_fea.npy`๋กœ์„œ ์ €์žฅ, ๊ทธ๊ฒƒ์„ ์‚ฌ์šฉํ•ด ํ•™์Šตํ•œ index๋ฅผ`/logs/์‹คํ—˜๋ช…/add_XXX.index`๋กœ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.

### ๋ฒ„ํŠผ ์„ค๋ช…
- ใƒขใƒ‡ใƒซใฎใƒˆใƒฌใƒผใƒ‹ใƒณใ‚ฐ (๋ชจ๋ธ ํ•™์Šต): step2b๊นŒ์ง€ ์‹คํ–‰ํ•œ ํ›„, ์ด ๋ฒ„ํŠผ์„ ๋ˆŒ๋Ÿฌ ๋ชจ๋ธ์„ ํ•™์Šตํ•ฉ๋‹ˆ๋‹ค.
- ็‰นๅพดใ‚คใƒณใƒ‡ใƒƒใ‚ฏใ‚นใฎใƒˆใƒฌใƒผใƒ‹ใƒณใ‚ฐ (ํŠน์ง• ์ง€์ˆ˜ ํ›ˆ๋ จ): ๋ชจ๋ธ์˜ ํ›ˆ๋ จ ํ›„, index๋ฅผ ํ•™์Šตํ•ฉ๋‹ˆ๋‹ค.
- ใƒฏใƒณใ‚ฏใƒชใƒƒใ‚ฏใƒˆใƒฌใƒผใƒ‹ใƒณใ‚ฐ (์›ํด๋ฆญ ํŠธ๋ ˆ์ด๋‹): step2b๊นŒ์ง€์˜ ๋ชจ๋ธ ํ›ˆ๋ จ, feature index ํ›ˆ๋ จ์„ ์ผ๊ด„๋กœ ์‹ค์‹œํ•ฉ๋‹ˆ๋‹ค.