anonderpling commited on
Commit
a623acd
1 Parent(s): 0412886

note dupe VAEs

Browse files
Files changed (1) hide show
  1. models/VAE/README.md +22 -0
models/VAE/README.md ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ I ram this script:
2
+
3
+ ```
4
+ find -type f|while read a; do cat "$a"|grep oid|cut -d" " -f2|sponge "$a"; done
5
+ jdupes . -r
6
+ ```
7
+
8
+ which toldnme that these files are identical.
9
+
10
+ ```
11
+ ./kl-f8-anime2.vae.pt
12
+ ./vaedelicatecolcors_v10.pt
13
+
14
+ ./babes_11.vae.pt
15
+ ./vae-ft-mse-840000-ema-pruned.ckpt
16
+
17
+ ./Anything-V3.0.vae.pt
18
+ ./Counterfeit-V2.5.vae.pt
19
+ ./NovelAI-vae via hf.co_Reviem/diffusion_pytorch_model.bin
20
+ ```
21
+
22
+ If you're storing on cloud storage, you can deduplicate with shortcuts (google drive) or notes; on linux you can use hard links (`rm vaedelicatecolors_v10.pt && ln kl-f8-anime2.vae.pt vaedelicatecolors_v10.pt`. if youre storing in a git repo, you can just use a sparse checkout like I do (the lfs backend will dedupe serverside because thats how it works, at least on HF)