Sintayew Gashaw

README.md

@@ -1,8 +1,7 @@
 ---
-pipeline_tag: image-to-video
-license: other
-license_name: stable-video-diffusion-nc-community
-license_link: LICENSE
+# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
+# Doc / guide: https://huggingface.co/docs/hub/model-cards
+{}
 ---
 
 # Stable Video Diffusion Image-to-Video Model Card
@@ -11,8 +10,6 @@ license_link: LICENSE
 ![row01](output_tile.gif)
 Stable Video Diffusion (SVD) Image-to-Video is a diffusion model that takes in a still image as a conditioning frame, and generates a video from it. 
 
-Please note: For commercial use, please refer to https://stability.ai/membership.
-
 ## Model Details
 
 ### Model Description
@@ -47,7 +44,7 @@ SVD-Image-to-Video is preferred by human voters in terms of video quality. For d
 
 ### Direct Use
 
-The model is intended for both non-commercial and commercial usage. You can use this model for non-commercial or research purposes under this [license](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt/blob/main/LICENSE). Possible research areas and tasks include
+The model is intended for research purposes only. Possible research areas and tasks include
 
 - Research on generative models.
 - Safe deployment of models which have the potential to generate harmful content.
@@ -55,8 +52,6 @@ The model is intended for both non-commercial and commercial usage. You can use
 - Generation of artworks and use in design and other artistic processes.
 - Applications in educational or creative tools.
 
-For commercial use, please refer to https://stability.ai/membership.
-
 Excluded uses are described below.
 
 ### Out-of-Scope Use
@@ -78,22 +73,11 @@ The model should not be used in any way that violates Stability AI's [Acceptable
 
 ### Recommendations
 
-The model is intended for both non-commercial and commercial usage.
+The model is intended for research purposes only.
 
 ## How to Get Started with the Model
 
 Check out https://github.com/Stability-AI/generative-models
 
-# Appendix: 
-
-All considered potential data sources were included for final training, with none held out as the proposed data filtering methods described in the SVD paper handle the quality control/filtering of the dataset. With regards to safety/NSFW filtering, sources considered were either deemed safe or filtered with the in-house NSFW filters.
-No explicit human labor is involved in training data preparation. However, human evaluation for model outputs and quality was extensively used to evaluate model quality and performance. The evaluations were performed with third-party contractor platforms (Amazon Sagemaker, Amazon Mechanical Turk, Prolific) with fluent English-speaking contractors from various countries, primarily from the USA, UK, and Canada. Each worker was paid $12/hr for the time invested in the evaluation.
-No other third party was involved in the development of this model; the model was fully developed in-house at Stability AI.
-Training the SVD checkpoints required a total of approximately 200,000 A100 80GB hours. The majority of the training occurred on 48 * 8 A100s, while some stages took more/less than that. The resulting CO2 emission is ~19,000kg CO2 eq., and energy consumed is ~64000 kWh.
-The released checkpoints (SVD/SVD-XT) are image-to-video models that generate short videos/animations closely following the given input image. Since the model relies on an existing supplied image, the potential risks of disclosing specific material or novel unsafe content are minimal. This was also evaluated by third-party independent red-teaming services, which agree with our conclusion to a high degree of confidence (>90% in various areas of safety red-teaming). The external evaluations were also performed for trustworthiness, leading to >95% confidence in real, trustworthy videos.
-With the default settings at the time of release, SVD takes ~100s for generation, and SVD-XT takes ~180s on an A100 80GB card. Several optimizations to trade off quality / memory / speed can be done to perform faster inference or inference on lower VRAM cards.
-The information related to the model and its development process and usage protocols can be found in the GitHub repo, associated research paper, and HuggingFace model page/cards. 
-The released model inference & demo code has image-level watermarking enabled by default, which can be used to detect the outputs. This is done via the imWatermark Python library.  
-The model can be used to generate videos from static initial images. However, we prohibit unlawful, obscene, or misleading uses of the model consistent with the terms of our license and Acceptable Use Policy. For the open-weights release, our training data filtering mitigations alleviate this risk to some extent. These restrictions are explicitly enforced on user-facing interfaces at stablevideo.com, where a warning is issued. We do not take any responsibility for third-party interfaces. Submitting initial images that bypass input filters to tease out offensive or inappropriate content listed above is also prohibited. Safety filtering checks at stablevideo.com run on model inputs and outputs independently. More details on our user-facing interfaces can be found here: https://www.stablevideo.com/faq. Beyond the Acceptable Use Policy and other mitigations and conditions described here, the model is not subject to additional model behavior interventions of the type described in the Foundation Model Transparency Index.
-For stablevideo.com, we store preference data in the form of upvotes/downvotes on user-generated videos, and we have a pairwise ranker that runs while a user generates videos. This usage data is solely used for improving Stability AI’s future image/video models and services. No other third-party entities are given access to the usage data beyond Stability AI and maintainers of stablevideo.com. 
-For usage statistics of SVD, we refer interested users to HuggingFace model download/usage statistics as a primary indicator. Third-party applications also have reported model usage statistics. We might also consider releasing aggregate usage statistics of stablevideo.com on reaching some milestones.
+
+

feature_extractor/preprocessor_config.json

@@ -1,28 +0,0 @@
-{
-  "crop_size": {
-    "height": 224,
-    "width": 224
-  },
-  "do_center_crop": true,
-  "do_convert_rgb": true,
-  "do_normalize": true,
-  "do_rescale": true,
-  "do_resize": true,
-  "feature_extractor_type": "CLIPFeatureExtractor",
-  "image_mean": [
-    0.48145466,
-    0.4578275,
-    0.40821073
-  ],
-  "image_processor_type": "CLIPImageProcessor",
-  "image_std": [
-    0.26862954,
-    0.26130258,
-    0.27577711
-  ],
-  "resample": 3,
-  "rescale_factor": 0.00392156862745098,
-  "size": {
-    "shortest_edge": 224
-  }
-}

image_encoder/config.json

@@ -1,23 +0,0 @@
-{
-  "_name_or_path": "/home/suraj_huggingface_co/.cache/huggingface/hub/models--diffusers--svd-xt/snapshots/9703ded20c957c340781ee710b75660826deb487/image_encoder",
-  "architectures": [
-    "CLIPVisionModelWithProjection"
-  ],
-  "attention_dropout": 0.0,
-  "dropout": 0.0,
-  "hidden_act": "gelu",
-  "hidden_size": 1280,
-  "image_size": 224,
-  "initializer_factor": 1.0,
-  "initializer_range": 0.02,
-  "intermediate_size": 5120,
-  "layer_norm_eps": 1e-05,
-  "model_type": "clip_vision_model",
-  "num_attention_heads": 16,
-  "num_channels": 3,
-  "num_hidden_layers": 32,
-  "patch_size": 14,
-  "projection_dim": 1024,
-  "torch_dtype": "float16",
-  "transformers_version": "4.34.0.dev0"
-}

image_encoder/model.fp16.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ae616c24393dd1854372b0639e5541666f7521cbe219669255e865cb7f89466a
-size 1264217240

image_encoder/model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ed1e5af7b4042ca30ec29999a4a5cfcac90b7fb610fd05ace834f2dcbb763eab
-size 2528371296

model_index.json

@@ -1,25 +0,0 @@
-{
-  "_class_name": "StableVideoDiffusionPipeline",
-  "_diffusers_version": "0.24.0.dev0",
-  "_name_or_path": "diffusers/svd-xt",
-  "feature_extractor": [
-    "transformers",
-    "CLIPImageProcessor"
-  ],
-  "image_encoder": [
-    "transformers",
-    "CLIPVisionModelWithProjection"
-  ],
-  "scheduler": [
-    "diffusers",
-    "EulerDiscreteScheduler"
-  ],
-  "unet": [
-    "diffusers",
-    "UNetSpatioTemporalConditionModel"
-  ],
-  "vae": [
-    "diffusers",
-    "AutoencoderKLTemporalDecoder"
-  ]
-}

scheduler/scheduler_config.json

@@ -1,20 +0,0 @@
-{
-  "_class_name": "EulerDiscreteScheduler",
-  "_diffusers_version": "0.24.0.dev0",
-  "beta_end": 0.012,
-  "beta_schedule": "scaled_linear",
-  "beta_start": 0.00085,
-  "clip_sample": false,
-  "interpolation_type": "linear",
-  "num_train_timesteps": 1000,
-  "prediction_type": "v_prediction",
-  "set_alpha_to_one": false,
-  "sigma_max": 700.0,
-  "sigma_min": 0.002,
-  "skip_prk_steps": true,
-  "steps_offset": 1,
-  "timestep_spacing": "leading",
-  "timestep_type": "continuous",
-  "trained_betas": null,
-  "use_karras_sigmas": true
-}

unet/config.json

@@ -1,38 +0,0 @@
-{
-  "_class_name": "UNetSpatioTemporalConditionModel",
-  "_diffusers_version": "0.24.0.dev0",
-  "_name_or_path": "/home/suraj_huggingface_co/.cache/huggingface/hub/models--diffusers--svd-xt/snapshots/9703ded20c957c340781ee710b75660826deb487/unet",
-  "addition_time_embed_dim": 256,
-  "block_out_channels": [
-    320,
-    640,
-    1280,
-    1280
-  ],
-  "cross_attention_dim": 1024,
-  "down_block_types": [
-    "CrossAttnDownBlockSpatioTemporal",
-    "CrossAttnDownBlockSpatioTemporal",
-    "CrossAttnDownBlockSpatioTemporal",
-    "DownBlockSpatioTemporal"
-  ],
-  "in_channels": 8,
-  "layers_per_block": 2,
-  "num_attention_heads": [
-    5,
-    10,
-    20,
-    20
-  ],
-  "num_frames": 25,
-  "out_channels": 4,
-  "projection_class_embeddings_input_dim": 768,
-  "sample_size": 96,
-  "transformer_layers_per_block": 1,
-  "up_block_types": [
-    "UpBlockSpatioTemporal",
-    "CrossAttnUpBlockSpatioTemporal",
-    "CrossAttnUpBlockSpatioTemporal",
-    "CrossAttnUpBlockSpatioTemporal"
-  ]
-}

unet/diffusion_pytorch_model.fp16.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:9fbc02e90f37d422f5e3a4aeaee95f6629dc8c45ca211b951626e930daf2bddf
-size 3049435868

unet/diffusion_pytorch_model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7783d82729af04f26ded4641a5952617fe331fc46add332fb9e47674fecc6ad7
-size 6098682464

vae/config.json

@@ -1,24 +0,0 @@
-{
-  "_class_name": "AutoencoderKLTemporalDecoder",
-  "_diffusers_version": "0.24.0.dev0",
-  "_name_or_path": "/home/suraj_huggingface_co/.cache/huggingface/hub/models--diffusers--svd-xt/snapshots/9703ded20c957c340781ee710b75660826deb487/vae",
-  "block_out_channels": [
-    128,
-    256,
-    512,
-    512
-  ],
-  "down_block_types": [
-    "DownEncoderBlock2D",
-    "DownEncoderBlock2D",
-    "DownEncoderBlock2D",
-    "DownEncoderBlock2D"
-  ],
-  "force_upcast": true,
-  "in_channels": 3,
-  "latent_channels": 4,
-  "layers_per_block": 2,
-  "out_channels": 3,
-  "sample_size": 768,
-  "scaling_factor": 0.18215
-}

vae/diffusion_pytorch_model.fp16.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:af602cd0eb4ad6086ec94fbf1438dfb1be5ec9ac03fd0215640854e90d6463a3
-size 195531910

vae/diffusion_pytorch_model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5d92aa595a53d9da9faf594f09910ee869d5d567c8bb0362d5095673c69997d6
-size 391017740