Upload model

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,31 @@
+{
+  "ACTIVATION": "gelu",
+  "BETA_SCHEDULER": "cosine",
+  "CFG_WEIGHT": 3,
+  "CFG_WEIGHT_INDIVIDUAL": 1,
+  "CFG_WEIGHT_INTERACTION": 3,
+  "CONTROL": "text",
+  "DIFFUSION_STEPS": 1000,
+  "DROPOUT": 0.1,
+  "FF_SIZE": 2048,
+  "FINETUNE": false,
+  "INPUT_DIM": 262,
+  "LATENT_DIM": 1024,
+  "MODE": "interaction",
+  "MOTION_REP": "global",
+  "NUM_HEADS": 8,
+  "NUM_LAYERS": 8,
+  "SAMPLER": "uniform",
+  "STRATEGY": "ddim50",
+  "TEXT_ENCODER": "clip",
+  "T_BAR": 700,
+  "architectures": [
+    "in2INModel"
+  ],
+  "auto_map": {
+    "AutoConfig": "config.in2INConfig",
+    "AutoModel": "model.in2INModel"
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.41.2"
+}

config.py

@@ -0,0 +1,48 @@
+from transformers import PretrainedConfig
+
+class in2INConfig(PretrainedConfig): 
+    def __init__(self, 
+                 num_layers=8,
+                 num_heads=8,
+                 dropout=0.1,
+                 input_dim=262,
+                 latent_dim=1024,
+                 ff_size=2048,
+                 activation="gelu",
+                 diffusion_steps=1000,
+                 beta_scheduler="cosine",
+                 sampler="uniform",
+                 motion_rep="global",
+                 finetune=False,
+                 text_encoder="clip",
+                 t_bar=700,
+                 control="text",
+                 strategy="ddim50",
+                 cfg_weight=3,
+                 cfg_weight_interaction=3,
+                 cfg_weight_individual=1, 
+                 mode="interaction", 
+                 **kwargs):
+        
+        self.NUM_LAYERS = num_layers
+        self.NUM_HEADS = num_heads
+        self.DROPOUT = dropout
+        self.INPUT_DIM = input_dim
+        self.LATENT_DIM = latent_dim
+        self.FF_SIZE = ff_size
+        self.ACTIVATION = activation
+        self.DIFFUSION_STEPS = diffusion_steps
+        self.BETA_SCHEDULER = beta_scheduler
+        self.SAMPLER = sampler
+        self.MOTION_REP = motion_rep
+        self.FINETUNE = finetune
+        self.TEXT_ENCODER = text_encoder
+        self.T_BAR = t_bar
+        self.CONTROL = control
+        self.STRATEGY = strategy
+        self.CFG_WEIGHT = cfg_weight
+        self.CFG_WEIGHT_INTERACTION = cfg_weight_interaction
+        self.CFG_WEIGHT_INDIVIDUAL = cfg_weight_individual
+        self.MODE = mode
+
+        super().__init__(**kwargs)

model.py

@@ -0,0 +1,72 @@
+import torch
+import copy
+import numpy as np
+
+from typing import OrderedDict
+from scipy.ndimage import gaussian_filter1d
+
+from transformers import PreTrainedModel
+from in2in.utils.configs import get_config
+from in2in.models.in2in import in2IN
+
+from .config import in2INConfig
+
+class in2INModel(PreTrainedModel):
+    config_class = in2INConfig
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = in2IN(config, mode=config.MODE)
+
+    def forward(self, prompt_interaction, prompt_individual1, prompt_individual2):
+        self.model.eval()
+        batch = OrderedDict({})
+
+        batch["motion_lens"] = torch.zeros(1,1).long().cuda()
+        batch["prompt_interaction"] = prompt_interaction
+
+        if self.mode != "individual":
+            batch["prompt_individual1"] = prompt_individual1
+            batch["prompt_individual2"] = prompt_individual2
+
+        window_size = 210
+        motion_output = self.generate_loop(batch, window_size)
+        return motion_output
+
+    def generate_loop(self, batch, window_size):
+        prompt_interaction = batch["prompt_interaction"]
+
+        if self.mode != "individual":
+            prompt_individual1 = batch["prompt_individual1"]
+            prompt_individual2 = batch["prompt_individual2"]
+
+        batch = copy.deepcopy(batch)
+        batch["motion_lens"][:] = window_size
+
+        batch["text"] = [prompt_interaction]
+        if self.mode != "individual":
+            batch["text_individual1"] = [prompt_individual1]
+            batch["text_individual2"] = [prompt_individual2]
+
+        batch = self.model.forward_test(batch)
+
+        if self.mode == "individual":
+            motion_output = batch["output"][0].reshape(-1, 262)
+            motion_output = self.normalizer.backward(motion_output.cpu().detach().numpy())
+            joints3d = motion_output[:,:22*3].reshape(-1,22,3)
+            joints3d = gaussian_filter1d(joints3d, 1, axis=0, mode='nearest')
+            return joints3d
+
+        motion_output_both = batch["output"][0].reshape(batch["output"][0].shape[0], 2, -1)
+        motion_output_both = self.normalizer.backward(motion_output_both.cpu().detach().numpy())
+        
+        sequences = [[], []]
+        for j in range(2):
+            motion_output = motion_output_both[:,j]
+            joints3d = motion_output[:,:22*3].reshape(-1,22,3)
+            joints3d = gaussian_filter1d(joints3d, 1, axis=0, mode='nearest')
+            sequences[j].append(joints3d)
+
+        sequences[0] = np.concatenate(sequences[0], axis=0)
+        sequences[1] = np.concatenate(sequences[1], axis=0)
+        return sequences

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:886d29d2234e2f114ef4221b99a1b01e3d4b739de69c97793c6d869f43606463
+size 1242367342