Update app.py

app.py

@@ -80,45 +80,64 @@ def predict(file_path: str):
 
 def create_demo():
 
-    USAGE = """## Input data format
-    Currently, the demo accepts a `.pkl` file containing an hand-object sequence organized as the following format:
+    USAGE = """# QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfer
+    **[Project](https://meowuu7.github.io/QuasiSim/) | [Github](https://github.com/Meowuu7/QuasiSim)**
+    This demo transforms the input human manipulation demonstration to the trajectory of `point set` (a relaxed representation of articulated rigid object introduced in QuasiSim). It is the first step of the first stage of our optimization process. Please checkout our [github repo](https://github.com/Meowuu7/QuasiSim) for more details and instructions of running locally. 
+    ## Input data format
+    Currently, the demo accepts a `.npy` file containing a human manipulation trajectory organized as the following format: 
     ```python
         {
-            "hand_pose": numpy.ndarray(seq_length, 48), # MANO pose at each frame
-            "hand_trans": numpy.ndarray(seq_length, 3), # hand global translation at each frmae
-            "hand_shape": numpy.ndarray(10), # MANO shape coefficients
-            "hand_verts": numpy.ndarray(seq_length, 778, 3), # MANO hand vertices
-            "hand_faces": numpy.ndarray(1538, 3), # MANO hand faces
-            "obj_verts": numpy.ndarray(seq_length, num_obj_verts, 3), # object vertices at each frame
-            "obj_faces": numpy.ndarray(num_obj_faces, 3), # object faces
-            "obj_pose": numpy.ndarray(seq_length, 4, 4), # object pose at each frame
+            "sv_dict": {
+                "rhand_global_orient_gt": numpy.ndarray(seq_length, 3), # MANO global orientation coefficient
+                "rhand_transl": numpy.ndarray(seq_length, 3), # MANO global translation coefficient
+                "rhand_verts": numpy.ndarray(seq_length, 778,  3), # MANO hand vertices
+                "object_global_orient": numpy.ndarray(seq_length,  3), # Object global orientation (represented as rotation vectors, check below for details w.r.t. how to convert it to the rotation matrix)
+                "object_transl": numpy.ndarray(seq_length,  3), # Object global translations 
+                "obj_faces": numpy.ndarray(nn_faces,  3), # Object mesh  faces
+                "obj_verts": numpy.ndarray(nn_vertices,  3), # Object mesh  vertices
+                "obj_vertex_normals": numpy.ndarray(nn_vertices,  3), # Object mesh vertex normals
+            }, 
+            "obj_sdf": numpy.ndarray(sdf_res, sdf_res, sdf_res), # Pre-processed object SDF values (see below for SDF processing details)
         }
     ```
-    We provide an example [here](https://drive.google.com/file/d/17oqKMhQNpRqSdApyuuCmTrPkrFl0Cqp6/view?usp=sharing). **The demo is under developing and will support more data formats in the future.**
+    **How to transform the object global orientation to the rotation matrix**: The object global orientation is represented as the rotation vector. To convert it to the rotation matrix, you can use `scipy.spatial.transform.Rotation` as follows:
+    ````python
+        from scipy.spatial.transform import Rotation
+        r = Rotation.from_rotvec(object_global_orient)
+        object_global_orient_rotmat = r.as_matrix()
+    ```
+    To transform the canonical object mesh vertices using the orientation vector (`object_global_orient`) and the global translation (`object_global_trans`), you can use the following code:
+    ```python
+        from scipy.spatial.transform import Rotation
+        r = Rotation.from_rotvec(object_global_orient)
+        object_global_orient_rotmat = r.as_matrix()
+        cur_transformed_verts = np.matmul(
+            obj_verts, object_global_orient_rotmat
+        ) + object_global_trans[None, :]
+    ```
+    We use [mesh-to-sdf](https://github.com/wang-ps/mesh2sdf) to pre-process the object mesh to the SDF values. The SDF values are stored in a 3D numpy array with the shape of `(sdf_res, sdf_res, sdf_res)`, where `sdf_res` is set to `128` in our experiments. Please check out [compute sdf](https://github.com/Meowuu7/QuasiSim/blob/main/utils/grab_preprocessing.py#L151) for our pre-processing function. 
+    
+    Currently, the demo only accepts input trajectories with 60 frames. 
+    We provide an example [here](https://1drv.ms/u/s!AgSPtac7QUbHgVncbYUdKI1f5TvE?e=sRhirK).
     
     
     ## To run the demo, 
-    1. Upload a `pickle` file to the left box by draging your file or clicking the box to open the file explorer.
+    1. Upload a `numpy` file to the left box by draging your file or clicking the box to open the file explorer.
     2. Clik the `Submit` button to run the demo.
-    3. The denoised sequence will be output as a `.npy` file and can be downloaded from the right box. 
-    
-    Since the model runs on CPU currently, the speed is not very fast. For instance, it takes abount 1200s to process the [example](https://drive.google.com/file/d/17oqKMhQNpRqSdApyuuCmTrPkrFl0Cqp6/view?usp=sharing) mentioned above which contains 288 frames. Please be patient and wait for the result. 
+    3. The optimized trajectory of the point set will be output as a `.npy` file and can be downloaded from the right box. 
     
-    To run the model faster, please visit our [github repo](https://github.com/Meowuu7/GeneOH-Diffusion), follow the instructions and run the model on your own server or local machine.
+    Since the model runs on CPU currently, the speed is quite slow. For instance, it takes around 32h (yeah, hours...) to process the [example](https://1drv.ms/u/s!AgSPtac7QUbHgVoY8jPkPZfrDkJw?e=JYFi5a) mentioned above which contains 60 frames. However, it takes only several minutes to complish when running on a GPU! Therefore, we highly recommand checking out our [github repo](https://github.com/Meowuu7/QuasiSim), setting up an environment with GPU support, and running it locally. 
     
     ## Output data format 
-    The output is a `.npy` file containing the denoised sequence organized as the following format:
+    The output is a `.npy` file containing the optimized trajectory of the point set sequence organized as a `dict` in the following format:
     ```python
         {
-            "predicted_info": xxx
-            "bf_ct_verts": numpy.ndarray(seq_length, 778, 3), # denoised MANO vertices
-            "bf_ct_rot_var": numpy.ndarray(seq_length, 3), # denoised MANO global rotation coefficients
-            "bf_ct_theta_var": numpy.ndarray(seq_length, 45), # denoised MANO global pose coefficients
-            "bf_ct_beta_var": numpy.ndarray(1, 10), # denoised MANO shape coefficients
-            "bf_ct_transl_var": numpy.ndarray(seq_length, 3), # denoised hand global translation
+            "ts_to_hand_obj_verts": { 
+                ts: (hand points (numpy.ndarray(number of points contained in the point set rep, 3)), object points (numpy.ndarray(nn_vertices, 3))) for ts in range(0, seq_length)
+            }
         }
     ```
-    The corresponding output file of the [example](https://drive.google.com/file/d/17oqKMhQNpRqSdApyuuCmTrPkrFl0Cqp6/view?usp=sharing) mentioned above can be downloaded [here](https://drive.google.com/file/d/1Ah-qwV6LXlOyaBBe0qQRu1lN-BpKt2Y3/view?usp=sharing).
+    The corresponding output file of the [example](https://1drv.ms/u/s!AgSPtac7QUbHgVoY8jPkPZfrDkJw?e=JYFi5a) mentioned above can be downloaded [here](https://1drv.ms/u/s!AgSPtac7QUbHgVoY8jPkPZfrDkJw?e=JYFi5a).
     """