Spaces:
Running
Running
Upload README.md
Browse files
README.md
CHANGED
|
@@ -1,145 +1,13 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
## Introduction
|
| 15 |
-
|
| 16 |
-
We present a novel approach to CAPE that leverages the inherent geometrical relations between keypoints through a newly designed Graph Transformer Decoder. By capturing and incorporating this crucial structural information, our method enhances the accuracy of keypoint localization, marking a significant departure from conventional CAPE techniques that treat keypoints as isolated entities.
|
| 17 |
-
|
| 18 |
-
## Citation
|
| 19 |
-
If you find this useful, please cite this work as follows:
|
| 20 |
-
```bibtex
|
| 21 |
-
@misc{hirschorn2023pose,
|
| 22 |
-
title={Pose Anything: A Graph-Based Approach for Category-Agnostic Pose Estimation},
|
| 23 |
-
author={Or Hirschorn and Shai Avidan},
|
| 24 |
-
year={2023},
|
| 25 |
-
eprint={2311.17891},
|
| 26 |
-
archivePrefix={arXiv},
|
| 27 |
-
primaryClass={cs.CV}
|
| 28 |
-
}
|
| 29 |
-
```
|
| 30 |
-
|
| 31 |
-
## Getting Started
|
| 32 |
-
|
| 33 |
-
### Docker [Recommended]
|
| 34 |
-
We provide a docker image for easy use.
|
| 35 |
-
You can simply pull the docker image from docker hub, containing all the required libraries and packages:
|
| 36 |
-
|
| 37 |
-
```
|
| 38 |
-
docker pull orhir/pose_anything
|
| 39 |
-
docker run --name pose_anything -v {DATA_DIR}:/workspace/PoseAnything/PoseAnything/data/mp100 -it orhir/pose_anything /bin/bash
|
| 40 |
-
```
|
| 41 |
-
### Conda Environment
|
| 42 |
-
We train and evaluate our model on Python 3.8 and Pytorch 2.0.1 with CUDA 12.1.
|
| 43 |
-
|
| 44 |
-
Please first install pytorch and torchvision following official documentation Pytorch.
|
| 45 |
-
Then, follow [MMPose](https://mmpose.readthedocs.io/en/latest/installation.html) to install the following packages:
|
| 46 |
-
```
|
| 47 |
-
mmcv-full=1.6.2
|
| 48 |
-
mmpose=0.29.0
|
| 49 |
-
```
|
| 50 |
-
Having installed these packages, run:
|
| 51 |
-
```
|
| 52 |
-
python setup.py develop
|
| 53 |
-
```
|
| 54 |
-
|
| 55 |
-
## Demo on Custom Images
|
| 56 |
-
We provide a demo code to test our code on custom images.
|
| 57 |
-
|
| 58 |
-
***A bigger and more accurate version of the model - COMING SOON!***
|
| 59 |
-
|
| 60 |
-
### Gradio Demo
|
| 61 |
-
We first require to install gradio:
|
| 62 |
-
```
|
| 63 |
-
pip install gradio==3.44.0
|
| 64 |
-
```
|
| 65 |
-
Then, Download the [pretrained model](https://drive.google.com/file/d/1RT1Q8AMEa1kj6k9ZqrtWIKyuR4Jn4Pqc/view?usp=drive_link) and run:
|
| 66 |
-
```
|
| 67 |
-
python app.py --checkpoint [path_to_pretrained_ckpt]
|
| 68 |
-
```
|
| 69 |
-
### Terminal Demo
|
| 70 |
-
Download
|
| 71 |
-
the [pretrained model](https://drive.google.com/file/d/1RT1Q8AMEa1kj6k9ZqrtWIKyuR4Jn4Pqc/view?usp=drive_link)
|
| 72 |
-
and run:
|
| 73 |
-
|
| 74 |
-
```
|
| 75 |
-
python demo.py --support [path_to_support_image] --query [path_to_query_image] --config configs/demo_b.py --checkpoint [path_to_pretrained_ckpt]
|
| 76 |
-
```
|
| 77 |
-
***Note:*** The demo code supports any config with suitable checkpoint file. More pre-trained models can be found in the evaluation section.
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
## MP-100 Dataset
|
| 81 |
-
Please follow the [official guide](https://github.com/luminxu/Pose-for-Everything/blob/main/mp100/README.md) to prepare the MP-100 dataset for training and evaluation, and organize the data structure properly.
|
| 82 |
-
|
| 83 |
-
We provide an updated annotation file, which includes skeleton definitions, in the following [link](https://drive.google.com/drive/folders/1uRyGB-P5Tc_6TmAZ6RnOi0SWjGq9b28T?usp=sharing).
|
| 84 |
-
|
| 85 |
-
**Please note:**
|
| 86 |
-
|
| 87 |
-
Current version of the MP-100 dataset includes some discrepancies and filenames errors:
|
| 88 |
-
1. Note that the mentioned DeepFasion dataset is actually DeepFashion2 dataset. The link in the official repo is wrong. Use this [repo](https://github.com/switchablenorms/DeepFashion2/tree/master) instead.
|
| 89 |
-
2. We provide a script to fix CarFusion filename errors, which can be run by:
|
| 90 |
-
```
|
| 91 |
-
python tools/fix_carfusion.py [path_to_CarFusion_dataset] [path_to_mp100_annotation]
|
| 92 |
-
```
|
| 93 |
-
|
| 94 |
-
## Training
|
| 95 |
-
|
| 96 |
-
### Backbone Options
|
| 97 |
-
To use pre-trained Swin-Transformer as used in our paper, we provide the weights, taken from this [repo](https://github.com/microsoft/Swin-Transformer/blob/main/MODELHUB.md), in the following [link](https://drive.google.com/drive/folders/1-q4mSxlNAUwDlevc3Hm5Ij0l_2OGkrcg?usp=sharing).
|
| 98 |
-
These should be placed in the `./pretrained` folder.
|
| 99 |
-
|
| 100 |
-
We also support DINO and ResNet backbones. To use them, you can easily change the config file to use the desired backbone.
|
| 101 |
-
This can be done by changing the `pretrained` field in the config file to `dinov2`, `dino` or `resnet` respectively (this will automatically load the pretrained weights from the official repo).
|
| 102 |
-
|
| 103 |
-
### Training
|
| 104 |
-
To train the model, run:
|
| 105 |
-
```
|
| 106 |
-
python train.py --config [path_to_config_file] --work-dir [path_to_work_dir]
|
| 107 |
-
```
|
| 108 |
-
|
| 109 |
-
## Evaluation and Pretrained Models
|
| 110 |
-
You can download the pretrained checkpoints from following [link](https://drive.google.com/drive/folders/1RmrqzE3g0qYRD5xn54-aXEzrIkdYXpEW?usp=sharing).
|
| 111 |
-
|
| 112 |
-
Here we provide the evaluation results of our pretrained models on MP-100 dataset along with the config files and checkpoints:
|
| 113 |
-
|
| 114 |
-
### 1-Shot Models
|
| 115 |
-
| Setting | split 1 | split 2 | split 3 | split 4 | split 5 |
|
| 116 |
-
|:-------:|:---------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------:|
|
| 117 |
-
| Tiny | 91.06 | 88024 | 86.09 | 86.17 | 85.78 |
|
| 118 |
-
| | [link](https://drive.google.com/file/d/1GubmkVkqybs-eD4hiRkgBzkUVGE_rIFX/view?usp=drive_link) / [config](configs/1shots/graph_split1_config.py) | [link](https://drive.google.com/file/d/1EEekDF3xV_wJOVk7sCQWUA8ygUKzEm2l/view?usp=drive_link) / [config](configs/1shots/graph_split2_config.py) | [link](https://drive.google.com/file/d/1FuwpNBdPI3mfSovta2fDGKoqJynEXPZQ/view?usp=drive_link) / [config](configs/1shots/graph_split3_config.py) | [link](https://drive.google.com/file/d/1_SSqSANuZlbC0utzIfzvZihAW9clefcR/view?usp=drive_link) / [config](configs/1shots/graph_split4_config.py) | [link](https://drive.google.com/file/d/1nUHr07W5F55u-FKQEPFq_CECgWZOKKLF/view?usp=drive_link) / [config](configs/1shots/graph_split5_config.py) |
|
| 119 |
-
| Small | 93.66 | 90.42 | 89.79 | 88.68 | 89.61 |
|
| 120 |
-
| | [link](https://drive.google.com/file/d/1RT1Q8AMEa1kj6k9ZqrtWIKyuR4Jn4Pqc/view?usp=drive_link) / [config](configs/1shot-swin/graph_split1_config.py) | [link](https://drive.google.com/file/d/1BT5b8MlnkflcdhTFiBROIQR3HccLsPQd/view?usp=drive_link) / [config](configs/1shot-swin/graph_split2_config.py) | [link](https://drive.google.com/file/d/1Z64cw_1CSDGObabSAWKnMK0BA_bqDHxn/view?usp=drive_link) / [config](configs/1shot-swin/graph_split3_config.py) | [link](https://drive.google.com/file/d/1vf82S8LAjIzpuBcbEoDCa26cR8DqNriy/view?usp=drive_link) / [config](configs/1shot-swin/graph_split4_config.py) | [link](https://drive.google.com/file/d/14FNx0JNbkS2CvXQMiuMU_kMZKFGO2rDV/view?usp=drive_link) / [config](configs/1shot-swin/graph_split5_config.py) |
|
| 121 |
-
|
| 122 |
-
### 5-Shot Models
|
| 123 |
-
| Setting | split 1 | split 2 | split 3 | split 4 | split 5 |
|
| 124 |
-
|:-------:|:---------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------:|
|
| 125 |
-
| Tiny | 94.18 | 91.46 | 90.50 | 90.18 | 89.47 |
|
| 126 |
-
| | [link](https://drive.google.com/file/d/1PeMuwv5YwiF3UCE5oN01Qchu5K3BaQ9L/view?usp=drive_link) / [config](configs/5shots/graph_split1_config.py) | [link](https://drive.google.com/file/d/1enIapPU1D8lZOET7q_qEjnhC1HFy3jWK/view?usp=drive_link) / [config](configs/5shots/graph_split2_config.py) | [link](https://drive.google.com/file/d/1MTeZ9Ba-ucLuqX0KBoLbBD5PaEct7VUp/view?usp=drive_link) / [config](configs/5shots/graph_split3_config.py) | [link](https://drive.google.com/file/d/1U2N7DI2F0v7NTnPCEEAgx-WKeBZNAFoa/view?usp=drive_link) / [config](configs/5shots/graph_split4_config.py) | [link](https://drive.google.com/file/d/1wapJDgtBWtmz61JNY7ktsFyvckRKiR2C/view?usp=drive_link) / [config](configs/5shots/graph_split5_config.py) |
|
| 127 |
-
| Small | 96.51 | 92.15 | 91.99 | 92.01 | 92.36 |
|
| 128 |
-
| | [link](https://drive.google.com/file/d/1p5rnA0MhmndSKEbyXMk49QXvNE03QV2p/view?usp=drive_link) / [config](configs/5shot-swin/graph_split1_config.py) | [link](https://drive.google.com/file/d/1Q3KNyUW_Gp3JytYxUPhkvXFiDYF6Hv8w/view?usp=drive_link) / [config](configs/5shot-swin/graph_split2_config.py) | [link](https://drive.google.com/file/d/1gWgTk720fSdAf_ze1FkfXTW0t7k-69dV/view?usp=drive_link) / [config](configs/5shot-swin/graph_split3_config.py) | [link](https://drive.google.com/file/d/1LuaRQ8a6AUPrkr7l5j2W6Fe_QbgASkwY/view?usp=drive_link) / [config](configs/5shot-swin/graph_split4_config.py) | [link](https://drive.google.com/file/d/1z--MAOPCwMG_GQXru9h2EStbnIvtHv1L/view?usp=drive_link) / [config](configs/5shot-swin/graph_split5_config.py) |
|
| 129 |
-
|
| 130 |
-
### Evaluation
|
| 131 |
-
The evaluation on a single GPU will take approximately 30 min.
|
| 132 |
-
|
| 133 |
-
To evaluate the pretrained model, run:
|
| 134 |
-
```
|
| 135 |
-
python test.py [path_to_config_file] [path_to_pretrained_ckpt]
|
| 136 |
-
```
|
| 137 |
-
## Acknowledgement
|
| 138 |
-
|
| 139 |
-
Our code is based on code from:
|
| 140 |
-
- [MMPose](https://github.com/open-mmlab/mmpose)
|
| 141 |
-
- [CapeFormer](https://github.com/flyinglynx/CapeFormer)
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
## License
|
| 145 |
-
This project is released under the Apache 2.0 license.
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: PoseAnything
|
| 3 |
+
emoji: 🏢
|
| 4 |
+
colorFrom: red
|
| 5 |
+
colorTo: red
|
| 6 |
+
sdk: gradio
|
| 7 |
+
sdk_version: 4.11.0
|
| 8 |
+
app_file: app.py
|
| 9 |
+
pinned: false
|
| 10 |
+
license: apache-2.0
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|