File size: 4,158 Bytes
8c6b5ee 6844bda 8c6b5ee 6844bda 8c6b5ee 4c25b54 8c6b5ee 6844bda 8c6b5ee 6844bda 8c6b5ee |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
# 📑 Evaluating and Reproducing SuPr Results
We provide ready-to-use bash scripts under the [scripts/](https://huggingface.co/tongyujun/Subspace_Prompting/tree/main/scripts) directory for evaluating **SuPr**, **SuPr+PromptSRC**, and **SuPrEns** models using our pre-trained checkpoints.
Please ensure that you update the `DATA` variable in each script to match your dataset path, and run all commands from the project root directory `SuPr/`.
We have already provided:
- Precomputed evaluation results under [output/](https://huggingface.co/tongyujun/Subspace_Prompting/tree/main/output)
- Aggregated and summarized statistics under [parse_results/](https://huggingface.co/tongyujun/Subspace_Prompting/tree/main/parse_results)
Below, we guide you through reproducing these results by yourself.
---
## 🔥 SuPr Reproduction Guide
We now explain how to reproduce our reported results step-by-step.
---
### 🛠️ Preliminary Setup
To reproduce the results (taking ImageNet as an example), follow these steps:
1. **Create the environment and install dependencies**
- Follow the instructions in [INSTALL.md](https://huggingface.co/tongyujun/Subspace_Prompting/blob/main/docs/INSTALL.md) to set up the environment and install the `Dassl.pytorch` library.
2. **Prepare datasets**
- Follow the dataset preparation guidelines provided in [DATASETS.md](https://huggingface.co/tongyujun/Subspace_Prompting/blob/main/docs/DATASETS.md).
3. **Download pre-trained weights**
- Download the zipped folder containing all pre-trained weights from this [link](https://huggingface.co/tongyujun/Subspace_Prompting/tree/main/weights).
- After extraction, the folder structure should look like:
```text
weights/
|–– SuPr/
| |–– base2new/
| | |–– imagenet/
| | |–– shots_16/
| | |–– seed1/
| | |–– seed2/
| | |–– seed3/
| ...
| |–– cross_dg/
| |–– fewshot/
|
|–– SubspacePromptSRC/
| |–– base2new/
| ...
|
|–– SuPrEns/
| |–– base2new/
| ...
```
> **Important:**
> If you place the `weights/` folder outside the `Subspace_Prompting/` root directory,
> remember to update the `${WEIGHTSPATH}` variable inside the following scripts:
> - `scripts/supr/reproduce_base2novel_setting.sh`
> - `scripts/supr/reproduce_fewshot.sh`
> - `scripts/supr/reproduce_xd.sh`
> - `scripts/supr_src/reproduce_base2novel_setting.sh`
> - `scripts/supr_ens/reproduce_base2novel_setting.sh`
---
### ⚡ Reproducing Experiments
After setting up, run the following command from the `SuPr/` root directory:
```bash
bash reproduce.sh
```
This command will automatically start evaluation across all settings, using the provided pre-trained models.
The evaluation logs and results will be saved under the `output/` directory.
---
### 📈 Aggregating Results
After running evaluations, you can aggregate the results across seeds and tasks by running:
```bash
# Base-to-Novel Evaluation Results
# SuPr
python parse_test_res.py -type base2new output/base2new/SuPr/reproduce_vit_b16_ep10_batch4_4+4ctx/test_new/caltech101/shots_16
# SuPr+PromptSRC
python parse_test_res.py -type base2new output/base2new/SubspacePromptSRC/reproduce_vit_b16_ep20_batch4_4+4ctx_promptsrc/test_new/imagenet/shots_16
# SuPr Ensemble
python parse_test_res.py -type base2new output/base2new/SuPrEns/reproduce_vit_b16_ep10_batch4_4+4ctx/test_new/imagenet/shots_16
# Cross-Dataset Generalization Results
python parse_test_res.py -type cross output/cross_dg/SuPr/reproduce_vit_b16_ep12_batch8_4+4ctx_cross_datasets/caltech101/shots_16
# Domain Generalization Results
python parse_test_res.py -type dg output/cross_dg/SuPr/reproduce_vit_b16_ep12_batch8_4+4ctx_cross_datasets/imagenet/shots_16
```
The aggregated results will be automatically compiled into Excel spreadsheets for easy reporting.
---
> **Tip:** If you want to evaluate on other datasets beyond ImageNet, simply adjust the dataset names and paths accordingly in the scripts.
> **Warning:** Ensure that datasets are correctly prepared and accessible by the scripts, otherwise evaluation may fail.
---
|