add: xfeat(dense)

README.md

@@ -16,7 +16,7 @@ license: mit
 [![Issues][issues-shield]][issues-url]
 
 <p align="center">
-  <h1 align="center"><br><ins>Image Matching WebUI</ins><br>find matches between 2 images</h1> 
+  <h1 align="center"><br><ins>Image Matching WebUI</ins><br>Identify matching points between two images</h1> 
 </p>
 
 ## Description
@@ -24,13 +24,22 @@ license: mit
 This simple tool efficiently matches image pairs using multiple famous image matching algorithms. The tool features a Graphical User Interface (GUI) designed using [gradio](https://gradio.app/). You can effortlessly select two images and a matching algorithm and obtain a precise matching result.
 **Note**: the images source can be either local images or webcam images.
 
+Try it on <a href='https://huggingface.co/spaces/Realcat/image-matching-webui'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue'></a> 
+<a target="_blank" href="https://lightning.ai/realcat/studios/image-matching-webui">
+  <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio"/>
+</a>
+
 Here is a demo of the tool:
 
 ![demo](assets/demo.gif)
 
 The tool currently supports various popular image matching algorithms, namely:
+- [x] [XFeat](https://github.com/verlab/accelerated_features), CVPR 2024
+- [x] [RoMa](https://github.com/Vincentqyw/RoMa), CVPR 2024
+- [x] [DeDoDe](https://github.com/Parskatt/DeDoDe), 3DV 2024
+- [ ] [Mickey](https://github.com/nianticlabs/mickey), CVPR 2024
+- [ ] [GIM](https://github.com/xuelunshen/gim), ICLR 2024
 - [x] [LightGlue](https://github.com/cvg/LightGlue), ICCV 2023
-- [x] [DeDoDe](https://github.com/Parskatt/DeDoDe), ArXiv 2023
 - [x] [DarkFeat](https://github.com/THU-LYJ-Lab/DarkFeat), AAAI 2023
 - [ ] [ASTR](https://github.com/ASTR2023/ASTR), CVPR 2023
 - [ ] [SEM](https://github.com/SEM2023/SEM), CVPR 2023
@@ -40,7 +49,6 @@ The tool currently supports various popular image matching algorithms, namely:
 - [x] [SOLD2](https://github.com/cvg/SOLD2), CVPR 2021
 - [ ] [LineTR](https://github.com/yosungho/LineTR), RA-L 2021
 - [x] [DKM](https://github.com/Parskatt/DKM), CVPR 2023
-- [x] [RoMa](https://github.com/Vincentqyw/RoMa), Arxiv 2023
 - [ ] [NCMNet](https://github.com/xinliu29/NCMNet), CVPR 2023
 - [x] [TopicFM](https://github.com/Vincentqyw/TopicFM), AAAI 2023
 - [x] [AspanFormer](https://github.com/Vincentqyw/ml-aspanformer), ECCV 2022
@@ -48,6 +56,7 @@ The tool currently supports various popular image matching algorithms, namely:
 - [ ] [LISRD](https://github.com/rpautrat/LISRD), ECCV 2022
 - [ ] [REKD](https://github.com/bluedream1121/REKD), CVPR 2022
 - [x] [ALIKE](https://github.com/Shiaoming/ALIKE), ArXiv 2022
+- [x] [RoRD](https://github.com/UditSinghParihar/RoRD), IROS 2021
 - [x] [SGMNet](https://github.com/vdvchen/SGMNet), ICCV 2021
 - [x] [SuperPoint](https://github.com/magicleap/SuperPointPretrainedNetwork), CVPRW 2018
 - [x] [SuperGlue](https://github.com/magicleap/SuperGluePretrainedNetwork), CVPR 2020
@@ -59,11 +68,15 @@ The tool currently supports various popular image matching algorithms, namely:
 - [ ] [SOSNet](https://github.com/scape-research/SOSNet), CVPR 2019
 - [x] [SIFT](https://docs.opencv.org/4.x/da/df5/tutorial_py_sift_intro.html), IJCV 2004
 
+
 ## How to use
 
-### HuggingFace 
+### HuggingFace / Lightning AI
 
-Just try it on HF <a href='https://huggingface.co/spaces/Realcat/image-matching-webui'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue'>  [![Open in OpenXLab](https://cdn-static.openxlab.org.cn/app-center/openxlab_app.svg)](https://openxlab.org.cn/apps/detail/Realcat/image-matching-webui)
+Just try it on <a href='https://huggingface.co/spaces/Realcat/image-matching-webui'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue'></a> 
+<a target="_blank" href="https://lightning.ai/realcat/studios/image-matching-webui">
+  <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio"/>
+</a>
 
 or deploy it locally following the instructions below.
 
@@ -74,8 +87,15 @@ cd image-matching-webui
 conda env create -f environment.yaml
 conda activate imw
 ```
+
+or using [docker](https://hub.docker.com/r/vincentqin/image-matching-webui):
+
+``` bash
+docker pull vincentqin/image-matching-webui:latest
+docker run -it -p 7860:7860 vincentqin/image-matching-webui:latest python app.py --server_name "0.0.0.0" --server_port=7860
+```
  
-### run demo
+### Run demo
 ``` bash
 python3 ./app.py
 ```
@@ -85,7 +105,7 @@ then open http://localhost:7860 in your browser.
 
 ### Add your own feature / matcher
 
-I provide an example to add local feature in [hloc/extractors/example.py](hloc/extractors/example.py). Then add feature settings in `confs` in file [hloc/extract_features.py](hloc/extract_features.py). Last step is adding some settings to `model_zoo` in file [extra_utils/utils.py](extra_utils/utils.py).
+I provide an example to add local feature in [hloc/extractors/example.py](hloc/extractors/example.py). Then add feature settings in `confs` in file [hloc/extract_features.py](hloc/extract_features.py). Last step is adding some settings to `model_zoo` in file [common/config.yaml](common/config.yaml).
 
 ## Contributions welcome!
 

common/config.yaml

@@ -28,10 +28,13 @@ matcher_zoo:
   aspanformer:
     matcher: aspanformer
     dense: true
-  xfeat:
+  xfeat(sparse):
     matcher: NN-mutual
     feature: xfeat
     dense: false
+  xfeat(dense):
+    matcher: xfeat_dense
+    dense: true
   dedode:
     matcher: Dual-Softmax
     feature: dedode

hloc/extract_features.py

@@ -211,7 +211,7 @@ confs = {
             "grayscale": False,
             "resize_max": 1600,
         },
-    },    
+    },
     "alike": {
         "output": "feats-alike-n5000-r1600",
         "model": {

hloc/extractors/xfeat.py

@@ -18,7 +18,7 @@ class XFeat(BaseModel):
             pretrained=True,
             top_k=self.conf["max_keypoints"],
         )
-        logger.info(f"Load XFeat model done.")
+        logger.info(f"Load XFeat(sparse) model done.")
 
     def _forward(self, data):
         pred = self.net.detectAndCompute(

hloc/match_dense.py

@@ -72,7 +72,7 @@ confs = {
             "height": 480,
         },
     },
-    # Use topicfm for matching feats
+    # Use aspanformer for matching feats
     "aspanformer": {
         "output": "matches-aspanformer",
         "model": {
@@ -90,6 +90,21 @@ confs = {
             "dfactor": 8,
         },
     },
+    "xfeat_dense": {
+        "output": "matches-xfeat_dense",
+        "model": {
+            "name": "xfeat_dense",
+            "max_keypoints": 8000,
+        },
+        "preprocessing": {
+            "grayscale": False,
+            "force_resize": False,
+            "resize_max": 1024,
+            "width": 640,
+            "height": 480,
+            "dfactor": 8,
+        },
+    },
     "dkm": {
         "output": "matches-dkm",
         "model": {

hloc/matchers/xfeat_dense.py

@@ -0,0 +1,57 @@
+import torch
+from pathlib import Path
+from hloc import logger
+from ..utils.base_model import BaseModel
+
+
+class XFeatDense(BaseModel):
+    default_conf = {
+        "keypoint_threshold": 0.005,
+        "max_keypoints": 8000,
+    }
+    required_inputs = [
+        "image0",
+        "image1",
+    ]
+
+    def _init(self, conf):
+        self.net = torch.hub.load(
+            "verlab/accelerated_features",
+            "XFeat",
+            pretrained=True,
+            top_k=self.conf["max_keypoints"],
+        )
+        logger.info(f"Load XFeat(dense) model done.")
+
+    def _forward(self, data):
+        # Compute coarse feats
+        out0 = self.net.detectAndComputeDense(
+            data["image0"], top_k=self.conf["max_keypoints"]
+        )
+        out1 = self.net.detectAndComputeDense(
+            data["image1"], top_k=self.conf["max_keypoints"]
+        )
+
+        # Match batches of pairs
+        idxs_list = self.net.batch_match(
+            out0["descriptors"], out1["descriptors"]
+        )
+        B = len(data["image0"])
+
+        # Refine coarse matches
+        # this part is harder to batch, currently iterate
+        matches = []
+        for b in range(B):
+            matches.append(
+                self.net.refine_matches(
+                    out0, out1, matches=idxs_list, batch_idx=b
+                )
+            )
+        # we use results from one batch
+        matches = matches[0]
+        pred = {
+            "keypoints0": matches[:, :2],
+            "keypoints1": matches[:, 2:],
+            "mconf": torch.ones_like(matches[:, 0]),
+        }
+        return pred