Spaces:
Runtime error
Runtime error
| # V-Doc : Visual questions answers with Documents | |
| This repository contains code for the paper [V-Doc : Visual questions answers with Documents](https://arxiv.org/pdf/2205.13724.pdf). The demo videos can be accessed by this [link](https://drive.google.com/file/d/1Ztp9LBcrEcJA3NlbFWn1RfNyfwt8Y6Qk/view). | |
| <h4 align="center"> | |
| <b>Ding, Y.*, Huang, Z.*, Wang, R., Zhang, Y., Chen, X., Ma, Y., Chung, H., & Han, C. (CVPR 2022) <br/><a href="https://arxiv.org/pdf/2205.13724.pdf">V-Doc : Visual questions answers with Documents</a><br/></b></span> | |
| </h4> | |
| <p align="center"> | |
| <img src="https://github.com/usydnlp/vdoc/blob/main/images/system_architecture.png"> | |
| </p> | |
| ### Dataset in Dataset Storage Module | |
| The dataset we used to trained the model is provided in following links: | |
| [PubVQA Dataset](https://drive.google.com/drive/folders/1YMuctGPJbsy45Iz23ygcN1VGHWQp3aaU?ths=true) for training Mac-Network. | |
| Dataset for training LayoutLMv2([FUNSD-QA](https://drive.google.com/file/d/1Ev_sLTx3U9nAr2TGgUT5BXB1rpfLMlcq/view?usp=sharing)). | |
| ### Dataset Generation | |
| To run the scene based question generation code, we need to fetch the JSON files from the source. | |
| #### Extract OCR information | |
| ```bash | |
| python3 ./document_collection.py | |
| ``` | |
| After the step above, a new folder called <code>./input_ocr</code> will be generated. | |
| #### Generate questions | |
| ```bash | |
| python3 ./scene_based/pdf_generate_question.py | |
| ``` | |
| To limit the number of generated questions, you can change the code in <code>pdf_generate_question.py</code> line 575 and line 591-596 | |
| After the steps above, you can see a json file under the <code>./output_qa_dataset</code>. | |