File size: 817 Bytes
82fe524 5f2135e 82fe524 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# Twin Peaks crawler
This crawler download texts and metadata from [Twin Peaks Fandom Wiki](https://twinpeaks.fandom.com/wiki/Twin_Peaks_Wiki). The output format is JSON. The crawler is based on the combination of [Scrapy](https://github.com/scrapy/scrapy) and [fandom-py](https://github.com/NikolajDanger/fandom-py).
*Several wiki pages are discarded, since they are not related to Twin Peaks plot and create noise in the Question Answering index.*
## Installation
- copy this folder (if needed, see [stackoverflow](https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repo))
- `pip install -r requirements.txt`
## Usage
- (if needed, activate the virtual environment)
- `cd tpcrawler`
- `scrapy crawl tpcrawler`
- you can find the downloaded pages in `data` subfolder
|