semvec/pypi-packages huggingface dataset:
teyespider-sdk 0.0.3
Python SDK
License: BSD-3-Clause

# TeyeSpider SDK for Python

The SDK for Python contains two parts:
1. CLI Tool
2. Utility Tools

## Installation Prerequisite

You should install the requirement below before installing the teyespider-sdk.
```text
Click==7.0
requests==2.22.0
prettytable==0.7.2
scrapy==2.2.0
pymongo==3.10.1
pymysql==0.9.3
psycopg2-binary==2.8.5
kafka-python==2.0.1
elasticsearch==7.8.0
pathspec==0.8.0
```


## CLI Tool

The CLI Tool is mainly designed for those who are more comfortable using command line tools to interact with teye-spider. 

The installation of the CLI Tool is simple:

```bash
pip install teyespider-sdk
```

Then, you can use the `teye-spider` command in the command prompt to action with teye-spider.

Check the help document below, or you can refer to [the official documentation (Chinese)](https://docs.teye-spider.cn/SDK/CLI.html). 

```
teye-spider --help
```


## Usage
```shell
# Hello to teye-spider
teye-spider hello
# Login on first time
teye-spider login -u admin -a  http://xxx-master.xxxxx.10.92.xx.xx.nip.io:xxx/api
# Upload spider to platform
teye-spider upload -N demo -d ~/Downloads/tutorial/ -n demo -c res_demo
# explore other command
teye-spider --help
```





## Utility Tools

Utility tools mainly provide some `helper` methods to make it easier for you to integrate your spiders into teye-spider, e.g. saving results.

Below are integration methods of Scrapy and general Python spiders with teye-spider.

⚠️Note: make sure you have already installed `teye-spider-sdk` using pip.

##### Scrapy Integration

In `settings.py` in your Scrapy project, find the variable named `ITEM_PIPELINES` (a `dict` variable). Add content below.

```python
ITEM_PIPELINES = {
    'teyespider.pipelines.teye-spiderMongoPipeline': 888,
}
```

Then, start the Scrapy spider. After it's done, you should be able to see scraped results in **Task Detail -> Result**

##### General Python Spider Integration

Please add below content to your spider files to save results.

```python
# import result saving method
from teyespider import save_item

# forexample,it your result is:
result = {'name': 'teyespider'}

# call result saving method
save_item(result)
```

Then, start the spider. After it's done, you should be able to see scraped results in **Task Detail -> Result**