semvec/pypi-packages huggingface dataset:
pushurls 0.0.3
a tool to push the spider's fail urls in mongodb to redis

## 重推失败的 url 回到 redis 的小工具

----

### 必须：

#### 1. 爬虫多次重试失败的 url 放入 mongodb

#### 2. mongodb 内失败 url 的 key 为 "url"，即 {"url": "www.xxx.com"}

#### 3. 爬虫设置可以使用 redis 的 start_urls

---

### 安装：

```shell script
# 看具体版本，包在 dist 文件夹下
$ pip install pushurls.tar.gz
```

### 使用：

```shell script
# 直接开始：
$ pushurls

# 指定配置文件：
$ pushurls /root/push_fail_urls_set.json
```

### 配置文件格式：(建议直接运行，让程序自动生成配置文件，下次就不必再输入配置)

```json
{"from": [
  {
    "adder_sep": ">>>",
    "condition": {},
    "db": "test_db",
    "from_collection": "test_data",
    "fromdb_str": "127.0.0.1.amazon.test_data",
    "host": "127.0.0.1",
    "password": "123456",
    "port": 27017,
    "source": "admin",
    "url_head": "",
    "url_tail": "**-fixed-**test_url",
    "user": "root"
  }],
  "to": [
  {
    "db": "0",
    "host": "127.0.0.1",
    "port": 6379,
    "spiders": {
      "spider_name1": "S1:start_urls",
      "spider_name2": "S2:start_urls"
    },
    "todb_str": "127.0.0.1:6379.0"
  }]}
```
deployee 0.14.0
Python tool to deploy python web application automatically
Keywords: django,automation,deploy

# deployee

[![Build Status](https://travis-ci.org/WqyJh/deployee.svg?branch=master)](https://travis-ci.org/WqyJh/deployee)
[![license](https://img.shields.io/badge/LICENCE-GPLv3-brightgreen.svg)](https://raw.githubusercontent.com/WqyJh/deployee/master/LICENSE)

Server requirements:

- rsync
- supervisor
- python3
- virtualenv
- pip


## Usage

Deploy an django application.

```bash
deployee /path/to/project production
```

Deploy a flask application.

```bash
./run.py /path/to/project production --framework flask --script "python -m app"
```

## Project Structure

```bash
/var
├── lib
│   ├── local_settings
│   │   └── project
│   │       ├── production
│   │       ├── staging
│   │       └── test
│   └── venv
│   │   └── project
│   │       ├── production
|   |       |   └── <hash>
│   │       ├── staging
|   |       |   └── <hash>
│   │       └── test
├── log
│   └── project
│       ├── production
│       ├── staging
│       └── test
├── media
│   └── project
│       ├── production
│       ├── staging
│       └── test
├── static
│   └── project
│       ├── production
│       ├── staging
│       └── test
└── www
│   └── project
│       ├── production
|       |   └── <hash>
│       ├── staging
|       |   └── <hash>
│       └── test
```
# Changelog


## 0.14.0
#### New Features

* create working directory under /tmp
* add parallel option for django test


## 0.12.0
#### New Features

* add inventory argument
* add pkg_resources
* add support for custom script
* add --dry-run
* support stages
* add click to create cli
* generate vars.yml
* run playbook with python
* add <hash> to the directory tree
* add ansible
* add attr evaluation to format
* add string formatter

#### Fixes

* fix log path