semvec/pypi-packages huggingface dataset:
Zhihu-Spider 1.2.5
Scrapy the Zhihu content and user social network information. Now it contains 314400 questions and 261376 users..

Zhihu_Spider
============

Scrapy the Zhihu content and user social network information. Now it contains 314400 questions and 261376 users.

### File Strcture

* ./zhihu/zhihu : The related files about crawling the zhihu.com
* ./zhihu/zhihu_dat/ : The structured data for baseline experiments on zhihu dataset
    * ./zhihu/zhihu_dat/item.dat: the corpus(bag of words) of all questions, using Blei’s LDA-C format. The line number represents qid
    * ./zhihu/zhihu_dat/users.dat: the corpus of all users, the features of users is the bag representations of all the questions they have answered.
    * ./zhihu/zhihu_dat/vocab.dat: the vocabulary of zhihu dataset
    * ./zhihu/zhihu_dat/item_adj.dat: the questions and their answerer ids, the first column is the number of answers, the line number is question id
    * ./zhihu/zhihu_dat/user_adj.dat: the users and their answered question ids, the line number the user id, 
    * ./zhihu/zhihu_dat/truth.dat: the questions and their answers, each answer has a score with them
zhihu-crawler-pro 2.2.5
This project is dedicated to sharing and teaching the fundamentals and techniques of web reverse engineering. Web reverse engineering involves analyzing how websites and web applications work, and cracking or modifying existing code to achieve specific objectives. This project includes a series of tutorials, practical tools, and case studies aimed at helping developers, security researchers, and enthusiasts understand how to effectively reverse engineer web technologies.

**此项目未完成**

# zhihu_crawler
自动登录,保存token,爬取知乎数据

# 使用方法

```python
from zhihu.client import Client

# 所有程序的入口
client = Client()
# 直接使用用户名和密码登录
client.login(username='+8615555555555', password='password')
# 不使用参数,根据命令行输入
# client.login()

# 登录后获取个人信息
myself = client.myself()

# 打印个人信息,json 格式
print(myself.info())
```
XieChengLogin 2.2.5
模拟携程的登录.

# XieChengLogin
模拟携程的登录
本项目只是单纯的模拟了携程的自动登录，并没有对数据的爬取，其实只要登陆成功的话，抓数据就很简单了。
对于携程模拟登陆来说，最最重点了，其实都让验证码平台做了，如果自己写模拟点击的话，不是特别容易，而且稳定性很低。
本项目模拟登录可作为参考，比如元素截图，轨迹模拟等等
因为只是练手，代码条理有点乱，但都有注释，见谅！
weibo-scrapy 2.2.5
WEIBO\_SCRAPY是一个PYTHON实现的，使用多线程抓取WEIBO信息的框架。WEIBO\_SCRAPY框架给用户提供WEIBO的模拟登录和多线程抓取微博信息的接口，让用户只需关心抓取的业务逻辑，而不用处理棘手的WEIBO模拟登录和多线程编程.

WEIBO_SCRAPY
============

WEIBO\_SCRAPY是一个PYTHON实现的，使用多线程抓取WEIBO信息的框架。WEIBO\_SCRAPY框架给用户提供WEIBO的模拟登录和多线程抓取微博信息的接口，让用户只需关心抓取的业务逻辑，而不用处理棘手的WEIBO模拟登录和多线程编程。

WEIBO\_SCRAPY is a **Multi-Threading** SINA WEIBO data extraction Framework in Python. WEIBO\_SCRAPY provides WEIBO login simulator and interface for WEIBO data extraction with multi-threading, it saves users a lot of time by getting users out of writing WEIBO login simulator from scratch and multi-threading programming, users now can focus on their own **extraction** logic.


=======

###WEIBO\_SCRAPY的功能
1\. 微博模拟登录

2\. 多线程抓取框架

3\. **抓取任务**接口

4\. 抓取参数配置

###WEIBO\_SCRAPY Provides
1\. WEIBO Login Simulator

2\. Multi-Threading Extraction Framework

3\. **Extraction Task** Interface

4\. Easy Way of Parameters Configuration

###How to Use WEIBO\_SCRAPY
	#!/usr/bin/env python
	#coding=utf8

	from weibo_scrapy import scrapy

	class my_scrapy(scrapy):
		
		def scrapy_do_task(self, uid=None):
		     '''
		    User needs to overwrite this method to perform uid-based scrapy task.
		    @param uid: weibo uid
		    @return: a list of uids gained from this task, optional
		    '''
		     super(my_scrapy, self).__init__(**kwds)
		     
		     #do what you want with uid here, note that this scrapy is uid based, so make sure there are uids in task queue, 
		     #or gain new uids from this function
		     print 'WOW...'
		     return 'replace this string with uid list which gained from this task'
		 
	if __name__ == '__main__':
		
		s = my_scrapy(uids_file = 'uids_all.txt', config = 'my.ini')
		s.scrapy()

###相关阅读(Readings)
[基于UID的WEIBO信息抓取框架WEIBO_SCRAPY](http://yoyzhou.github.io/blog/2013/04/08/weibo-scrapy-framework-with-multi-threading/)
wechat-admin 1.2.5
本地测试直接执行`python app.py`就够了，如果想用其他的web server，可自行考虑部署方案.

## wechat-admin

### Background
***
写了一堆背景，后来都删掉了，简单说就是迁移公众号后台到自己的服务器，需求你懂的。


### Usage
***


1. 创建虚拟环境

    * 如果你是用`virtualenvwrapper`管理的运行环境

    ```
    mkvirtualenv wechat
    pip install -r requirements.txt
    ```
    
    * 如果你希望把依赖包安装到global环境(三思啊)

    ```
    sudo pip install -r requirements.txt 
    ```
    
2. 配置

    可以将你的`app_id`及`secret_key`配置到`settings.py`中，为了方便区分开发环境和生产环境，建议把配置项写到`local_settings.py`中，它会覆盖`settings.py`的配置项

    ```
    cp settings.py local_settings.py
    ```

3. 创建数据库

    数据库的配置也是在settings.py或local_settings.py里的，配好以后

    ```
    make initdb
    ```
    这个初始化脚本写的比较简单，可以根据自家的习惯修改

4. 部署

    本地测试直接执行`python app.py`就够了，如果想用其他的web server，可自行考虑部署方案



### TODO

1. UI
2. see those `TODO` comments