semvec/pypi-packages huggingface dataset:
web2db 0.1.5
Fetch webpage full-text, persist link and full text to SQLITE3 db, resumable with tqdm progressbar.
License: MIT

# web2db


Fetches the full text of input URLs and persists them to sqlite3 DB file.  
Fetching is resumable and comes with a progressbar.  


### Install:  
```pip install web2db```


### Quickstart:  

```python
import web2db  
web2db.dump('data.db', urls=[
    'https://www.google.com',
    'https://www.yahoo.com',
    'https://www.msn.com'
])
```

Query the DB file:
```python
df = web2db.to_df(sqlite3_file_path)
print(df.shape)
print(df)
```

### SQL Schema:  
- Table:  
	- > WebPages
	
	  | url  | fulltext | status_code |
	  | ---- | -------- | ----------- |
	  | text | text     | int         |


### Features:
- Resumable webpage fetching
- Saves to local SQLITE3 DB
- tqdm progress bar