semvec/pypi-packages huggingface dataset:
youcreep 0.1.42
A powerful, asynchronous web crawler designed specifically for YouTube.

# Youcreep

A powerful, asynchronous web crawler designed specifically for YouTube. Built to efficiently interact with web pages in a headless mode, this tool facilitates the seamless extraction of YouTube video comments and video information directly from the platform.

## Features

- **Asynchronous Crawling**: Uses Python's asynchronous features for fast and efficient data extraction.
- **Headless Interaction**: Operates seamlessly without opening a web browser.
- **Modular Design**: Built with modularity in mind, allowing for easy extension and maintenance.

## Modules

### 1. YoutubeCommentCrawler

Extract comments from any given YouTube video.

#### Usage:

```python
from youcreep import YoutubeCommentCrawler

with YoutubeCommentCrawler() as crawler:
   comments = await crawler.crawl('YOUR_YOUTUBE_VIDEO_URL', n_target=NUMBER_OF_COMMENTS_TO_FETCH)
```

### 2. YoutubeVideoInfoCrawler

Search for videos based on a keyword and extract their information.

#### Usage:

```python
from youcreep import YoutubeVideoInfoCrawler

with YoutubeVideoInfoCrawler() as crawler:
   videos_info = await crawler.crawl('YOUR_SEARCH_TERM', n_target=NUMBER_OF_VIDEOS_TO_FETCH,
                                     filter_options=YOUR_FILTER_OPTIONS)
```

## Getting Started

1. **Clone the repository**

   ```bash
   git clone https://github.com/stevieflyer/youtube_crawler.git
   ```
2. **Navigate to the project directory**

   ```bash
   cd youcreep
   ```
3. **Install the dependencies**

   ```bash
   pip install -r requirements.txt
   ```
4. **Start crawling**

   Use the examples provided in the modules section above.

## Contributing

Contributions are welcome! Please raise an issue or submit a pull request.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

Happy crawling! :)
dynamicscrapper 10.0.7
A package to scrap the web dynamically
Requires Python: >=3.6

# Dynamic Scrapper **STILL IN DEVELOPMENT**

## The Goal Of This Package:

The goal of this package is being able to **scrap any site given** and **gather any data** no matter they type/style of said site.

This project was a huge part of understanding webscrapping in general.

## How to use:

The package is equipped with a **CLICK CLI** to give the input needed to extract the data needed.

### **Steps:**
1. First step is giving the link of the site you want scrapped. (it can be either a page that contains a bunch of links to click and extract data from inside every link or one single page that you want data extracted from).
2. If the link you gave is of a page containing a bunch of links to get data from, write the class name of the <div> containing the "a" tag that contains these links (in most cases these links always have the same class name)**[LEAVE EMPTY IF IT'S A SINGLE PAGE EXTRACTION]**.
3. Choose the type of pagination used in said page to be able to extract data from all pages:
  - number: for sites with numbered pagination
  - see_more: for sites with a see more button that expands the rest of the information
  - none: incase of a single page or a infinite scrollp age 
4. Write the class name of the pagination class used according to:
  - number: write the class name of the button "next" or ">" that takes you to the next page
  - see_more: write the class name of the see_more button
  - none: leave empty
5. Choosing the items to scrap
This step is gonna look like this: 
  - Choose a name for the item 
  - Choose the type you want to use to extract said item:
            - tag-name: using html tag names
            - class-name: using class-names
            - id: using id
            - name: using name attribute
            - link-text: by giving the text inside a "a" tag
  - Choose the name of the type you specified before:
            - tag-name : the tag name you want to extract from (h1, h2, a ...)
            - class-name: the class name needed
            - id: the id needed
            - name: the name inside the attribute
            - link-text: the text inside the "a" tag 
  - **IF** you are trying to get the value inside an attribute of a tag, write the tag name   you want to get (for example you need img url inside an "img" tag, you write "src" to get the text inside the src)
  - Choose Y or N to continue adding items to scrap or stop.

## Tips for number 5:
  - use tag-name when there is only one tag of that kind (h1..)
  - class-name is the most used but make sure to take a unique name so you dont get other items by mistake.
  - be carefull when using id as an id can change from page to page of the same element.