omnicrawl 0.1.0
A Python library for crawling LinkedIn posts and GitHub repos and scrap websites.
Requires Python: >=3.6



# OmniCrawlers

OmniCrawlers is a Python library designed to simplify the extraction of data from various online platforms, including LinkedIn, GitHub repositories, and general websites. With this library, you can easily crawl and collect posts from LinkedIn, repository data from GitHub, and content from any website.

## Features

- **LinkedIn Crawler**: Extracts posts from LinkedIn profiles.
- **GitHub Crawler**: Retrieves file data from GitHub repositories.
- **Website Scraper**: Scrapes and retrieves content from specified websites.

### Output Formats

- **LinkedIn**: Returns a list of posts in the format `["post1", "post2", "post3", ...]`.
- **GitHub**: Returns a dictionary of filenames and their respective data in the format `{"filename": "data", "filename2": "data", ...}`.
- **Website**: Returns simple scraped data in a string format `data......`.

## Installation

You can install the OmniCrawlers library via pip:

```bash
pip install omnicrawlers
```

## Usage

Hereâ€™s a quick example demonstrating how to use the library to extract data from GitHub, LinkedIn, and general websites.

```python
from omnicrawlers import GithubCrawler
from omnicrawlers import LinkedInCrawler
from omnicrawlers import WebsiteScraper

# Extract data from GitHub repository
def githubtest():
    github = GithubCrawler()  # Initialize the GitHub Crawler
    link = input("Enter your GitHub repo link: ")
    output = github.extract(link)  # Extract data using the main method
    print(output)

# Extract data from a website
def websitetest():
    website = WebsiteScraper()  # Initialize the Website Scraper
    link = input("Enter website link: ")
    output = website.extract(link)  # Extract data from the website
    print(output)

# Extract data from LinkedIn profile
def linkedintest():
    email = input("Enter your LinkedIn username: ")
    password = input("Enter your LinkedIn password: ")
    link = input("Enter your LinkedIn profile link: ")
    linkedin = LinkedInCrawler(email, password)  # Initialize the LinkedIn Crawler
    output = linkedin.extract(link)  # Extract data from the LinkedIn profile
    print(output)

if __name__ == "__main__":
    print("1. Test GitHub")
    githubtest()
    print("2. Test LinkedIn")
    linkedintest()
    print("3. Test Website")
    websitetest()
```

### Class and Method Descriptions

```python
# GitHubCrawler:
  Initializes a new instance of the GithubCrawler().
  extract(link): This method takes a GitHub repository link and retrieves file data from it.

# LinkedInCrawler:
  Initializes a new instance of the LinkedInCrawler(email, password).
  extract(link): This method takes a LinkedIn profile link and extracts posts from that profile.

# WebsiteScraper:
  Initializes a new instance of the WebsiteScraper().
  extract(link): This method takes a website link and scrapes content from that page.
```

### Example Outputs

- **LinkedIn Output**:
    ```python
    ["post1", "post2", "post3"]
    ```

- **GitHub Output**:
    ```python
    {"filename": "data", "filename2": "data"}
    ```

- **Website Output**:
    ```
    data......
    ```

## Contributing

Contributions are welcome! If you have suggestions for improvements or new features, feel free to create an issue or submit a pull request.

## License

This project is licensed under the MIT License. See the LICENSE file for details.

## Contact

For any inquiries, please reach out to [Vinayak Pratap Rana](perrcroshado@gmail.com).
```