semvec/pypi-packages huggingface dataset:
scrapy-feed-storage-internetarchive 0.0.1
Scrapy Item Feed Storage Backend for archive.org
Requires Python: >=3.6

# scrapy-feed-storage-internetarchive

This is a [Storage Backend](https://docs.scrapy.org/en/latest/topics/feed-exports.html#storages) for [Scrapy](https://scrapy.org/) [Item Feeds](https://docs.scrapy.org/en/latest/topics/feed-exports.html) that uploads feed files to [archive.org](https://archive.org) when a scrape job ends.

This was created to make it easy to archive data at the Internet Archive which you are authorised to distribute. e.g. to archive public data.


## Usage

### Install the custom storage backend

We recommend the scheme `internetarchive`

```
FEED_STORAGES = {
    "internetarchive": "feedstorage_internetarchive.storages.InternetArchiveStorage",
}
```


### Configure the Internet Archive metadata template

Metadata values can be specified using the settings key `FEED_STORAGE_INTERNETARCHIVE`, e.g.

```
FEED_STORAGE_INTERNETARCHIVE = {
    "metadata": {
        "mediatype": "data",
        "coverage": "South Africa",
        "title": "eTender Portal %(name)s %(time)s %(filetype)s",
    }
}
```


### Configure the storage for your feeds

Use the Feed Exporter configuration with the URI scheme you used for installing the backend.

The Internet Archive feed exporter should have the hostname `archive.org`, Internet Archive S3 API access key and secret in the username and password positions.

Only one level of path is allowed. This will be used as the filename, and will be transformed into a unique identifier, meaning it should be unique on all of the Internet Archive. Including the scrape job timestamp in this path is useful for ensuring uniqueness.

Extra parameters can be provided as query string parameters, which will then be templated into the metadata values.

e.g.

```
FEEDS = {
    "internetarchive://YourIAS3AccessKey:YourIAS3APISecretKey@archive.org/south-africa-%(name)s-%(time)s.csv?time=%(time)s&name=%(name)s&filetype=csv": {
        "format": "csv",
    },
    "internetarchive://YourIAS3AccessKey:YourIAS3APISecretKey@archive.org/south-africa-%(name)s-%(time)s.jsonlines?time=%(time)s&name=%(name)s&filetype=jsonlines": {
        "format": "jsonlines",
    },
}
```

You probably don't want to put credentials into your project settings module, since it can then easily be discovered if added to source control. So try to set it in the environment where you will run your spider.


### Scrapinghub

You can set the `FEEDS` key in scrapinghub by providing the value dictionary as JSON on a single line in your spider's Raw Settings. For the above example, you would add the following line in the scrapinghub Raw settings:

```
FEEDS = {"internetarchive://YourIAS3AccessKey:YourIAS3APISecretKey@archive.org/south-africa-%(name)s-%(time)s.csv?time=%(time)s&name=%(name)s": {"format": "csv"}, "internetarchive://YourIAS3AccessKey:YourIAS3APISecretKey@archive.org/south-africa-%(name)s-%(time)s.jsonlines?time=%(time)s&name=%(name)s": { "format": "jsonlines" }}
```

After saving, you should see it parsed into a key and value on the standard settings pane.
paipline-zkchong 0.0.1
Simple pipeline alternative to Scikit Learn's Pipeline.
Requires Python: >=3.0

# What is Paip? And Why Paip?
"Paip" is a Malay word of "pipe", a lightweight library that is alternative to Scikit Learn pipeline. 

In general, Scikit Learn's pipeline facilitates the chaining of transformers in series and execute them in sequential order. Then, the pipeline can be serialized and re-used easily at different environment. However, the existing Scikit Learn's pipeline only allows objects to be loaded into the step list. If there is a small piece of code that is needed in between transformers (e.g. change of column name of data frame), we need to revise the small piece of code into object-oriented paradigm in order to obtain the corresponding objects. Need less to say, the class definition has to be imported at the new environment during the unpicke process. 

To overcome this issue, we have developed a lightweight library named Paip that allows both runtime code and transformers to be executed in the pipelines.

For example, we have a data frame, `X` that needs to go through two feature transformations, i.e. `transformer1` and `transformer2`. In between the transformers, we need to rename the column, "x1" to "x1_new". The code is presented in the following.

```python
from paipline import Paip

# Somewhere here, we have created the data frame, `X`.
# Somewhere here, we have created feature transformation, named `transformer1`.
# Somewhere here, we have created feature transformation, named `transformer2`.

step_list = []

# Add the transformer 1 into the step list.
step_list.append(
    {'name': 'Transformer 1', # name
     'obj_dict' : {'transformer1': transformer1}, # The objects to be serialized in pickle.
      'run_dict':{
                'transform': ['X = transformer1.transform(X)'], # Run this code.
            },
     'output_list' : ['X', ] # Whitelist these variables will go to next step.
    }
)

# Add the transformer 2 into the step list.
step_list.append(
    {'name': 'Transformer 1',
     'obj_dict' : {'transformer1': transformer1},
      'run_dict':{
                'transform': [
                    'X = X.rename(columns={"x1": "x1_new")',  # Rename column.
                    'X = transformer1.transform(X)'
                ],
            },
     'output_list' : ['X', ]
    }
)


paip1 = Paip(step_list)

paip1.run('transform', # run those command that tag with "transform"
          {'X': X,},    # the variable to bring into the pipeline.
          debug = True)   # Debug output 

# Retrieve the output
X_out = paip1.output_dict['X']

```

And to serialize Paip, we use the existing joblib.
```
import joblib
joblib('./test.paip', paip1)

```
notirss 0.1.0
Simple CLI RSS real-time notifier via webhook.
Requires Python: >=3.6

# notiRSS

Simple CLI RSS real-time notifier via webhook.

## Installation
From PyPi:

`pip install notirss`

From Github:

`pip install git+https://github.com/scratchmex/notirss#egg=notirss`

## Usage

```console
❯ notirss -h
usage: notirss [-h] -f FEED -c ENTRIES [-x EXTRACT] -w WEBHOOK [-t TIME] [-v]

Simple CLI RSS real-time notifier via webhook.

optional arguments:
  -h, --help            show this help message and exit
  -f FEED, --feed FEED  the RSS feed URL to check
  -c ENTRIES, --entries ENTRIES
                        amount of entries returned by the feed
  -x EXTRACT, --extract EXTRACT
                        data mapping in JSON format to extract from RSS feed, default all data
  -w WEBHOOK, --webhook WEBHOOK
                        webhook url to GET when new item/s found
  -t TIME, --time TIME  periodic time interval to check in seconds
  -v, --verbose         set logging level to DEBUG
```