semvec/pypi-packages huggingface dataset:
pyscraper-framework 0.0.42
A python based scraper for any data source.
License: MIT
Keywords: etl scraping

# pyscrape

### Setup Local Dev
Install python, virtualenv and deps. To get started go [here](https://realpython.com/blog/python/flask-by-example-part-1-project-setup).

Once

Then: `pip install -r requirements.txt`

### Pushing to Production/Staging on Heroku (Don't do this, ci should do this)
This is an example of how a sample app would deploy. This shouldn't be here.
git remote add heroku-staging git@heroku.com:pyscrape-staging.git
git remote add heroku-production git@heroku.com:pyscrape-production.git
Or
`make deploy`

### Release
First, create a new pip package. This will bump the patch version and write it to `VERSION`.
`make package`

Then, to push to the package to the repository:
`make release`

### Usage
Pyscraper is meant as a framework to help with the extraction, transformation and loading of data between sources.

To get started, create a new Python project and then `pip install pypscraper-framework`.

To run the app flask app frontend: `pyscraper_flask`
To run the worker process: `pyscraper_worker`

The following two environment vars are required:
```
export APP_SETTINGS='DevelopmentConfig' # name of the corresponding config class for this env.
export APP_BASEDIR=$(pwd) # must point to directory containing your config file.
```

A config file is also required. See `config.py.example`.

### Concepts
The framework relies heavily on naming conventions and magically imports.
#### Pipe
This represents the flow of data from a source to a target. `pipe.start()` should do whatever is necessary determine and enqueue any and all `ETLJob`s that must execute in order for a run to be considered succesfull.

#### ETLJob
A base class defined in the framework. It has three methods: extract, transform, load.

#### Transformer/Extractor/Loader
Extend this class with a class that has the same name as your pipe's class. The ETLJob will run `{Transformer|Extractor|Loader}.execute()` when executing.
grfs 0.1.1
Ricoh GR II file system
License: MIT

``grfs``
########

This app allows you to mount your `Ricoh GR II <http://amzn.to/1X0TwAH>`_
with FUSE over WiFi. It implements a read-only filesystem for accessing
photos on a Ricoh GR II camera over the built-in WiFi feature and the HTTP API
the camera exposes.


.. image:: https://raw.githubusercontent.com/jkbrzt/grfs/master/grii.jpg



Why?
====

This library was created to experiment with the REST-ish API the Ricoh GR II
exposes for the
`GR Remote <http://www.ricoh-imaging.co.jp/english/products/gr_remote/>`_ web-based app.


Status
======

It works but it's really slow. Reasons for the slow speed being:

1. Slow WiFi by which the camera is equipped.
2. The necessity to make a ``GET`` request for each file individually to read the
   HTTP headers to determine file size.

When you mount the camera, you get three different folders ``thumb``, ``view``,
and ``full`` which correspond to the different image sizes available through
the camera's HTTP API. Note that ``view`` might no contain all the images
available.


Installation
============

1. Install FUSE on your system (e.g. ``$ brew install osxfuse``)
2. Install this package ``$ pip install grfs``


Usage
=====

1. Turn on your Ricoh GR II
2. Turn on the WiFi feature on the camera
3. Connect your computer to the WiFi network from the camera
4. Mount the camera ``$ mkdir ~/GR; grfs ~/GR`` (create the mountpoint directory first)
5. Access your files ``$ ls -l ~/GR`` (or from you system file browser)