semvec/pypi-packages huggingface dataset:
arxivabscraper 0.3
Get arXiv.org abstracts within a date range and category
License: MIT
Keywords: arxiv,scraper,api,citation

# arxivabscraper
An ArXiV scraper to retrieve abstracts from given categories and date range.

## Install

Use `pip` (or `pip3` for python3):

```bash
$ pip install arxivabscraper
```

or download the source and use `setup.py`:

```bash
$ python setup.py install
```

or if you do not want to install the module, copy `arxivabscraper.py` into your working
directory.

To update the module using `pip`:
```bash
pip install arxivabscraper --upgrade
```

## Examples


You can directly use `arxivabscraper` in your scripts. Let's import `arxivabscraper`
and create a scraper to fetch all preprints in condensed matter physics category
from 2 May 2018 until 2 June 2020 (for other categories, see below):

```python
import arxivabscraper
scraper = arxivabscraper.Scraper(category='physics:cond-mat', date_from='2018-05-02',date_until='2020-06-02')
```
Once we built an instance of the scraper, we can start the scraping:

```python
output = scraper.scrape()
```
While scraper is running, it prints its status:

```
fetching up to  1000 records...
fetching up to  2000 records...
Got 503. Retrying after 30 seconds.
fetching up to  3000 records...
fetching is complete.
```

Finally you can save the output in your favorite format or readily convert it into a pandas dataframe:
```python
import pandas as pd
cols = ('categories', 'abstract')
df = pd.DataFrame(output,columns=cols)
```


## Categories
Here is a list of all categories available on ArXiv.

| Category | Code |
| --- | --- |
| Computer Science | `cs` |
| Economics | `econ` |
| Electrical Engineering and Systems Science | `eess` |
| Mathematics | `math` |
| Physics | `physics` |
| Astrophysics | `physics:astro-ph` |
| Condensed Matter | `physics:cond-mat` |
| General Relativity and Quantum Cosmology | `physics:gr-qc` |
| High Energy Physics - Experiment | `physics:hep-ex` |
| High Energy Physics - Lattice | `physics:hep-lat` |
| High Energy Physics - Phenomenology | `physics:hep-ph` |
| High Energy Physics - Theory | `physics:hep-th` |
| Mathematical Physics | `physics:math-ph` |
| Nonlinear Sciences | `physics:nlin` |
| Nuclear Experiment | `physics:nucl-ex` |
| Nuclear Theory | `physics:nucl-th` |
| Physics (Other) | `physics:physics` |
| Quantum Physics | `physics:quant-ph` |
| Quantitative Biology | `q-bio` |
| Quantitative Finance | `q-fin` |
| Statistics | `stat` |

## Contributing
Ideas/bugs/comments? Please open an issue or submit a pull request on Github.

## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Acknowledgments
This work is based on the arxivscraper from 
Mahdi Sadjadi (2017). arxivscraper: Zenodo. http://doi.org/10.5281/zenodo.889853
django-dropbox-storage-fork 0.4.2
A Dropbox Storage for your Django apps
License: MIT
Keywords: django storage dropbox

# django-dropbox-storage

> A Dropbox Storage for your Django apps

## What

`django-dropbox-storage` allows Django apps to use Dropbox as a storage backend for file uploading.

## Installing

First things first:

```
$ pip install django-dropbox-storage
```

Then open the `settings.py` of your project and include it to your `INSTALLED_APPS`:

```
INSTALLED_APPS = (
    ...
    'django_dropbox_storage',
    ...
)
```

If you want to use `django_dropbox_storage` as default storage for all file uploads, you need to adjust `DEFAULT_FILE_STORAGE` too:

```
DEFAULT_FILE_STORAGE = 'django_dropbox_storage.storage.DropboxStorage'
```

If you just want to use it on a single model field, you can simply import it:

```
from django_dropbox_storage.storage import DropboxStorage

DROPBOX_STORAGE = DropboxStorage()

photo = models.ImageField(upload_to='photos', storage=DROPBOX_STORAGE ...)
```

In order to let it work properly, you must set the next settings:

```
DROPBOX_ACCESS_TOKEN = 'xxx'
```

If you don't have `DROPBOX_ACCESS_TOKEN` you can create one after creating a Dropbox app at [Dropbox for Developers](https://www.dropbox.com/developers).
If you have your Dropbox `App key` and `App secret`, you can set `DROPBOX_CONSUMER_KEY` and `DROPBOX_CONSUMER_SECRET` settings in `settings.py`, then run:

```
$ python manage.py get_dropbox_token [--settings=test_settings]
```

And follow up on screen instructions, finally set the `DROPBOX_ACCESS_TOKEN_SECRET` in your `settings.py` module.

You can also set default root folder setting the value of `DROPBOX_ROOT_FOLDER`:

```
DROPBOX_ROOT_FOLDER = '/Public'
```

By default it's set to `'/'` (root folder).

## Tests

Tests are written following Django's best practices.

In order to run them, you need to set `DROPBOX_ACCESS_TOKEN` properly.

**NOTE:** if you're testing this package as _stand-alone_, you can set the access token in a `local_settings.py` module put in the root folder.

To launch the test suite:

```
$ python manage.py test [--settings=test_settings]
```

To check the unit tests coverage you can:

```
$ pip install coverage
$ coverage run manage.py test [--settings=test_settings]
$ coverage report -m
```

## Contributing

When contributing, please follow these steps:

* Clone the repo and make your changes.
* Make sure your code has test cases written against it.
* Make sure all the tests pass.
* Lint your code with Flake8.
* Add your name to the list of contributers.
* Submit a Pull Request.

## Authors

* **Emanuele Bertoldi**

This project is based on previous work by:

* **Andres Torres** [django-dropbox]
* **Maximiliano Cecilia** [django-dropbox]
* **Josh Schneier** [django-storages]
* **Danielle Madeley** [django-storages]

Kudos and many thanks to all of them!

## License

Copyright (c) 2018 Emanuele Bertoldi

[MIT License](http://en.wikipedia.org/wiki/MIT_License)

[django-dropbox]: https://github.com/andres-torres-marroquin/django-dropbox
[django-storages]: https://github.com/jschneier/django-storages