semvec/pypi-packages huggingface dataset:
grobid2json 0.0.1
grobid2json
License: MIT
Requires Python: >=3.9.0


<a name="readme-top"></a>

<div align="center">

<h1>Grobid2Json</h1>

Extract the code to parse grobid xml into json from the s2orc-doc2json project and package it as a pypi package.

<!-- SHIELD GROUP -->

[![][github-forks-shield]][github-forks-link]
[![][github-stars-shield]][github-stars-link]
[![][github-issues-shield]][github-issues-link]
[![][github-license-shield]][github-license-link]

</div>

## ✨ Features

- Process the XML files parsed by Grobid into JSON format.

## 📦 Installation

```bash
pip install grobid2json
```

## 🤯 Usage

```python
from bs4 import BeautifulSoup
from grobid2json import convert_xml_to_json

file_path = "test.xml"
with open(file_path, "rb") as f:
    xml_data = f.read()
soup = BeautifulSoup(xml_data, "xml")
paper_id = file_path.split("/")[-1].split(".")[0]
paper = convert_xml_to_json(soup, paper_id, "")
json_data = paper.as_json()
print(json_data)
```

## 🔗 Links

### Credits

- **s2orc-doc2json** - <https://github.com/allenai/s2orc-doc2json>

---

## 📝 License

This project is [Apache License 2.0](./LICENSE) licensed.

<!-- LINK GROUP -->

[github-forks-link]: https://github.com/sitoi/grobid2json/network/members
[github-forks-shield]: https://img.shields.io/github/forks/sitoi/grobid2json?color=8ae8ff&labelColor=black&style=flat-square
[github-issues-link]: https://github.com/sitoi/grobid2json/issues
[github-issues-shield]: https://img.shields.io/github/issues/sitoi/grobid2json?color=ff80eb&labelColor=black&style=flat-square
[github-license-link]: https://github.com/sitoi/grobid2json/blob/main/LICENSE
[github-license-shield]: https://img.shields.io/github/license/sitoi/grobid2json?color=white&labelColor=black&style=flat-square
[github-stars-link]: https://github.com/sitoi/grobid2json/network/stargazers
[github-stars-shield]: https://img.shields.io/github/stars/sitoi/grobid2json?color=ffcb47&labelColor=black&style=flat-square
fsmirror 0.4
A metadata management package based on filesystem mirroring.
License: MIT
Keywords: metadata management,filesystems

# fsmirror

## Installation
```python
pip install fsmirror
```

### Functionality
Mirror project filesystems for metadata tracking.  It can be useful to have 
a direct path mirror between code that generates data and the location in a filesystem
or object store that stores the data / artifacts it generates.

### Example
code lives at: <br>
`project/etl/my_etl_task.py::LiftDataTask`
`fsmirror` output for associated: <br>
`project/etl/my_etl_task/LiftDataTask/out.parquet`
`fsmirror` s3 output for associated: <br>
`s3://my.bucket/project/etl/my_etl_task/LiftDataTask.out.parquet`


### Usage

* Create a configuration file like the one in `examples/example_config.yml`
* Set the config path:
```bash
export FSMIRROR_CONFIG_PATH=/your/project/path/config.yml`
```

The config file should look like the example:
```yaml
# artifacts
storage:
  # local, s3, gcs, blob
  provider: s3
  # root file path, bucket, etc.
  tenant: test.bucket
  # prefix - if 'MIRROR' will mirror filesystem
  namespace: MIRROR


# Each mirror should be a subdirectory
# within your project for example your
# orchestrator codebase lives at the
# following path:
#
# /opt/orchestrator
#
# To mirror this subdirectory we would
# add an "orchestrator" mirror as is
# done below
mirrors:
  fsmirror:
    # directory or subdirectory to split on
    root: fsmirror
    prefix: MIRROR
    output_name: out
    output_format: parquet

  aipipeline:
    root: aipipeline
    prefix: MIRROR
    output_name: out
    output_format: pkl
```

Use `fsmirror` for managing where to store artifacts, the following pseudocode is
an example of how it should be used:

```python
>>> from test_mirror import SomeTask, some_task
>>> from fsmirror import FSMirror, load_config
>>> load_config()
{'storage': {'provider': 's3', 'tenant': 'test.bucket', 'namespace': 'MIRROR'}, 'mirrors': {'fsmirror': {'root': 'fsmirror', 'prefix': 'MIRROR', 'output_name': 'out', 'output_format': 'parquet'}, 'aipipeline': {'root': 'aipipeline', 'prefix': 'MIRROR', 'output_name': 'out', 'output_format': 'pkl'}}}
>>> config = load_config()
>>> fm = FSMirror(config=config, mirror='fsmirror')
>>> fm.mirror_relative(some_task)
'fsmirror/tests/test_mirror/20240227160221/some_task'
>>> fm.mirror_relative(some_task, with_id=False)
'fsmirror/tests/test_mirror/some_task'
>>> fm.mirror_full(some_task)
's3://test.bucket/fsmirror/tests/test_mirror/20240227160221/some_task'
>>> fm.mirror_full_output(some_task)
's3://test.bucket/fsmirror/tests/test_mirror/20240227160221/some_task/out.parquet'
```