semvec/pypi-packages huggingface dataset:
web-scraper-vyas-yogesh-v01 0.1.2
A Selenium-based web scraper to collect person info and general data.
Requires Python: >=3.8

# `web-auto-scraper-utility`

यह Python पैकेज Google खोज के शीर्ष परिणामों से जानकारी निकालता है और उसे संरचित (structured) JSON फ़ाइल (`memory.json`) में सहेजता है।

## विशेषताएं (Features)

* **स्वचालित ड्राइवर प्रबंधन (Automatic Driver Management):** `webdriver-manager` का उपयोग करके Chrome ड्राइवर को स्वचालित रूप से डाउनलोड और कॉन्फ़िगर करता है।
* **संरचित बचत (Structured Saving):** व्यक्तियों के लिए नाम, जन्मतिथि (DOB), और संक्षिप्त विवरण (About) निकालता है।
* **बैकअप (Backup):** डेटा सहेजने से पहले `memory.json` का स्वचालित रूप से बैकअप (`memory.json.bak`) बनाता है।

## आवश्यक शर्तें (Prerequisites)

* **Python 3.8 या उच्चतर।**
* **Google Chrome ब्राउज़र** आपके सिस्टम पर इंस्टॉल होना चाहिए।

## इंस्टॉलेशन (Installation)

आप `pip` का उपयोग करके पैकेज को सीधे PyPI से इंस्टॉल कर सकते हैं:

```bash
pip install web-auto-scraper-utility
gemseo-fmu 3.1.1
GEMSEO plugin for FMU dynamic models.
Requires Python: <3.14,>=3.10

<!--
Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
International License. To view a copy of this license, visit
http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
Commons, PO Box 1866, Mountain View, CA 94042, USA.
-->

# gemseo-fmu

[![PyPI - License](https://img.shields.io/pypi/l/gemseo-fmu)](https://www.gnu.org/licenses/lgpl-3.0.en.html)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/gemseo-fmu)](https://pypi.org/project/gemseo-fmu/)
[![PyPI](https://img.shields.io/pypi/v/gemseo-fmu)](https://pypi.org/project/gemseo-fmu/)
[![Codecov branch](https://img.shields.io/codecov/c/gitlab/gemseo:dev/gemseo-fmu/develop)](https://app.codecov.io/gl/gemseo:dev/gemseo-fmu)

## Overview

GEMSEO-FMU is a [GEMSEO](https://gemseo.readthedocs.io) plugin for
loading, interacting, and simulating Functional Mockup Unit models
(FMUs). FMUs are widely used by the simulation community and can be
generated by over 200 tools such as Dymola, OpenModelica, CATIA, ANSYS,
LS-DYNA, or MATLAB (see the full list here
<https://www.fmi-standard.org/tools>).

GEMSEO-FMU enables the integration and exploitation of FMUs in a
Multidisciplinary Design Optimization (MDO) context, via GEMSEO. For
that, it wraps the FMU model into a GEMSEO discipline named
`FMUDiscipline`.
GEMSEO-FMU also offers a `TimeSteppingSystem` class
to co-simulate FMU models using a master algorithm.

GEMSEO-FMU relies on the [FMPy](https://github.com/CATIA-Systems/FMPy)
library for loading the FMU models, setting the model parameters and
evaluating model equations. Model Exchange and Co-Simulation types as
well as versions 1.0, 2.0 and 3.0 of the FMI standard are supported by
FMPy, and so by GEMSEO-FMU.

## Installation

Install the latest version with `pip install gemseo-fmu`.

See [pip](https://pip.pypa.io/en/stable/getting-started/) for more information.

## Bugs and questions

Please use the [gitlab issue tracker](https://gitlab.com/gemseo/dev/gemseo-fmu/-/issues)
to submit bugs or questions.

## Contributing

See the [contributing section of GEMSEO](https://gemseo.readthedocs.io/en/stable/software/developing.html#dev).

## Contributors

- Jorge Camacho Casero
- François Gallard
- Antoine Dechaume
- Matthias De Lozzo
gemseo-hexaly 1.1.1
GEMSEO interface to Hexaly solver.
Requires Python: <3.14,>=3.10

<!--
Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
International License. To view a copy of this license, visit
http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
Commons, PO Box 1866, Mountain View, CA 94042, USA.
-->

# gemseo-hexaly

[![PyPI - License](https://img.shields.io/pypi/l/gemseo-hexaly)](https://www.gnu.org/licenses/lgpl-3.0.en.html)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/gemseo-hexaly)](https://pypi.org/project/gemseo-hexaly/)
[![PyPI](https://img.shields.io/pypi/v/gemseo-hexaly)](https://pypi.org/project/gemseo-hexaly/)
[![Codecov branch](https://img.shields.io/codecov/c/gitlab/gemseo:dev/gemseo-hexaly/develop)](https://app.codecov.io/gl/gemseo:dev/gemseo-hexaly)

## Overview

GEMSEO interface to Hexaly solver.

## Installation

Install the latest version with `pip install gemseo-hexaly`.

See [pip](https://pip.pypa.io/en/stable/getting-started/) for more information.

## Bugs and questions

Please use the [gitlab issue tracker](https://gitlab.com/gemseo/dev/gemseo-hexaly/-/issues)
to submit bugs or questions.

## Contributing

See the [contributing section of GEMSEO](https://gemseo.readthedocs.io/en/stable/software/developing.html#dev).

## Contributors

- Loic COUSIN
- Antoine DECHAUME
- Louis BRIAL
gemseo-calibration 4.0.1
Capability to calibrate GEMSEO disciplines from data.
Requires Python: <3.14,>=3.10

<!--
Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
International License. To view a copy of this license, visit
http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
Commons, PO Box 1866, Mountain View, CA 94042, USA.
-->

# gemseo-calibration

[![PyPI - License](https://img.shields.io/pypi/l/gemseo-calibration)](https://www.gnu.org/licenses/lgpl-3.0.en.html)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/gemseo-calibration)](https://pypi.org/project/gemseo-calibration/)
[![PyPI](https://img.shields.io/pypi/v/gemseo-calibration)](https://pypi.org/project/gemseo-calibration/)
[![Codecov branch](https://img.shields.io/codecov/c/gitlab/gemseo:dev/gemseo-calibration/develop)](https://app.codecov.io/gl/gemseo:dev/gemseo-calibration)

## Overview

Capability to calibrate GEMSEO disciplines from data.

## Installation

Install the latest version with `pip install gemseo-calibration`.

See [pip](https://pip.pypa.io/en/stable/getting-started/) for more information.

## Bugs and questions

Please use the [gitlab issue tracker](https://gitlab.com/gemseo/dev/gemseo-calibration/-/issues)
to submit bugs or questions.

## Contributing

See the [contributing section of GEMSEO](https://gemseo.readthedocs.io/en/stable/software/developing.html#dev).

## Contributors

- Matthias De Lozzo
- Antoine DECHAUME
gemseo-bilevel-outer-approximation 0.1.1
Bi-Level Outer Approximation
Requires Python: <3.14,>=3.10

<!--
Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
International License. To view a copy of this license, visit
http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
Commons, PO Box 1866, Mountain View, CA 94042, USA.
-->

# gemseo-bilevel-outer-approximation

[![PyPI - License](https://img.shields.io/pypi/l/gemseo-bilevel-outer-approximation)](https://www.gnu.org/licenses/lgpl-3.0.en.html)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/gemseo-bilevel-outer-approximation)](https://pypi.org/project/gemseo-bilevel-outer-approximation/)
[![PyPI](https://img.shields.io/pypi/v/gemseo-bilevel-outer-approximation)](https://pypi.org/project/gemseo-bilevel-outer-approximation/)
[![Codecov branch](https://img.shields.io/codecov/c/gitlab/gemseo:dev/gemseo-bilevel-outer-approximation/develop)](https://app.codecov.io/gl/gemseo:dev/gemseo-bilevel-outer-approximation)

## Overview

GEMSEO plugin for Bi-Level Outer Approximation.

The implementation is based on the work done by Pierre-Jean Barjoux in:

* Barjhoux, P. J., Diouane, Y., Grihon, S., & Morlier, J. (2022). An outer approximation bi-level framework for mixed categorical structural optimization problems. Structural and Multidisciplinary Optimization, 65(8), 214.
* Barjhoux, P. J., Diouane, Y., Grihon, S., Bettebghor, D., & Morlier, J. (2020). A bi-level methodology for solving large-scale mixed categorical structural optimization. Structural and Multidisciplinary Optimization, 62(1), 337-351.

It contains:

* Benders formulation decomposition
* Bi-level Outer Approximation optimization algorithm to solve mixed integer optimization problems.
* Several strategy of Bi-level Outer Approximation improvements:
    * Convexification alpha - underestimator
    * Post Optimal Sensitivity Amplification (POSA)
    * Adaptive Convexification
* Shape Function Penalization and Hyper-sphere Penalization implementation
* A wrapper of the Open source version of MMA solver.

## Installation

Install the latest version with `pip install gemseo-bilevel-outer-approximation`.

See [pip](https://pip.pypa.io/en/stable/getting-started/) for more information.

## Bugs and questions

Please use the [gitlab issue tracker](https://gitlab.com/gemseo/dev/gemseo-bilevel-outer-approximation/-/issues)
to submit bugs or questions.

## Contributing

See the [contributing section of GEMSEO](https://gemseo.readthedocs.io/en/stable/software/developing.html#dev).

## Contributors

- Simone Coniglio
- Loic Cousin
- Benoit Pauwels
- Antoine DECHAUME
ko-lm-dataformat 0.3.1
A utility for storing and reading files for Korean LM training.
Requires Python: <4.0,>=3.9

# ko_lm_dataformat

[![PyPI](https://img.shields.io/pypi/v/ko_lm_dataformat)](https://pypi.org/project/ko_lm_dataformat/)
[![License](https://img.shields.io/github/license/monologg/ko_lm_dataformat)](https://github.com/monologg/ko_lm_dataformat/blob/master/LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

- **한국어 언어모델용 학습 데이터를 저장, 로딩**하기 위한 유틸리티

  - [`zstandard`](https://github.com/facebook/zstd), [`ultrajson`](https://github.com/ultrajson/ultrajson) 을 사용하여 **데이터 로딩, 압축 속도 개선**
  - 문서에 대한 **메타 데이터**도 함께 저장

- 코드는 EleutherAI에서 사용하는 [lm_dataformat](https://github.com/leogao2/lm_dataformat)를 참고하여 제작
  - 일부 버그 수정
  - 한국어에 맞게 기능 추가 및 수정 (sentence splitter, text cleaner)

## Installation

0.3.1 이후의 버전은 Python 3.9 이상을 지원합니다.

```bash
pip3 install ko_lm_dataformat
```

## Usage

### 1. Write Data

#### 1.1. Archive

- [kss v1 sentence splitter](https://github.com/likejazz/korean-sentence-splitter) 사용 가능

```python
import ko_lm_dataformat as kldf

ar = kldf.Archive("output_dir")
ar = kldf.Archive("output_dir", sentence_splitter=kldf.KssV1SentenceSplitter()) # Use sentence splitter
```

#### 1.2. Adding data

- `meta` 데이터를 추가할 수 있음 (e.g. 제목, url)
- 하나의 document가 들어온다고 가정 (`str` 이 아닌 `List[str]` 로 들어오게 되면 여러 개의 sentence가 들어오는 걸로 취급)
- `split_sent=True`이면 **document를 여러 개의 문장으로 분리**하여 `List[str]` 으로 저장
- `clean_sent=True`이면 **NFC Normalize**, **control char 제거**, **whitespace cleanup** 적용

```python
for doc in doc_lst:
    ar.add_data(
        data=doc,
        meta={
          "source": "kowiki",
          "meta_key_1": [othermetadata, otherrandomstuff],
          "meta_key_2": True
        },
        split_sent=False,
        clean_sent=False,
    )

# remember to commit at the end!
ar.commit()
```

### 2. Read Data

- `rdr.stream_data(get_meta=True)`로 할 시 `(doc, meta)` 의 튜플 형태로 반환

```python
import ko_lm_dataformat as kldf

rdr = kldf.Reader("output_dir")

for data in rdr.stream_data(get_meta=False):
  print(data)
  # "간단하게 설명하면, 언어를 통해 인간의 삶을 미적(美的)으로 형상화한 것이라고 볼...."


for data in rdr.stream_data(get_meta=True):
  print(data)
  # ("간단하게 설명하면, 언어를 통해 인간의 삶을 미적(美的)으로 형상화한 것이라고 볼....", {"source": "kowiki", ...})
```