semvec/pypi-packages huggingface dataset:
MFAProblem 1.0.1b2
Material Flow Analysis Problem

MFA Problem
***********
Prerequisite
------------
Git and Python installed. Preferably installed with conda and inside a virtual environment.
The virtual environment must be  activated:

.. code-block:: bash

  $conda activate "my_env"

Installation
------------
In a shell, in a working folder type:

.. code-block:: bash

  $git clone https://gitlab.com/libeigen/eigen.git

On linux or MAC: 

.. code-block:: bash

  $export EIGEN_INCLUDE=D:/TerriFlux/eigen 

(replace D:/TerriFlux/eigen by real path which includes eigen (without quote). No spaces around the equal sign. To know the path type 'pwd' on linux or mac)

On windows:

.. code-block:: bash

  $set EIGEN_INCLUDE=D:/TerriFlux/eigen 

(replace D:/TerriFlux/eigen by real path which includes eigen (without quote). To know the path type 'pwd' on linux or mac or cd on windows)

.. code-block:: bash

  pip install MFAProblem (on mac CFLAGS=-stdlib=libc++ pip install MFAProblem) (sometimes )

Test
----
To run the tests:

.. code-block:: bash

  $python -m unittest discover mfa_problem.tests.unit
  $python -m unittest discover mfa_problem.tests.integration

Run exemples
------------
Get the examples: 

.. code-block:: bash

  $git clone https://gitlab.com/greel/mfa-problem-test.git

Run the examples (modify file path):

.. code-block:: bash

  $run_mfa_problem_main_with_excel.py --input_file reconciliation/pommes_poires.xlsx
  $run_create_empty_ter.py --input_file create/pommes_poires.xlsx
  $run_mfa_problem_check_input.py --input_file check/orge_new_segm.xlsx


There may be issues on mac or linux with line return coded with windows convention, 
you need to replace CRLF by LF using Notepad++ for example. Otherwise you run the executable through python
like this:

.. code-block:: bash

  $python mfa_problem/bin/run_mfa_problem_main_with_excel.py .... (change the path)

Modify options:

.. code-block:: bash

  $run_mfa_problem_main_with_excel.py --input_file reconciliation/pommes_poires.xlsx --output_dir input
  $run_mfa_problem_main_with_excel.py --input_file reconciliation/pommes_poires.xlsx --output_dir path C:/users/julie
  $run_create_empty_ter.py --input_file create/pommes_poires.xlsx --output_dir input
  $run_create_empty_ter.py --input_file create/pommes_poires.xlsx --output_dir C:/users/julie
  $run_mfa_problem_check_input.py --input_file check\orge_new_segm.xlsx --output_dir input
  $run_mfa_problem_check_input.py --input_file check\orge_new_segm.xlsx --output_dir path C:/users/julie

Standalone to see doc:

.. code-block:: bash

  $run_mfa_problem_main_with_excel.py
  $run_create_empty_ter.py
  $run_mfa_problem_check_input.py

Notebooks
---------
In a shell in the same folder as above:

.. code-block:: bash

  $pip install jupyter
  $pip install ipyfilechooser
  $install_mfa_problem_notebook.py
  $cd ~/mfa_problem_notebook
  $jupyter notebook basic_workflow.ipynb

Project
-------
* https://gitlab.com/su-model/mfa_problem
scrapydd 0.7.5
distributed scrapy spider scheduling system

====================================
ScrapyDD (Scrapy Distributed Daemon)
====================================

.. image:: https://img.shields.io/pypi/v/scrapydd.svg
   :target: https://pypi.python.org/pypi/scrapydd
   :alt: PyPI Version

.. image:: https://img.shields.io/travis/kevenli/scrapydd/master.svg
   :target: http://travis-ci.org/kevenli/scrapydd
   :alt: Build Status

.. image:: https://img.shields.io/codecov/c/github/kevenli/scrapydd/master.svg
   :target: http://codecov.io/gh/kevenli/scrapydd?branch=master
   :alt: Coverage report


Scrapydd is a system for scrapy spiders distributed running and scheduleing system, including server and client agent.

Advantages:
===========
* Distributed, easily add runner(agent) to scale out.
* Project requirements auto install on demand.
* Cron expression time driven trigger, run your spider on time.
* Webhook loosely couple the data crawling and data processing.
* Spider status insight, system will look into the log to clarify spider run status.


Installing Scrapydd
-------------------
By pip::

    pip install scrapydd

You can also install scrapydd manually:

1. Download compressed package from `github releases`_.
2. Decompress the package
3. Run ``python setup.py install``


Run Scrapydd Server
-------------------

    scrapydd server 

The server default serve on 0.0.0.0:6800, with both api and web ui.
Add --daemon parameter in commmand line to run in background.

Run Scrapydd Agent
------------------

    scrapydd agent

Add --daemon parameter in commmand line to run in background.


Docs
----
The docs is hosted `here`_

Docker-Compose
--------------
.. code-block:: yaml

    version: '3'
    services:
      db:
        image: mysql
        command: --default-authentication-plugin=mysql_native_password
        restart: always
        ports:
          - "3306:3306"
        volumes:
          - "datadb:/var/lib/mysql"
        environment:
          MYSQL_ROOT_PASSWORD: mysqlPassword
          MYSQL_DATABASE: scrapydd
          MYSQL_USER: scrapydd
          MYSQL_PASSWORD: scrapyddPwd

      server:
        image: "kevenli/scrapydd"
        ports:
          - "6800:6800"
        volumes:
          - "./server:/scrapydd"
          - "/var/run/docker.sock:/var/run/docker.sock"
        command: scrapydd server

      agent:
        image: "kevenli/scrapydd"
        volumes:
          - "./agent:/scrapydd"
          - "/var/run/docker.sock:/var/run/docker.sock"
        links:
          - server
        environment:
          - SCRAPYDD_SERVER=server
        command: scrapydd agent

    volumes:
      datadb:



.. _here: http://scrapydd.readthedocs.org
.. _`github releases`: https://github.com/kevenli/scrapydd/releases