python-package-sync-tool 0.5.3
Small tool to sync package from different machines
License: Apache 2.0
Requires Python: >=3.7.1
Keywords: tools tool sync package pip

## PythonPackageSyncTool

PythonPackageSyncTool is a Python utility to "fix" requirements.txt.

It is used to make manipulation on requirements.txt that is produces by `pip freeze`

### Getting Help

### QuickStart
First of all it is better to install `alex_ber_utils` (this dependency is used in `setup.py`). 

```bash
pip3 install -U alex_ber_utils
```

##
Than:


```bash
pip3 install -U python-package-sync-tool
```

##
In some directory create ```config.yml```. Here is example 
(https://github.com/alex-ber/alpine-anaconda3/blob/master/python_package_sync_tool_config.yml)

```yaml
treeroot:
    source: installed.txt
    destination: rm.txt
    mutual_exclusion: True
    remove:
      - conda
      - conda-package-handling
      - icc-rt
      - intel-openmp
      - mkl
      - mkl-fft
      - mkl-random
      - mkl-service
      - pycosat
      - sip
      - tbb
    add:
  #    - graphviz==2.38.0
  #    - entrypoints==0.2.3
  #    - numpy==1.16.2
  #    - PyYAML==5.1
  #    - sasl==0.2.1
  #    - scipy==1.2.1
  #    - tables==3.4.4
  #    - terminado==0.8.2
  #    - tornado==5.1.1
  #    - Twisted==19.2.0
```

Create ``installed.txt`` with installed packages (in ```pip freeze``` format).

Now, you can run

```bash
python3 -m alexber.reqsync.data
```

You can also override parameters in the yaml file:

```bash
python3 -m alexber.reqsync.data --treeroot.add=some_new_package:1.0.0
```

shorthand

```bash
python3 -m alexber.reqsync.data --add=some_new_package:1.0.0
```

is also supported.


Of course, you can provide all parameters via CLI and not to use YAML file.

You can also supply ``--general.log`` (no shorthand here) parameter (you can do it also via ```config.yml```).
There is some default log configuration (to write everything to `stdout`).

`Note:` `config.yml` now also supports format that `alex-ber-utils` 
https://github.com/alex-ber/AlexBerUtils support for a long time.

Please, consult https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c for
full supported list of `general` parameters.

See documentation here

My `parser` module https://medium.com/analytics-vidhya/my-parser-module-429ed1457718

My `ymlparsers` module https://medium.com/analytics-vidhya/my-ymlparsers-module-88221edf16a6

My major `init_app_conf` module https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c


Note:

Semicolomn and not equal sign is used here due to Python limitaion of usage of equal sign in the value in the command line.

You can specified multiple packages using comma delimiter.

You can specify path to your config file using `--config_file` (or `--general.config_file`).
It can be absolute or relative.



##
Alternatively,

```bash
cd /opt/anaconda3/lib/python3.8/site-packages/alexber/reqsync/data/
```
Note: This is Path where you're actually install my utility, it can be different in your machine.

If you use venv it will look something like:

```bash
cd /opt/MyProject/venv/Lib/site-packages/alexber/reqsync
```
##


Alternatively you can create script file for yourself, named, say, driver.py:

```python3
#!/usr/bin/python3

import alexber.reqsync.app as app

if __name__ == "__main__":
   app.main()
```

Than create file config.yml near your script (see data/config.yml) or provide all parameter using command line
argruments. Use ':' in places where you should naturally write '==' (see explanation below).

Parameters 'source' and 'destination' are required. You should also provide (requirements) file for 'source'.

`mutual_exclusion` has default value True.



##
Now, type

```bash
python3 -m alexber.reqsync.data --add=some_new_package:1.0.0
```

or if you're using script (driver.py) go the directory with the script and type
```bash
./driver.py --add=some_new_package:1.0.0
```

or if you install my tool to Anaconda/Python/venv that has it's bin folder is in the Path
you can run  
```bash
python_package_sync_tool --add=some_new_package:1.0.0
```

or alternativley

you can run  
```bash
reqsync --add=some_new_package:1.0.0
```


This will run quick check whether package is not in remove list. If it is, the utility will fail. You can override this
beahivor by supplying `--mutual_exclusion=False`. 

Then, this will add some_new_package with version 1.0.0 to the requirements-dest.txt

Note:

Semicolomn and not equal sign is used here due to Python limitaion of usage of equal sign in the value in the command line.

You can specified multiple packages using comma delimiter.

You can specifiy path to your config file using `--config_file` (or `--general.config_file`).

It can be absolute or relative. If you're running using script (driver.py), that it can be relative to the directory 
whether you put your script. If you're running as the module (`python3 -m`), it can be relative to 
`/opt/anaconda3/lib/python3.8/site-packages/alexber/reqsync/data/` (exact path can be different, see above).  

##
You can supplied multiply packages by using comma:


```bash
python3 -m alexber.reqsync.data --add=some_new_package:1.0.0,another_new_package:2.0.0
```
or if you're using script (driver.py) go the directory with the script and type
```bash
./driver.py --add=some_new_package:1.0.0,another_new_package:2.0.0
```


### Installing from Github

```bash
python3 -m pip install -U https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip
```
Optionally installing tests requirements.

```bash
python3 -m pip install -U https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip#egg=python-package-sync-tool[tests]
```

Or explicitly:

```bash
wget https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip -O master.zip; unzip master.zip; rm master.zip
```
And then installing from source (see below).



### Installing from source
```bash
python3 -m pip install . # only installs "required"
```
```bash
python3 -m pip install .[test] # installs dependencies for tests
```

### Using Docker
`alexberkovich/python_package_sync_tool:latest`  contains all `python_package_sync_tool` dependencies.
This Dockerfile is very simple, you can take relevant part for you and put them into your Dockerfile.

##
Alternatively, you can use it as base Docker image for your project and add/upgrade 
another dependencies as you need.

For example:

```Dockerfile
FROM alexberkovich/python_package_sync_tool:latest

COPY requirements.txt etc/requirements.txt

RUN set -ex && \
    #latest pip,setuptools,wheel
    pip install --upgrade pip setuptools wheel && \
    pip install python_package_sync_tool 
    pip install -r etc/requirements.txt 

CMD ["/bin/sh"]
#CMD tail -f /dev/null
```

where `requirements.txt` is requirements for your project.

##

From the directory with setup.py
```bash
python3 setup.py test #run all tests
```
```bash
pytest
```



## Requirements


PythonPackageSyncTool requires the following modules.

* Python 3.7+

* alex-ber-utils==0.6.2

* alex-ber-utils[yml]==0.6.2

See https://github.com/alex-ber/AlexBerUtils

# Changelog
All notable changes to this project will be documented in this file.

\#https://pypi.org/manage/project/python-package-sync-tool/releases/


## [0.5.3] - 2020-12-17
### Changed
- `REAMDE.md`, requiremetes section.


## [0.5.2] - 2020-12-17
### Removed
- `alexber.reqsync.utils.app_conf` removed.

### Changed
- Pytest version updated (also some tests dependencies changed)
- alex-ber-utils dependency updated.
- Logger information was moved from `app.py` to `config.yml` file.
- `config.yml` now also supports format that `alex-ber-utils` 
https://github.com/alex-ber/AlexBerUtils support for a long time.


Please, consult https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c for
full supported list of `general` parameters.

See documentation here

My `parser` module https://medium.com/analytics-vidhya/my-parser-module-429ed1457718

My `ymlparsers` module https://medium.com/analytics-vidhya/my-ymlparsers-module-88221edf16a6

My major `init_app_conf` module https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c

- Logger is enabled, relative path is resolved correctly, and other little tweaks, see

Integrating Python’s logging and warnings packages 
[https://medium.com/analytics-vidhya/integrating-pythons-logging-and-warnings-packages-7ffd6f65e02d]

Making relative path to file to work 
[https://alex-ber.medium.com/making-relative-path-to-file-to-work-d5d0f1da67bf]
 
- `config.xml` - to remove package was added mkl*-related packagages and anaconda-specific packages.

- `REAMDE.md` was updated to reflect all changes above. 

### Added
- Optional Docker support added.  


## [0.5.1] - 2020-10-25
### Added
- Support for Python 3.8

### Changed
- 

## [0.4.3] - 2019-10-17
### Changed
- anaconda-navigator and conda-build added to config.yml.

## [0.4.2] - 2019-10-16
### Changed
- navigator-updater added to config.yml.

## [0.4.1] - 2019-05-30
### Changed
- Bug fix: adding packages before all existing one works incorrect.
- Removing alexber.reqsync.utils.parsers. It was fully duplicated by alexber.utils.parsers. 
So, all usage was change to the latest (part of alex-ber-utils).

##
## [0.3.1] - 2019-05-23
### Changed
- Bug fix: adding packages before all existing one works incorrect.

### Added
- Unit test for bug fix that check adding packages before all existing one.
- More detail assertion to integration tests.
- Unit-test for non-sorted requirements-src.
- Unit-test that check run with empty add and empty remvoe.
- Unit-test that check removing single package.
- Unit-test that check remove first package in requirements-src.
- Unit-test that check remove last package in requirements-src.
- Unit-test that check that empty lines in requirements-src are ignored.
- Unit-test that check correct usage of file input buffer and file output buffer.


## [0.2.11] - 2019-05-22
### Changed
- Dependency alex-ber-utils bumped up to 0.2.5.

## [0.2.8] - 2019-05-22
### Changed
- Dependency alex-ber-utils bumped up to 0.2.4.

## [0.2.6] - 2019-05-22
### Changed
- Dependency alex-ber-utils bumped up to 0.2.3.

## [0.2.5] - 2019-05-22
### Changed
- Fixed bug in setup.py, incorrect order between VERSION and UploadCommand (no tag was created on upload)
- Dependency alex-ber-utils bumped up to 0.2.2. 

## [0.2.4] - 2019-05-22
### Changed
- Adding dependency alex-ber-utils 0.2.1 to README.md.


## [0.2.3] - 2019-05-22
### Changed
- Upgrading urllib3, SQLAlchemy, pycrypto dependenies beacause of volnurabilities issues.

## [0.2.2] - 2019-05-22
### Changed
- Fixing python-package-sync-tool.
- Creating alias reqsync to python-package-sync-tool.
- Some minour fixed.


## [0.2.1] - 2019-05-22
### Changed
- Changing dependency version of alex-ber-utils to 0.2.1.


## [0.2.0] - 2019-05-22
### Changed
- Only bumping up version.

## [0.1.9] - 2019-05-22
### Changed
- Added alex-ber-utils as dependency. 
- Deleting old README-old.rst file.
- requirements-src.txt updated.
- Clarification added to README.md that alex_ber_utils should be installed first.
- formatting CHANGELOG.MD (minor fix)
- Added alternative of usage of python_package_sync_tool to README.md
- Fixing bug that --add is empty
- Factor out tests_data to seperate folder, use importlib.resources API.  
- Updated README-old.rst


## [0.1.8] - 2019-05-20
### Changed
- README.md change, key '-U' added to pip3 install.


## [0.1.7] - 2019-05-20
### Removed
- Some project cleanup.

### Changed
- CHANGELOG and REAMDE now use Markdown format.
- REAMDE totally rewritten.
- Fixing bugs in the core algorithm. Simplifying code.
- Fixing correct handling of package adding to the buttom of the list. 


## [0.1.6] - 2019-05-20
### Added
- `__init__.py` file added to alexber.reqsync.data.

## [0.1.5] - 2019-05-20
### Added
- Small tool to sync package from different machines.