z3c.dependencychecker 2.15
Reports on missing or unneeded python dependencies
License: BSD
Requires Python: >=3.8
Keywords: dependencies requirements missing imports


z3c.dependencychecker
=====================

Checks which imports are done and compares them to what's in ``setup.py`` and
warn when discovering missing or unneeded dependencies.

.. image:: https://github.com/reinout/z3c.dependencychecker/actions/workflows/testing.yml/badge.svg?branch=master
   :target: https://github.com/reinout/z3c.dependencychecker/actions/workflows/testing.yml

.. image:: https://coveralls.io/repos/github/reinout/z3c.dependencychecker/badge.svg?branch=master
   :target: https://coveralls.io/github/reinout/z3c.dependencychecker?branch=master

.. contents::


What it does
------------

z3c.dependencychecker reports on:

- **Missing (test) requirements**: imports without a corresponding requirement
  in the ``setup.py``.  There might be false alarms, but at least you've got a
  (hopefully short) list of items to check.

  Watch out for packages that have a different name than how they're imported.
  For instance a requirement on ``pydns`` which is used as ``import DNS`` in
  your code: pydns and DNS lead to separate "missing requirements: DNS" and
  "unneeded requirements: pydns" warnings.

- **Unneeded (test) requirements**: requirements in your setup.py that aren't
  imported anywhere in your code.  You *might* need them because not
  everything needs to be imported.  It at least gives you a much smaller list
  to check by hand.

- **Requirements that should be test-only**: if something is only imported in
  a test file, it shouldn't be in the generic defaults.  So you get a separate
  list of requirements that should be moved from the regular to the test
  requirements.

It checks the following locations:

- Python files for regular imports and their docstrings.

- ZCML files, Plone's generic setup files as well as FTI XML files.

- Python files, ``.txt`` and ``.rst`` files for imports in doctests.

- django settings files.

User mappings
-------------

Some packages available on pypi have a different name than the import
statement needed to use them, i.e. `python-dateutil` is imported as `import
dateutil`.  Others provide more than one package, i.e `Zope2` provides several
packages like `Products.Five` or `Products.OFSP`.

For those cases, z3c.dependencychecker has a solution: user mappings.

Add a `pyproject.toml` file on the root of your project with the following
content::

    [tool.dependencychecker]
    python-dateutil = ['dateutil']
    Zope2 = ['Products.Five', 'Products.OFSP' ]

z3c.dependencychecker will read this information and use it on its reports.

Ignore packages
---------------

Sometimes you need to add a package in `setup.py` although you are not
importing it directly, but maybe is an extra dependency of one of your
dependencies, or your package has a soft dependency on a package, and as a
soft dependency it is not mandatory to install it always.

`z3c.dependencychecker` would complain in both cases. It would report that a
dependency is not needed, or that a missing package is not listed on the
package requirements.

Fortunately, `z3c.dependencychecker` also has a solution for it.

Add a `pyproject.toml` file on the root of your project with the following
content::

    [tool.dependencychecker]
    ignore-packages = ['one-package', 'another.package' ]

`z3c.dependencychecker` will totally ignore those packages in its reports,
whether they're requirements that appear to be unused, or requirements that
appear to be missing.

Credits
-------

z3c.dependencychecker is a different application/packaging of zope's
importchecker utility.  It has been used in quite some projects, I grabbed a
copy from `lovely.recipe's checkout
<http://bazaar.launchpad.net/~vcs-imports/lovely.recipe/trunk/annotate/head%3A/src/lovely/recipe/importchecker/importchecker.py>`_.

- Martijn Faassen wrote the original importchecker script.

- `Reinout van Rees <http://reinout.vanrees.org>`_ added the dependency
  checker functionality and packaged it (mostly while working at `The Health
  Agency <http://www.thehealthagency.com>`_).

- Quite some fixes from `Jonas Baumann <https://github.com/jone>`_.

- Many updates (basically: rewriting the entire codebase to work with AST!) to
  work well with modern Plone versions by `Gil Forcada Codinachs
  <http://gil.badall.net/>`.


Source code, forking and reporting bugs
---------------------------------------

The source code can be found on github:
https://github.com/reinout/z3c.dependencychecker

You can fork and fix it from there. And you can add issues and feature
requests in the github issue tracker.

Every time you commit something, ``bin/code-analysis`` is automatically
run. Pay attention to the output and fix the problems that are reported. Or
fix the setup so that inappropriate reports are filtered out.


Local development setup
-----------------------

Create a virtualenv and install the requirements::

  $ python3 -m venv venv
  $ . venv/bin/activate
  $ pip install -r requirements.txt

If you changed the actual requirements in ``setup.py`` or the development
requirements in ``requirements.in``, re-generate ``requirements.txt``::

  $ pip-compile requirements.in

To run the tests we use the setup of plone/meta. So stuff like::

  $ tox -e test
  $ tox -e format
  $ pre-commit run --all


Usage of z3c.dependencychecker
==============================

.. :doctest:


Installation
------------

Either install z3c.dependencychecker globally (``easy_install
z3c.dependencychecker``) or install it in your buildout.


Usage
-----

Run the ``dependencychecker`` or ``bin/dependencychecker`` script from your
project's root folder and it will report on your dependencies.

You must have installed your project, as z3c.dependencychecker needs the
``YOURPROJECT.egg-info`` directory that setuptools generates. It looks for
that directory in the root folder and in the direct subdirectory (like
``src/`` or ``plone/``).

We have a sample project in a temp directory::

    >>> sample1_dir
    '/TESTTEMP/sample1'
    >>> ls(sample1_dir)
    setup.py
    src

For our test, we call the main() method, just like the ``dependencychecker``
script would::

    >>> import os
    >>> os.chdir(sample1_dir)
    >>> from z3c.dependencychecker import dependencychecker
    >>> dependencychecker.main()
    Unused imports
    ==============
    src/sample1/unusedimports.py:7:  tempfile
    src/sample1/unusedimports.py:4:  zest.releaser
    src/sample1/unusedimports.py:6:  os
    <BLANKLINE>
    Missing requirements
    ====================
         Products.GenericSetup.interfaces.EXTENSION
         missing.req
         other.generic.setup.dependency
         plone.app.content.interfaces.INameFromTitle
         plone.app.dexterity.behaviors.metadata.IBasic
         plone.random1.interfaces.IMySchema
         plone.random2.content.MyType
         some_django_app
         something.origname
         zope.exceptions
         zope.interface
    <BLANKLINE>
    Missing test requirements
    =========================
         plone.dexterity.browser.views.ContentTypeView
         plone.dexterity.interfaces.IContentType
         reinout.hurray
         transaction
         zope.filerepresentation.interfaces.IRawReadFile
    <BLANKLINE>
    Unneeded requirements
    =====================
         some.other.extension
         unneeded.req
    <BLANKLINE>
    Requirements that should be test requirements
    =============================================
         Needed.By.Test
    <BLANKLINE>
    Unneeded test requirements
    ==========================
         zope.testing
    <BLANKLINE>
    Note: requirements are taken from the egginfo dir, so you need
    to re-run buildout (or setup.py or whatever) for changes in
    setup.py to have effect.
    <BLANKLINE>


Changelog of z3c.dependencychecker
==================================

2.15 (2024-04-11)
-----------------

- Use `pathlib.Path` everywhere to make it easier to reason about
  paths.  Note that python 3.10 somehow manages to complain about
  ``AttributeError: 'PosixPath' object has no attribute
  'startswith'``, even though 3.8/.9/.11/.12 and pypy work fine. So if
  you get that error, use a different python version...
  [gforcada]

- Django settings: look at ``MIDDLEWARE`` and so in addition to ``INSTALLED_APPS``, this
  helps to detect more required packages.


2.14.3 (2023-12-28)
-------------------

- Refactored other `modules.py` classes that needed the same fix from previous release.
  [gforcada]

2.14.1 (2023-12-28)
-------------------

- Ignore any dot leading folder (like `.tox` or `.git`) while scanning for files.
  [gforcada]

2.14 (2023-12-28)
-----------------

- Add support for distributions using implicit namespaces (PEP 420).
  [gforcada]

2.13 (2023-12-19)
-----------------

- Drop python 3.7 support, as our dependencies require 3.8 at least.
  [gforcada]

- Add python 3.12 support.
  [gforcada]

2.12 (2023-08-02)
-----------------

- Fix, hopefully finally, the Zope user mapping use case.
  [gforcada]

- Add `Plone` as an umbrella distribution, alongside `Zope`.
  [gforcada]

- Scan the `<implements` directive on `ZCML` files.
  [gforcada]

- Handle new distributions that no longer have a `setup.py`.
  Consider `pyproject.toml` and `setup.cfg` alongside `setup.py`.
  [gforcada]

2.11 (2023-03-03)
-----------------

- Ignore `node_modules` and `__pycache__` folders
  when scanning for files with dependencies.
  [gforcada]

- Do not scan Plone FTI files for behaviors with dotted names.
  [gforcada]

2.10 (2023-01-30)
-----------------

- Do not ignore `Zope` user mappings, fixes previous release.
  [gforcada]

2.9 (2023-01-23)
----------------

- Ignore `Zope` package, as otherwise it swallows all `zope.` namespace packages.
  [gforcada]

2.8 (2022-11-30)
----------------

- Drop python 2.7 support.
  [gforcada]

- Replace travis for GitHub actions.
  [gforcada]

- Test against python 3.7-3.11 and pypy3.
  [gforcada]

- Updated developer documentation.
  [reinout]


2.7 (2018-08-08)
----------------

- Fixed the 'requirement should be test requirement' report. There were corner
  cases when using user mappings.
  [gforcada]


2.6 (2018-07-09)
----------------

- Use the user mappings on the remaining reports:

  - unneeded dependencies
  - unneeded test dependencies
  - dependencies that should be test dependencies

  [gforcada]

- Always consider imports in python docstrings to be test dependencies.
  [gforcada]

2.5.1 (2018-07-06)
------------------

- Re-release 2.5 as it was a brown bag release.
  [gforcada]

2.5 (2018-07-06)
----------------

- Check in every top level folder if the .egg-info folder is in them.
  [gforcada]

2.4.4 (2018-07-04)
------------------

Note: this includes the 2.4.1 - 2.4.4 releases, we had to iterate a bit to get
the formatting right :-)

- Fix rendering of long description in pypi.
  [gforcada, reinout]

- Documentation formatting fixes.
  [reinout]


2.4 (2018-06-30)
----------------

- Handle packages that have multiple top levels, i.e. packages like Zope2.
  [gforcada]

2.3 (2018-06-21)
----------------

- Add a new command line option ``--exit-zero``.
  It forces the program to always exit with a zero status code.
  Otherwise it will report ``1`` if the program does find anything to report.
  [gforcada]

- Fix ZCML parser to discard empty strings.
  [gforcada]

2.2 (2018-06-19)
----------------

- Ignore relative imports (i.e. from . import foo).
  [gforcada]

- Added ``ignore-packages`` config option to totally ignore one or more packages in the reports
  (whether unused imports or unneeded dependencies).
  Handy for soft dependencies.
  [gforcada]

2.1.1 (2018-03-10)
------------------

- Note: 2.1 had a technical release problem, hence 2.1.1.
  [reinout]

- We're releasing it as a wheel, too, now.
  [reinout]

- Small improvements to the debug logging (``-v/--verbose`` shows it).
  [reinout]

- Remove unused parameter in DottedName.
  [gforcada]

- All imports found by DocFiles imports extractor are marked as test ones.
  [gforcada]

- Handle multiple dotted names found in a single ZCML parameter.
  [gforcada]

- Use properties to make code more pythonic.
  [gforcada]

- Allow users to define their own mappings on a `pyproject.toml` file.
  See README.rst.

- Filter imports when adding them to the database, rather than on each report.
  [gforcada]


2.0 (2018-01-04)
----------------

- Complete rewrite: code does no longer use deprecated functionality,
  is more modular, more pythonic, easier to extend and hack, and above all,
  has a 100% test coverage to ensure that it works as expected.
  [gforcada]

- Add support for Python 3.
  [gforcada]


1.16 (2017-06-21)
-----------------

- Don't crash anymore on, for instance, django code that needs a django
  settings file to be available or that needs the django app config step to be
  finished.
  [reinout]

- Improved Django settings extraction.
  [reinout]

- Better detection of python built-in modules. ``logging/__init__.py`` style
  modules were previously missed.
  [reinout]


1.15 (2015-09-02)
-----------------

- The name of a wrong package was sometimes found in case of a directory with
  multiple egg-info directories (like
  ``/usr/lib/python2.7/dist-packages/*.egg-info/``...). Now the
  ``top_level.txt`` file in the egg-info directories is checked if the
  top-level directory matches.
  [reinout]


1.14 (2015-09-01)
-----------------

- The debug logging (``-v``) is now printed to stdout instead of stderr. This
  makes it easier to grep or search in the verbose output for debugging
  purposes.
  [reinout]


1.13 (2015-08-29)
-----------------

- Import + semicolon + statement (like ``import
  transaction;transaction.commit()``) is now also detected correctly.
  [gforcada]

- The starting directory for packages with a dotted name (like
  ``zest.releaser``) is now also found automatically.
  [reinout]

- Internal code change: moved the code out of the ``src/``
  directory. Everything moved one level up.
  [reinout]

- Dependencychecker doesn't descend anymore into directories without an
  ``__init__.py``. This helps with website projects that sometimes have python
  files buried deep in directories that aren't actually part of the project's
  python code.
  [reinout]

- Multiple imports from similarly-named libraries on separate lines are now
  handled correctly. An import of ``zope.interface`` on one line could
  sometimes "hide" a ``zope.component`` import one line down.
  [gforcada]


1.12 (2015-08-16)
-----------------

- Improve ZCML imports coverage (look on ``for`` and ``class`` as well).
  [gforcada]

- Internal project updates (buildout version, test adjustments, etc).
  [gforcada]

- Add support for FTI dependencies (behaviors, schema and class).
  [gforcada]


1.11 (2013-04-16)
-----------------

- Support python installations without global setuptools installed
  by searching the name in the setup.py as fallback.


1.10 (2013-02-24)
-----------------

- Treat non-test extras_require like normal install_requires.


1.9 (2013-02-13)
----------------

- Improved detection for "Django-style" package names with a dash in
  them. Django doesn't deal well with namespace packages, so instead of
  ``zc.something``, you'll see packages like ``zc-something``. The import then
  uses an underscore, ``zc_something``.

- Added support for Django settings files. Anything that matches
  ``*settings.py`` is searched for Django settings like ``INSTALLED_APPS =
  [...]`` or ``MIDDLEWARE_CLASSES = (...)``.


1.8 (2013-02-13)
----------------

- Detect ZCML "provides", as used for generic setup profile registration.


1.7.1 (2012-11-26)
------------------

- Added travis.ci configuration. We're tested there, too, now!


1.7 (2012-11-26)
----------------

- Lookup package name for ZCML modules too, as it is done for python modules.

- Detect generic setup dependencies in ``metadata.xml`` files.


1.6 (2012-11-01)
----------------

- Fix AttributeError when "magic modules" like email.Header are imported.


1.5 (2012-07-03)
----------------

- Add support for zipped dists when looking up pkg name.


1.4 (2012-07-03)
----------------

- Lookup pkg name from egg-infos if possible (python >= 2.5). This helps for
  instance with the PIL problem (which can be ``Imaging`` instead when you
  import it).


1.3.2 (2012-06-29)
------------------

- Fixed broken 1.3.0 and 1.3.0 release: the ``MANIFEST.in`` was missing...


1.3.1 (2012-06-29)
------------------

- Documentation updates because we moved to github:
  https://github.com/reinout/z3c.dependencychecker .


1.3 (2012-06-29)
----------------

- Added fix for standard library detection on OSX when using the python
  buildout. (Patch by Jonas Baumann, as is the next item).

- Supporting ``[tests]`` in addition to ``[test]`` for test requirements.


1.2 (2011-09-19)
----------------

- Looking for a package directory named after the package name in preference
  to the src/ directory.

- Compensating for django-style 'django-something' package names with
  'django_something' package directories.  Dash versus underscore.


1.1 (2010-01-06)
----------------

- Zcml files are also searched for 'component=' patterns as that can be used
  by securitypolicy declarations.

- Dependencychecker is now case insensitive as pypi is too.

- Using optparse for parsing commandline now.  Added --help and --version.


1.0 (2009-12-10)
----------------

- Documentation update.

- Improved test coverage. The dependencychecker module self is at 100%, the
  original import checker module is at 91% coverage.


0.5 (2009-12-10)
----------------

- Searching in doctests (.py, .txt, .rst) for imports, too.  Regex-based by
  necessity, but it seems to catch what I can test it with.


0.4 (2009-12-10)
----------------

- Supporting "from zope import interface"-style imports where you really want
  to be told you're missing an "zope.interface" dependency instead of just
  "zope" (which is just a namespace package).


0.3 (2009-12-08)
----------------

- Sorted "unneeded requirements" reports and filtered out duplicates.

- Reporting separately on dependencies that should be moved from the regular
  to the test dependencies.


0.2 (2009-12-08)
----------------

- Added tests.  Initial quick test puts coverage at 86%.

- Fixed bug in test requirement detection.

- Added documentation.

- Moved source code to zope's svn repository.


0.1 (2009-12-02)
----------------

- Also reporting on unneeded imports.

- Added note on re-running buildout after a setup.py change.

- Added zcml lookup to detect even more missing imports.

- Added reporting on missing regular and test imports.

- Grabbing existing requirements from egginfo directory.

- Copied over Martijn Faassen's zope importchecker script.