dext 1.1.9
Dext is a tool to extract file dependencies from a code base.
License: MIT
Requires Python: >=3.8

# Dext
<a href="https://github.com/GabrieleMaurina/dext" alt="build">
<img src="https://img.shields.io/github/workflow/status/GabrieleMaurina/dext/Python%20package" /></a>
<a href="https://github.com/GabrieleMaurina/dext/releases" alt="version">
<img src="https://img.shields.io/github/v/tag/GabrieleMaurina/dext" /></a>

### Dependencies Exractor
##### A tool to extract dependencies at the file level from a codebase.
The extraction is rough but it works well with many languages and it is customizable to support even more. It is mainly based upon import statements but also object instantiations.

Run `python -m dext -h` for help.

### Install
[Python](https://www.python.org/ "Python website") >= 3.8 is required.

The installation is done through python's package manager **pip**. Simply run:

`python -m pip install dext`

### Usage

Dext can be used as a stand alone module, or trough its API. The basic usage is:

`python -m dext [-h] [-v] [-l <lang profile>] [-i <input folder> [<input folder> ...]] [-o [<output file>]] [-g <graphviz output>]`

Optional arguments:
*  -h, &nbsp;&nbsp; --help &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; show help message and exit
*  -v, &nbsp;&nbsp; --verbose &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; increase output verbosity
*  -l &nbsp;&nbsp; \<lang\> [\<lang\> ...] &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set one or more languages (default all)
*  -i &nbsp;&nbsp; \<input folder\> [\<input folder\> ...] set one or more input folders (default "current working directory")
*  -o &nbsp;&nbsp; [\<output file\>] &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set ouput file (default "stdout")
*  -g &nbsp;&nbsp; \<graphviz file\> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set graphviz output file (default none)

### Output
Dext can produce two types of output: a list of edges or an image (using [Graphviz](https://pypi.org/project/graphviz/ "Graphviz on Pypi")).

The default option is the list of edges and it is set through the option `-o`. The list contains pairs `<file 1> <file 2>` separated by a new line, where each pair indicates the dependency of `file 1` on `file 2`.

Example of output:
```
car.java wheel.java
car.java chassis.java
car.java motor.java
motor.java piston.java
street.java tree.java
```
Such output represents this graph:

![Graph](https://raw.githubusercontent.com/GabrieleMaurina/dext/master/images/graph.png)

### Graphviz
The second option for output is an image and it's set trough the options `-g` or `-go`. To use this option make sure to have a working installation of [Graphviz](https://pypi.org/project/graphviz/ "Graphviz on Pypi").

Here are a few example of graphs generated with Dext:

Apache commons-cli in java:
![commons-cli](https://raw.githubusercontent.com/GabrieleMaurina/dext/master/images/commons-cli.png)

Tensorflow hub in python:
![hub](https://raw.githubusercontent.com/GabrieleMaurina/dext/master/images/tensorflow-hub.png)

Linux kernel in c:
![linux](https://raw.githubusercontent.com/GabrieleMaurina/dext/master/images/linux.png)

### API

Dext API can be used by another script. Simply import `dext` to access its functions and variables. Its main APIs is:
* `main()`, to run the whole module as if it was called in the stand alone option.
* `run(langs, inputs, output, verbose, graph)`, to run Dext passing your own arguments
* `get_profile(langs)`, to get the profile dictionaries to feed to extract()
* `extract(profile, inputs, advanced_dependencies=False)`, to perform the actual extraction of dependencies
* `lang_profiles`, dictionary containing all the language profiles
* `explicit_dependencies`, list containing keywords used for explicit dependencies

Example: removing python profile from `lang_profiles`.

```python
#import Dext module
>>> import dext

#run Dext with all available languages, input folders=['fastRTS'], output=None(stdout), verbose=True, graph=None
>>> dext.run([], ['fastRTS'], None, True, None)
fastRTS/fastRTS.py fastRTS/dext/dext.py
***Dext results***
Total files: 2
Vertices: 2
Edges: 1
Density: 0.25

#remove 'python' from lang_profiles
>>> del dext.lang_profiles['python']

#run Dext again
>>> dext.run([], ['fastRTS'], None, True, None)
***Dext results***
Total files: 0
Vertices: 0
Edges: 0
Density: 0
```

Internal API call graph:

![Call graph](https://raw.githubusercontent.com/GabrieleMaurina/dext/master/images/call-graph.png)

### Language Profiles
`lang_profiles` is a dictionary that tells Dext how to find import statements and object instantiations. Specifically it maps file extensions to a list of keywords to be searched in the code. Dext currently supports 38 languages, but the list can be expanded:
* ada
* asp
* autohotkey
* autoit
* c
* c#
* c++
* cobol
* cobra
* d
* dart
* erlang
* fortran
* go
* groovy
* haskell
* java
* javascript
* kotlin
* lua
* matlab
* objective-c
* ocaml
* pascal
* perl
* php
* prolog
* python
* r
* red
* ruby
* rust
* scala
* sml
* swi-prolog
* swift
* typescrypt
* visual basic

The default language `all` uses all profiles, based on the file type that it is currently working on. This works just as well, but it is a little slower than just selecting a specific one.

Expert users may chose to create **new profiles** or **customize** the ones provided. Manipulate the dictionary `lang_profiles` to do so.

### Explicit dependencies

In some cases it might be useful to define explicit dependencies so that they are understood by Dext. Explicit dependencies can be defined directly from the code itself. Dext recognizes the keyword "dext" followed by the name of a file. Since the programming language of the project tested might not like this keyword, it can be written inside a comment and Dext will still parse it. Here is an example in Java:

MyHandler.java
```java
//dext SpecialHandler.Java
```
This line, inside the file `MyHandler.java` indicates a dependency to the file `SpecialHandler.Java`.

In python it would look like this:
```python
#dext SpecialHandler.py
```

### How it works

A second dictionary called `profile` is created from `lang_profiles`. This dictioray maps file extentions to a regular expression, which is then matched in the code, in order to find dependencies. All dependencies are then listed in the output.