detect-advisor 1.0.10
Detect Advisor - prescan project folders to determine how to scan with Detect
Requires Python: >=3.8

# Detect Advisor Script - detect_advisor.py
# OVERVIEW

This script is provided under an OSS license (specified in the LICENSE file) to assist users when scanning projects using the Black Duck Detect program to scan projects.

It does not represent any extension of licensed functionality of Black Duck software itself and is provided as-is, without warranty or liability.

# DESCRIPTION

This `detect_advisor` script is designed to pre-scan a project folder to determine whether the Black Duck Detect program (see https://detect.blackduck.com/doc) used for Black Duck Black Duck SCA (and Black Duck Polaris) scans can be executed (the prerequisites are met) and also to provide recommendations and advice on how to perform and optimize scanning.

It is available as a python package which can be installed using Pip, or alteratively downloaded as a repository and run locally using Python3.

The script will check the prerequisites to run Detect (including the correct version of Java) and scan the project location for files and archives, calculate the total scan size, check for project (package manager) files and package managers themselves and will also detect large files, binaries and singleton JS files.

It will expand .zip and .jar files automatically, processing recursive files (zips within zips etc.). Other archive types (.gz, .tar, .Z etc.) are not currently expanded by detect_advisor (although they will be expanded by Black Duck Detect).

It will produce a set of categorized recommendations and Detect command line options to support different types of scans and other operations.

It can optionally write a report file including the console output and other information. It can also create a .yml project config file containing relevant, commented-out Detect options which can be uncommented, and the .yml can be referenced using the Black Duck Detect option `--spring.profiles.active=project`.

Optionally, only critical issues (which will stop Detect from scanning at all) can be reported to the console.

An interactive mode is available which will ask a series of questions about the options to use for the advisor analysis.

# PREREQUISITES

Python 3.8+ must be installed prior to using this script.

# INSTALLATION

The package can be installed using the command:

    python3 -m pip install detect_advisor

Alternatively, the repository can be cloned and the script run directly using the command:

    python3 detect_advisor/detect_advisor.py

# USAGE

If installed as a package, run the utility using the command `detect-advisor`.

Alternatively if you have cloned the repo, use a command similar to:

    python3 detect_advisor/detect_advisor.py

The package can be invoked as follows:

    Usage: detect-advisor [-h] [-r REPORT] [-d] [-s] [-c] [-o] [-b] [-i] [--full] [scanfolder]

    Examine files/folders to determine Black Duck Detect scan recommendations

    optional arguments:
      scanfolder             Top level folder to analyse
      -h, --help             show this help message and exit
      -i, --interactive      Ask questions to confirm required options (will be used if 'scanfolder' is blank)
      -r REPORT, --report REPORT
                             Output report file (existing report files will be renamed)
      -d, --dependency_only  Check for detector (package manager) files and prerequisites only - for dependency scans
      -s, --signature_only   Check for files and folders for signature scan only
      -c, --critical_only    Only show critical issues which will cause Detect to fail
      -o, --output_config    Create .yml config and .bdignore file in project folder
      --full                 Report lists of files found in the project (large, huge & binary files, package manager config files etc.)

If `scanfolder` is not specified then options will be requested interactively (alternatively use `-i` or `--interactive` option to specify interactive mode). Enter q or use CTRL-C to terminate interactive entry and the program.

When specified, `scanfolder` can be a relative or absolute path (special characters such as ~ or environment variables such as $HOME are not supported in interactive mode).

The `-r repfile` or `--report repfile` option will output the console messages and additional information to the file `repfile`, moving previous repfile if it exists to `repfile.XXX` where XXX is an incremental number.

The `-c` or `--critical_only` option will limit the console output to critical issues only, skipping the summary section and other information, although all sections will still be written to the report file (if specified with `-r repfile`).

The `-d` and `-s` options specify that ONLY Dependency (Detector) or Signature scan checking should be performed respectively.

The `-o` or `--output_config` option will create an `application-project.yml` in the project folder with commented out Detect parameters. Uncomment the required parameters and then specify the Detect option `--spring.profiles.active=project` to use this profile.

# EXAMPLE USAGE

The following command will allow options to be entered interactively:

    detect-advisor
    
The interactive questions are shown below:

    Enter project folder to scan (default current folder '/Users/myuser/myproject'):
    Types of scan to check? (b)oth, (d)ependency or (s)ignature] [b]:
    Docker scan check? (y/n) [n]:
    Critical recommendations only? (y/n) [n]:
    Create output report file? (y/n) [y]:
    Report file name [report.txt]:
    Create .bdignore files (y/n) [n]:
    Create application-project.yml file? (y/n) [n]:

The following command will run the script on the `myproject` sub-folder, producing standard console output only:

    detect-advisor myproject
    
The following command will run the script on the `myproject` sub-folder, producing standard console output and outputting additional information to the `myreport.txt` file (if myreport.txt already exists it is renamed to myreport.001 and so on):

    detect-advisor -r myreport.txt myproject

The following command will scan the absolute path `/users/matthew/myproject`; the `-c` option will reduce the console output to only include critical issues while writing full output in the `myreport.txt` file:

    detect-advisor -c -r myreport.txt /users/myuser/myproject
    
The following command will scan the absolute path `/users/myuser/myproject`; the `-o` option will cause the `application-project.yml` file to be written to the /users/myuser/myproject folder:

    detect-advisor -o /users/myuser/myproject

The following command will scan the absolute path `/users/myuser/myproject`; the `-b` option will cause multiple .bdignore files to be created in relevant sub-folders to ignore large duplicate folders or folders containing only binary files. USE WITH CAUTION as it will cause specified folders to be permanently ignored by the Signature scan (until the .bdignore files are removed):

    detect-advisor -b /users/myuser/myproject

The following command will scan the absolute path `/users/myuser/myproject` and output to the myreport.txt file; the `-s` option will cause only Signature scan checking to be performed:

    detect-advisor -s -r myreport.txt /users/myuser/myproject

# SUMMARY INFO

This section includes counts and size analysis for the files and folders beneath the project location.

The `Size Outside Archives` value in the `ALL FILES (Scan Size)` row represents the total scan size as calculated by Detect (used for capacity license).

Note that the `Archives(exc. Jars)` row covers all archive file types but that only .zip files are extracted by `detect_advisor` (whereas Black Duck Detect extracts other types of archives automatically). The final 3 `Inside Archives` columns indicate items found within .zip archives for the different types (except for the Jar row which references .jar/.ear/.war files). The `Inside Archives` columns for the Archives row itself reports archive files within .zips (or nested deeper - zips within zips within zips etc.).

    SUMMARY INFO:
    Total Scan Size = 5,856 MB

                             Num Outside     Size Outside      Num Inside     Size Inside     Size Inside
                                Archives         Archives        Archives        Archives        Archives
                                                            (UNcompressed)    (compressed)
    ====================  ==============   ==============   =============   =============   =============
    Files (exc. Archives)        297,415         4,905 MB         130,126          653 MB          160 MB
    Archives (exc. Jars)              39           951 MB               9            0 MB            0 MB
    ====================  ==============   ==============   =============   =============   =============
    ALL FILES (Scan size)        297,454         5,856 MB         130,135          654 MB          160 MB
    ====================  ==============   ==============   =============   =============   =============
    Folders                       30,435              N/A          10,309             N/A             N/A   
    Ignored Folders                4,169         2,319 MB               0            0 MB            0 MB
    Source Files                 164,240         1,024 MB          53,740          171 MB           34 MB
    JAR Archives                       6             6 MB               0            0 MB            0 MB
    Binary Files                      33            99 MB              10            0 MB            0 MB
    Other Files                  129,476         2,988 MB          75,282          478 MB          124 MB
    Package Mgr Files              3,633            25 MB           1,094            2 MB            0 MB
    OS Package Files                   0             0 MB               0            0 MB            0 MB
    --------------------  --------------   --------------   -------------   -------------   -------------
    Large Files (>5MB)                38           336 MB               1            9 MB            4 MB
    Huge Files (>20MB)                27         1,875 MB               1           35 MB            6 MB
    --------------------  --------------   --------------   -------------   -------------   -------------

    PACKAGE MANAGER CONFIG FILE SUMMARY:
                    MinDepth  MaxDepth    Count   Info
      - PIP                2        11      107   
      - PYTHON             2         9       12   Package Manager missing - (buildless scan supported but not recommended - see recommendations)
      - NPM                2        10       67   
      - MAVEN              2        11       50   
      - GRADLE             2         9       12   
      - PNPM               2        10        3   Package Manager missing
      - SWIFT              3        10        5   

# RECOMMENDATIONS

This section includes a list of findings categorised into CRITICAL (will cause Detect to fail), IMPORTANT (may impact the scope and type of scan) and INFORMATION (potential additional options depending on requirements but which will not impact scope of the standard scan):

    RECOMMENDATIONS:
    - CRITICAL: Overall scan size (6,520 MB) is too large
        Impact:  Scan will fail
        Action:  Ignore folders or remove large files

    -----------------------------------------------------------------------------------------------------    
    - IMPORTANT: No package manager files found in invocation folder but do exist in sub-folders
        Impact:  Dependency scan will not be run
        Action:  Specify --detect.detector.depth=5
        
    - IMPORTANT: Package manager programs (dotnet) missing for package files in sub-folders
        Impact:  The scan will fail if the scan depth is modified from the default
        Action:  Either install package manager programs or
                 consider specifying --detect.accuracy.required=NONE
                 
    - IMPORTANT: Large amount of data (396 MB) in 66 binary files found
        Impact:  Binary files not analysed by standard scan, will impact Capacity license usage
        Action:  Remove files or ignore folders, also consider zipping
                 binary files and using Binary scan
                 
    - IMPORTANT: Large amount of data (594 MB) in 8 duplicate folders
        Impact:  Scan capacity potentially utilised without detecting additional
                 components, will impact Capacity license usage
        Action:  Remove or ignore duplicate folders
        
    - IMPORTANT: Large amount of data (556 MB) in 17 duplicate files
        Impact:  Scan capacity potentially utilised without detecting additional
                 components, will impact Capacity license usage
        Action:  Remove duplicate files or ignore folders
    
    -----------------------------------------------------------------------------------------------------    
    - INFORMATION: Overall number of files (653,981,695) is large
        Impact:  Scan time could be long
        Action:  Ignore folders or split project (scan sub-projects)
        
    - INFORMATION: License or notices files found
        Impact:  Local license text may need to be scanned
        Action:  Add options --detect.blackduck.signature.scanner.license.search=true
                 and optionally --detect.blackduck.signature.scanner.upload.source.mode=true

    - INFORMATION: Source files found for which Snippet analysis supported
        Impact:  Snippet analysis can discover copied OSS source files and functions
        Action:  Add options --detect.blackduck.signature.scanner.snippet.matching=SNIPPET_MATCHING

    - INFORMATION: 5739 singleton .js files found
        Impact:  OSS components within JS files may not be detected
        Action:  Consider specifying Single file matching
                 (--detect.blackduck.signature.scanner.individual.file.matching=SOURCE)

# DETECT CLI

This section includes recommended CLI options for Black Duck Detect. If connectivity to the download locations is not verified by detect_advisor, then proxy options will also be added in case a proxy connection is required.

    DETECT CLI:
    
        DETECT COMMAND:
        bash <(curl -s -L https://detect.blackduck.com/detect10.sh)
    
        MINIMUM REQUIRED OPTIONS:
            --blackduck.url=https://YOURSERVER
            --blackduck.api.token=YOURTOKEN
            --detect.source.path='/Users/myuser/myproject'
            --detect.detector.buildless=true
                (OR install package managers 'dotnet' OR use --detect.XXXX.path=<LOCATION>
                where XXX is package manager)

            (Note that .bdignore exclude file is recommended - see the report file 'repfile.txt' or use '-o' option)


        OPTIONS TO IMPROVE SCAN COVERAGE:
            --detect.detector.depth=2
                (To find package manager files within sub-folders; note depth 2 would find
                all PM files in sub-folders but higher level projects may already include them
            --detect.binary.scan.file.path=binary_files.zip
                (To scan binary files within the project; zip files first - see list of binary
                files in report file; binary scan license required)
            --detect.blackduck.signature.scanner.individual.file.matching=SOURCE
                (To check singleton .js files for OSS matches)


        OPTIONS TO OPTIMIZE DEPENDENCY SCAN:
             For dotnet:
            --detect.nuget.config.path=PATH
                (OPTIONAL The path to the Nuget.Config file to supply to the nuget exe)
            --detect.nuget.packages.repo.url=URL
                (OPTIONAL Nuget Packages Repository URL (default: https://api.nuget.org/v3/index.json).)
            --detect.nuget.excluded.modules=PROJECT
                (OPTIONAL Nuget Projects Excluded: The names of the projects in a solution to exclude.)
            --detect.nuget.ignore.failure=true
                (OPTIONAL Ignore Nuget Failures: If true errors will be logged and then ignored.)
            --detect.nuget.included.modules=PROJECT
                (OPTIONAL Nuget Modules Included: The names of the projects in a solution to include (overrides exclude).)


        PROJECT OPTIONS:
            --detect.project.name=PROJECT_NAME
            --detect.project.version.name=VERSION_NAME
                (Optionally specify project and version names)
            --detect.project.tier=X
                (Optionally define project tier numeric)
            --detect.project.version.phase=ARCHIVED/DEPRECATED/DEVELOPMENT/PLANNING/PRERELEASE/RELEASED
                (Optionally specify project phase - default DEVELOPMENT)
            --detect.project.version.distribution=EXTERNAL/SAAS/INTERNAL/OPENSOURCE
                (Optionally specify version distribution - default EXTERNAL)


# FULL OUTPUT

The --full option will list files found in the project including:

- Large (>10MB) and Huge (>100MB) files
- Binary files
- Singleton JS files
- Package Manager files including the depth they are found at

Note that reported paths which include the characters `##` indicate items within zip or similar archives.

# OUTPUT REPORT FILE

An optional report file can be specified (`-r repfile` or `--report repfile`) which will have the full console output stored.

# OUTPUT CONFIG FILES

The `-o` or `--output_config` option will create the `application-project.yml` files in the project folder if it does not already exist. The `application-project.yml` file will contain a list of commented Detect options from the DETECT CLI section, which you will need to modify and uncomment according to your needs. The `application-project.yml` config file can be used to configure Detect using the single `--spring.profiles.active=project` option.