xnlinkfinder 8.1
A python script to find endpoints from a URL, a file of URLs, a directory of files, a Burp XML file or a ZAP ASCII message file. It also gets potential parameters and a target specific wordlist.

<center><img src="https://github.com/xnl-h4ck3r/xnLinkFinder/blob/main/xnLinkFinder/images/title.png"></center>

## About - v8.1

This is a tool used to discover endpoints, discover potential parameters, generate a target specific wordlist and find secrets for a given target. It can find them by:

- crawling a target (pass a domain/URL)
- crawling multiple targets (pass a file of domains/URLs)
- searching files in a given directory (pass a directory name)
- search a single file's contents
- get them from a **Burp** project (pass location of a Burp XML file)
- get them from an **ZAP** project (pass location of a ZAP ASCII message file)
- get them from a **Caido** project (pass location of a Caido export CSV file)
- get them from a **HAR (HTTP Archive) file** (pass location of a HAR JSON file)
- processing a [waymore](https://github.com/xnl-h4ck3r/waymore) results directory (searching archived response files from `waymore -mode R` and also requesting URLs from `waymore.txt` and the original URLs from `waymore_index.txt` - see [waymore README.md](https://github.com/xnl-h4ck3r/waymore/blob/main/README.md))

The python script is based on the link finding capabilities of my Burp extension [GAP](https://github.com/xnl-h4ck3r/burp-extensions).
As a starting point, I took the amazing tool [LinkFinder](https://github.com/GerbenJavado/LinkFinder) by Gerben Javado, and used the Regex for finding links, but with additional improvements to find even more.

## Installation

`xnLinkFinder` supports **Python 3**.

Install `xnLinkFinder` in default (global) python environment.

```bash
pip install xnLinkFinder
```

OR

```bash
pip install git+https://github.com/xnl-h4ck3r/xnLinkFinder.git -v
```

You can upgrade with

```bash
pip install --upgrade xnLinkFinder
```

### pipx

Quick setup in isolated python environment using [pipx](https://pypa.github.io/pipx/)

```bash
pipx install git+https://github.com/xnl-h4ck3r/xnLinkFinder.git
```

**NOTE: To extract links from PDF files, you can install `pdftotext` on your system. You can install it using `sudo apt install -y poppler-utils` on Linux or `brew install poppler` on macOS. If you don't have `pdftotext` installed, `xnLinkFinder` will fall back to using the `pypdf` Python library. If no text is found (e.g. for scanned PDFs), it will attempt to use `ocrmypdf` if installed (e.g. `sudo apt install -y ocrmypdf`). Using `pdftotext` and `ocrmypdf` will generally give much better results than `pypdf` alone.**

## Usage

| Arg         | Long Arg                   | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ----------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| -i          | --input                    | Input a: URL, text file of URLs, a Directory of files to search, a Burp XML output file, a ZAP output file, a Caido CSV file, a HAR JSON file, or a single file to search contents.                                                                                                                                                                                                                                                                                                                                   |
| -o          | --output                   | The file to save the Links output to, including path if necessary (default: output.txt). If set to `cli` then output is only written to STDOUT. If the file already exist it will just be appended to (and de-duplicated) unless option `-ow` is passed.                                                                                                                                                                                                                                           |
| -op         | --output-params            | The file to save the Potential Parameters output to, including path if necessary (default: parameters.txt). If set to `cli` then output is only written to STDOUT (but not piped to another program). If the file already exist it will just be appended to (and de-duplicated) unless option `-ow` is passed.                                                                                                                                                                                     |
| -owl        | --output-wordlist          | The file to save the target specific Wordlist output to, including path if necessary (default: No wordlist output). If set to `cli` then output is only written to STDOUT (but not piped to another program). If the file already exist it will just be appended to (and de-duplicated) unless option -ow is passed.                                                                                                                                                                               |
| -oo         | --output-oos               | The file to save the out of scope links output to, including path if necessary (default: No OOS output). If set to `cli` then output is only written to STDOUT (but not piped to another program). If the file already exist it will just be appended to (and de-duplicated) unless option -ow is passed.                                                                                                                                                                                     |
| -os         | --output-secrets           | The file to save any secrets found in responses to (JSON format), including path if necessary (default: No secrets output). If the file already exists, secrets will be merged unless option -ow is passed. Secrets are grouped by type and value with an array of sources where each was found.                                                                                          |
| -ow         | --output-overwrite         | If the output file already exists, it will be overwritten instead of being appended to.                                                                                                                                                                                                                                                                                                                                                                             |
| -sp         | --scope-prefix             | Any links found starting with `/` will be prefixed with scope domains in the output instead of the original link. If the passed value is a valid file name, that file will be used, otherwise the string literal will be used.                                                                                                                                                                                                                                                                     |
| -spo        | --scope-prefix-original    | If argument `-sp` is passed, then this determines whether the original link starting with `/` is also included in the output (default: false)                                                                                                                                                                                                                                                                                                                                                      |
| -spkf       | --scope-prefix-keep-failed | If argument `-spkf` is passed, then this determines whether a prefixed link will be kept in the output if it was a 404 or a RequestException occurs (default: false)                                                                                                                                                                                                                                                                                                                               |
| -sf         | --scope-filter             | Will filter output links to only include them if the domain of the link is in the scope specified. If the passed value is a valid file name, that file will be used, otherwise the string literal will be used. This argument is now mandatory if input is a domain/URL (or file of domains/URLs) to prevent crawling sites that are not in scope and also preventing misleading results.                                                                                                          |
| -c          | --cookies †                | Add cookies to pass with HTTP requests. Pass in the format `'name1=value1; name2=value2;'`                                                                                                                                                                                                                                                                                                                                                                                                         |
| -H          | --headers †                | Add custom headers to pass with HTTP requests. Pass in the format `'Header1: value1; Header2: value2;'`                                                                                                                                                                                                                                                                                                                                                                                            |
| -ra         | --regex-after              | RegEx for filtering purposes against found endpoints before output (e.g. `/api/v[0-9]\.[0-9]\*` ). If it matches, the link is output.                                                                                                                                                                                                                                                                                                                                                              |
| -d          | --depth †                  | The level of depth to search. For example, if a value of 2 is passed, then all links initially found will then be searched for more links (default: 1). This option is ignored for Burp files, ZAP files, Caido files, and HAR files because they can be huge and consume lots of memory. It is also advisable to use the `-sp` (`--scope-prefix`) argument to ensure a request to links found without a domain can be attempted.                                                                              |
| -p          | --processes †              | Basic multithreading is done when getting requests for a URL, or file of URLs (not a Burp file, ZAP file, Caido file, or HAR file). This argument determines the number of processes (threads) used (default: 25)                                                                                                                                                                                                                                                                                             |
| -rl         | --rate-limit †             | Maximum number of requests to send per second (default: 0, no rate limiting). This helps prevent overwhelming the target server.                                                                                                                                                                                                                                                                                                                                                                   |
| -x          | --exclude                  | Additional Link exclusions (to the list in `config.yml`) in a comma separated list, e.g. `careers,forum`                                                                                                                                                                                                                                                                                                                                                                                           |
| -orig       | --origin                   | Whether you want the origin of the link to be in the output. Displayed as `LINK-URL [ORIGIN-URL]` in the output (default: false)                                                                                                                                                                                                                                                                                                                                                                   |
| -prefixed   |                            | Whether you want to see which links were prefixed in the output. Displays `(PREFIXED)` after link and origin in the output (default: false)                                                                                                                                                                                                                                                                                                                                                        |
| -xrel       | --exclude-relative-links   | By default, if any links in the results start with `./` or `../`, they will be included. If this argument is used, these relative links will not be added.                                                                                                                                                                                                                                                                                                                                         |
| -t          | --timeout †                | How many seconds to wait for the server to send data before giving up (default: 10 seconds)                                                                                                                                                                                                                                                                                                                                                                                                        |
| -r          | --retries †                | The number of times to retry a request after a failure (e.g. timeout, connection error, etc). This is limited to 5 retries (default: 0)                                                                                                                                                                                                                                                                                                                                                            |
| -inc        | --include                  | Include input (`-i`) links in the output (default: false)                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| -u          | --user-agent †             | What User Agents to get links for, e.g. `-u desktop mobile`. Possible values are `desktop`, `mobile`, `set-top-boxes` and `game-console`. Also there are `mobile-apple`, `mobile-android` and `mobile-windows` that are subsets of `mobile` but can be used separately.                                                                                                                                                                                                                            |
| -uc         | --user-agent-custom †      | A custom User Agent string to use for all requests. This will override the `-u`/`--user-agent` argument. This can be used when a program requires a specific User Agent header to identify you for example.                                                                                                                                                                                                                                                                                        |
| -insecure   | †                          | Whether TLS certificate checks should be disabled when making requests (delfault: false)                                                                                                                                                                                                                                                                                                                                                                                                           |
| -s429       | †                          | Stop when > 95 percent of responses return 429 Too Many Requests (default: false)                                                                                                                                                                                                                                                                                                                                                                                                                  |
| -s403       | †                          | Stop when > 95 percent of responses return 403 Forbidden (default: false)                                                                                                                                                                                                                                                                                                                                                                                                                          |
| -sTO        | †                          | Stop when > 95 percent of requests time out (default: false)                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| -sCE        | †                          | Stop when > 95 percent of requests have connection errors (default: false)                                                                                                                                                                                                                                                                                                                                                                                                                         |
| -m          | --memory-threshold         | The memory threshold percentage. If the machines memory goes above the threshold, the program will be stopped and ended gracefully before running out of memory (default: 95)                                                                                                                                                                                                                                                                                                                      |
| -mfs        | --max-file-size †          | The maximum file size (in bytes) of a file to be checked if -i is a directory. If the file size is over, it will be ignored (default: 500 MB). Setting to 0 means no files will be ignored, regardless of size.                                                                                                                                                                                                                                                                                    |
| -mrs        | --max-response-size †      | The maximum response size (in MB) to download. Responses larger than this will be skipped (default: 100 MB).                                                                                                                                                                                                                                                                                                                                                                                               |
| -fp         | --forward-proxy †          | For active link finding with URL (or file of URLs), replay the requests through a proxy such as Burp or Caido, e.g `http://127.0.0.1:8080`.                                                                                                                                                                                                                                                                                                                                                                        |
| -rp         | --request-proxy            | Request proxy to use. Can be a proxy string (e.g. `http://user:pass@1.2.3.4:8000`, `socks5://host:port`) or a file containing proxy list (one per line, random selection).                                                                                                                                                                        |
|             | --heap                     | Whether to take a heap snapshot of visited URLs and extract links from browser memory. This will take longer but could discover dynamically generated links that standard static analysis might miss (Requires Playwright to be installed).                                                                                                                                                                                                                                                                                                                                                                                                         |
| -ro         | --readable-only            | If passed, only human-readable text will be extracted from HTML responses/files using BeautifulSoup before searching for links. This excludes content from `<script>`, `<style>`, `<noscript>`, and `<template>` tags.                                                                                                                                                                                                                                                                                                                                                                                                                              |
| -ascii-only |                            | Whether links and parameters will only be added if they only contain ASCII characters. This can be useful when you know the target is likely to use ASCII characters and you also get a number of false positives from binary files for some reason.                                                                                                                                                                                                                                               |
| -mtl        | --max-time-limit           | The maximum time limit (in minutes) to run before stopping (default: 0). If 0 is passed, there is no limit.                                                                                                                                                                                                                                                                                                                                                                                        |
|             | --config                   | Path to the YML config file. If not passed, it looks for file `config.yml` in the default directory, typically `~/.config/xnLinkFinder`.                                                                                                                                                                                                                                                                                                                                                           |
| -nwlpl      | --no-wordlist-plurals      | When words are found for a target specific wordlist, by default new words are added if there is a singular word from a plural, and vice versa. If this argument is used, this process is not done.                                                                                                                                                                                                                                                                                                 |
| -nwlpw      | --no-wordlist-pathwords    | By default, any path words found in the links will be processed for the target specific wordlist. If this argument is used, they will not be processed. **NOTE: if the YAML config value of `respParamPathWords` is `True` then this argument will not have any effect unless `-nwlpm`/`--no-wordlist-parameters` is also passed.**                                                                                                                                                                |
| -nwlpm      | --no-wordlist-parameters   | By default, any parameters found in the links will be processed for the target specific wordlist. If this argument is used, they will not be processed.                                                                                                                                                                                                                                                                                                                                            |
| -nwlc       | --no-wordlist-comments     | By default, any comments in pages will be processed for the target specific wordlist. If this argument is used, they will not be processed.                                                                                                                                                                                                                                                                                                                                                        |
| -nwlia      | --no-wordlist-imgalt       | By default, any image 'alt' attributes will be processed for the target specific wordlist. If this argument is used, they will not be processed.                                                                                                                                                                                                                                                                                                                                                   |
| -nwld       | --no-wordlist-digits       | Exclude any words from the target specific wordlist with numerical digits in.                                                                                                                                                                                                                                                                                                                                                                                                                      |
| -nwll       | --no-wordlist-lowercase    | By default, any word added with any uppercase characters in will also add the word in lowercase. If this argument is used, the lowercase words will not be added.                                                                                                                                                                                                                                                                                                                                  |
| -wlml       | --wordlist-maxlen          | The maximum length of words to add to the target specific wordlist, excluding plurals (default: 0 - no limit)                                                                                                                                                                                                                                                                                                                                                                                      |
| -swf        | --stopwords-file           | A file of additional Stop Words (in addition to "stopWords" in the YML Config file) used to exclude words from the target specific wordlist. Stop Words are used in Natural Language Processing and different lists can be found in different libraries. You may want to add words in different languages, depending on your target.                                                                                                                                                               |
| -brt        | --burpfile-remove-tags     | When the input passed with `-i` is a Burp file, the user is asked interactively whether they want to remove unnecessary tags from that file (sometimes there is a problem in Burp XML files that can often be resolved by removing unnecessary tags which will also make the file smaller). If you are using xnLinkFinder in a script, you don't want to break for user input, so you can set that by passing this argument with a `true` or `false`. NOTE: This is a permanent change to the file |
| -all        | --all-tlds                 | All links found will be returned, even if the TLD is not common. This can result in a number of false positives where variable names, etc. may also be a possible genuine domain. By default, only links that have a TLD in the common TLDs (`commonTLDs` in `config.yml`) will be returned.                                                                                                                                                                                                       |
| -cl         | --content-length           | Show the Content-Length of the response when crawling.                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| -nb         | --no-banner                | Hides the tool banner.                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| -v          | --verbose                  | Verbose output                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| -vv         | --vverbose                 | Increased verbose output                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
|             | --version                  | Show current version number.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| -h          | --help                     | show the help message and exit                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

† NOT RELEVANT FOR INPUT OF DIRECTORY, BURP XML FILE, ZAP FILE, CAIDO CSV FILE, OR HAR FILE

## config.yml

The `config.yml` file (typically in `~/.config/xnLinkFinder/`) has the keys which can be updated to suit your needs:

- `linkExclude` - A comma separated list of strings (e.g. `.css,.jpg,.jpeg` etc.) that all links are checked against. If a link includes any of the strings then it will be excluded from the output. If the input is a directory, then file names are checked against this list.
- `contentExclude` - A comma separated list of strings (e.g. `text/css,image/jpeg,image/jpg` etc.) that all responses `Content-Type` headers are checked against. Any responses with the these content types will be excluded and not checked for links.
- `fileExtExclude` - A comma separated list of strings (e.g. `.zip,.gz,.tar` etc.) that all files in Directory mode are checked against. If a file has one of those extensions it will not be searched for links. Also, in normal mode, if a response doesn't have a content-type to check for exclusions, it will check for these extensions at the end of the URL to determine if to search for links.
- `regexFiles` - A list of file types separated by a pipe character (e.g. `php|php3|php5` etc.). These are used in the Link Finding Regex when there are findings that aren't obvious links, but are interesting file types that you want to pick out. If you add to this list, ensure you escape any dots to ensure correct regex, e.g. `js\.map`
- `respParamLinksFound` † - Whether to get potential parameters from links found in responses: `True` or `False`
- `respParamPathWords` † - Whether to add path words in retrieved links as potential parameters: `True` or `False`
- `respParamJSON` † - If the MIME type of the response contains JSON, whether to add JSON Key values as potential parameters: `True` or `False`
- `respParamJSVars` † - Whether javascript variables set with `var`, `let` or `const` are added as potential parameters: `True` or `False`
- `respParamXML` † - If the MIME type of the response contains XML, whether to add XML attributes values as potential parameters: `True` or `False`
- `respParamInputField` † - If the MIME type of the response contains HTML or JAVASCRIPT (because HTML could be built in HTML), whether to add NAME and ID attributes of any INPUT (or TEXTAREA/SELECT/BUTTON) fields as potential parameters: `True` or `False`
- `respParamMetaName` † - If the MIME type of the response contains HTML, whether to add NAME attributes of any META tags as potential parameters: `True` or `False`
- `wordsContentTypes` - A comma separated list of strings (e.g. `text/html,text/plain`) to specify which response content types will be searched for words to go in the target specific wordlist.
- `stopWords` - A comma separated list of strings (e.g. `then,this,that`) to specify words that are excluded from the target specific wordlist. This default list is initially made up of English determiners, coordinating conjuctions and prepositions, plus a list of stop words from Scikit-Learn, a python machine learning library.
- `commonTLDs` - A comma separated list of the most common TLDs. Unless `-all`/`--all-tlds` argument is passed, only links with domains that have a TLD in this list are returned.

† IF THESE ARE NOT FOUND IN THE CONFIG FILE THEY WILL DEFAULT TO `True`

## Examples

### Find Links from a specific target - Basic

```
xnLinkFinder -i target.com -sf target.com
```

### Find Links from a specific target - Detailed

Ideally, provide scope prefix (`-sp`) with the primary domain (including schema), and a scope filter (`-sf`) to filter the results only to relevant domains (this can be a file or in scope domains). Also, you can pass cookies and customer headers to ensure you find links only available to authorised users.
Specifying the User Agent (`-u desktop mobile`) will first search for all links using desktop User Agents, and then try again using mobile user agents. There could be specific endpoints that are related to the user agent given. Giving a depth value (`-d`) will keep sending request to links found on the previous depth search to find more links.

```
xnLinkFinder -i target.com -sp target_prefix.txt -sf target_scope.txt -spo -inc -vv -H 'Authorization: Bearer XXXXXXXXXXXXXX' -c 'SessionId=MYSESSIONID' -u desktop mobile -d 10
```

### Find Links from a list of URLs - Basic

If you have a file of JS file URLs for example, you can look for links in those:

```
xnLinkFinder -i target_js.txt -sf target.com
```

NOTE: A passed file is assumed to be a list of URLs if the first line starts with `//` or `http` or is a format of a domain (e.g. `example.com`, `sub.example.co.uk`), otherwise it is considered to be a file to search the contents for (unless it is a Burp, Zap, Caido, or HAR file).

### Find Links from the contents of a file - Basic

If you have a saved response for example, you can look for links in those:

```
xnLinkFinder -i response.txt -sf target.com
```

NOTE: A passed file is assumed to be a list of URLs if the first line starts with `//` or `http` or is a format of a domain (e.g. `example.com`, `sub.example.co.uk`), otherwise it is considered to be a file to search the contents for (unless it is a Burp, Zap, Caido, or HAR file).

### Find Links from a files in a directory - Basic

If you have a files, e.g. JS files, HTTP responses, etc. you can look for links in those:

```
xnLinkFinder -i ~/.config/waymore/results/target.com
```

NOTE: Sub directories are also checked. The `-mfs` option can be specified to skip files over a certain size.

### Fi