watchlist-utils 0.0.1
A package for interacting with the Watchlist sources
Requires Python: >=3.6

# DeFi Watchlist

Early stage code for fetching, modifying, and uploading watchlists.

## Install requirements

First, you need to install the dependencies:

Install Python 3.9 in your favorite manner, and create a virtual environment.

```sh
python3 -m venv venv
source ./venv/bin/activate
```

Insert your gitlab personal access token in the `requirements-private.txt file` instead of `ACCESS_TOKEN` (Can be created in preferences -> access tokens)

Then you can tell git not to track changes in this file any longer, so you won't accidentally push any personal tokens over the repository.
[Reference](https://stackoverflow.com/questions/6791889/git-untrack-a-file-in-local-repo-only-and-keep-it-in-the-remote-repo)

```sh
git update-index --assume-unchanged requirements-private.txt
```

Now install the python packages:

```sh
pip install -r requirements.txt
pip install -r requirements-private.txt
```

## Setting Environment Variables

You will need following environment variables set already to run the script:
`SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_USER`, `SNOWFLAKE_PASSWORD`, `TOKENSNIFFER_API_KEY`, `LUABASE_API_KEY`

## GET OFAC Sactioned Digital Currenty Addresses

The following script will fetch the OFAC sacntioned digital currency address list and save it as a JSON file.

But before that you have to manually download the sdn_advanced.xml file from https://www.treasury.gov/ofac/downloads/sanctions/1.0/sdn_advanced.xml and save it in the `./utils/ofac/` directory.(You can use following command)

```shell
wget https://www.treasury.gov/ofac/downloads/sanctions/1.0/sdn_advanced.xml -P ./utils/ofac/
```

Then you can run the following command to get the list of OFAC sanctioned addresses for the given asset.

```shell
./utils/ofac/get_ofac_sanctioned_addresses.py ETH -f JSON
```

### arguments for the command
* `assets`:- The list of Id's from the following choices ["XBT", "ETH", "XMR", "LTC", "ZEC", "DASH", "BTG", "ETC", "BSV", "BCH", "XVG", "USDT", "XRP"]
* `-sdn` :- The path to the sdn_advanced.xml file (can be downloaded from https://www.treasury.gov/ofac/downloads/sanctions/1.0/sdn_advanced.xml) and the default value would be "./utils/ofac/sdn_advanced.xml") 
* `-f` :- The output file format of the address list (default: TXT) (Choice from ["TXT", "JSON", "CSV"])
* `-o` :- The path where the lists should be written to (default: current working directory (".//watchlists/ofac/")



## Normalize watchlist

The following script will fetch data from the data sources given as arguments, and normalize it into a single watchlist.

```shell
./normalize_watchlist.py "__path_to_whitelist_csv__" -ds all
```
### arguments for the command
* `csv` :- file path to the csv file containing the whitelist addresses
* `-ds` :- `,` seperated datasources from where the flagged addresses we fetch. Choices are ["luabase", "tokensniffer", "scamsniffer", "all"]

Running this script will create few csv file in the following folder `./watchlists/__main__/`
The schema is ["address", "utc_time", "meta", "internal_source", "internal_tags", "internal_chain"].

## Uploading watchlist to Snowflake

Please be cautious with the following variable in `snowflake_client.py` file before running `DATABASE = "SOLIDUS_DEFI"`, becasue it is a prod table, you can use `SOLIDUS_DEFI_TEST` which is a test db.

And once CSV was fetched as above, run:

```shell
./upload_watchlist.py csv "__path_to_normalized_watchlist_csv__"
```
### arguments for the command
* `csv` :- file path to the csv file containing the watchlist

## AWS S3

You may want to upload up to date watchlists to s3, or fetch existing ones.

#### Authenticating

To fetch from or upload to s3, you must authenticate with MFA to AWS.

```shell
source ~/mfatoken.sh -r us-east-2 __user_name__ 123456
```

#### Fetching from S3

```shell
aws s3 cp s3://solidus-defi-watchlists/watchlists/ ./watchlists/ --recursive
```

#### Uploading to S3

```shell
aws s3 cp ./watchlists/ s3://solidus-defi-watchlist/watchlists/ --recursive
```