dpk-malware-transform-python 0.2.0
Malware Python Transform
License: Apache-2.0
Requires Python: >=3.10

# Malware Transform 
Please see the set of
[transform project conventions](../../../README.md#Transform-Project-Conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary 
This filter scans the 'contents' column of an input table using [ClamAV](https://www.clamav.net/), and outputs corresponding tables containing 'virus_detection' column (by default).

If a virus is detected, the 'virus_detection' column contains the detected virus signature name; otherwise [null](https://arrow.apache.org/docs/python/generated/pyarrow.null.html).

### Pre-requisites for Mac

For testing and running this transform on local, we are using a unix socket shared with a docker container.
However, docker for mac doesn't support a shared unix socket.
For Mac users, ClamAV will be set up by running `make venv`.
If thet script doesn't work for you, please ensure that you have installed `clamd` command, and it runs with a local unix socket: `/var/run/clamav/clamd.ctl`.

#### Example for manual set up for Mac:

1. Install ClamAV with Homebrew
    ```sh
    brew install clamav
    ```
1. Copy and edit config files.
    ```sh
    cp $(brew --prefix)/etc/clamav/clamd.conf.sample $(brew --prefix)/etc/clamav/clamd.conf
    sed -i '' -e 's/^Example/# Example/' $(brew --prefix)/etc/clamav/clamd.conf
    echo "DatabaseDirectory /var/lib/clamav" >> $(brew --prefix)/etc/clamav/clamd.conf
    echo "LocalSocket /var/run/clamav/clamd.ctl" >> $(brew --prefix)/etc/clamav/clamd.conf
    cp $(brew --prefix)/etc/clamav/freshclam.conf.sample $(brew --prefix)/etc/clamav/freshclam.conf
    sed -i '' -e 's/^Example/# Example/' $(brew --prefix)/etc/clamav/freshclam.conf
    echo "DatabaseDirectory /var/lib/clamav" >> $(brew --prefix)/etc/clamav/freshclam.conf
    ```
1. Create a directory for a local unix socket
    ```sh
    sudo mkdir -p /var/run/clamav
    sudo chown $(id -u):$(id -g) /var/run/clamav
    ```
1. Create a direcotry for a database of ClamAV
    ```sh
    sudo mkdir -p /var/lib/clamav
    sudo chown $(id -u):$(id -g) /var/lib/clamav
    ```
1. Update a database of ClamAV
    ```sh
    freshclam
    ```
1. Edit `venv/bin/activate`, and add following lines to start `clamd` by `source venv/bin/activate`
    ```sh
    if [ ! -e /var/run/clamav/clamd.ctl ]; then
        clamd --config-file=$(brew --prefix)/etc/clamav/clamd.conf
    fi
    ```

## Configuration and Command Line Options

The set of dictionary keys holding [MalwareTransform](src/malware_transform.py) 
configuration for values are as follows:

* _malware_input_column_ - specifies the input column's name to scan. (default: `contents`)
* _malware_output_column_ - specifies the output column's name of the detected virus signature name. (default: `virus_detection`)

## Metadata Fields
As shown in the output of the local run of malware transform, the metadata contains several statistics:
* Global statistics:  
  * `infected`: total number of documents (rows) in which any malwares were detected.   
  * `clean`: total number of documents (rows) in which no malwares were detected.

## Running

### Launched Command Line Options 
The following command line arguments are available in addition to 
the options provided by the [python launcher](../../../../data-processing-lib/doc/python-launcher-options.md)
and the [python launcher](../../../../data-processing-lib/doc/python-launcher-options.md).

```
  --malware_input_column MALWARE_INPUT_COLUMN
                        input column name
  --malware_output_column MALWARE_OUTPUT_COLUMN
                        output column name
```

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/malware_transform_python.py using command line args
* `run-local-sample` - runs src/malware_local.py
* `run-local-python-sample` - runs src/malware_local_python.py

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example, 
```shell
make run-cli-sample
...
```
Then 
```shell
ls output
```
To see results of the transform.