bayse-tools 1.0.5
A collection of tools useful across the Bayse ecosystem.
Requires Python: >=3.9

# Bayse Tools
The `bayse-tools` package provides functionality that allows users to convert a growing number of network flow formats 
into the lightweight BayseFlow format. Doing so offers the ability to interact with the Bayse labeling and 
knowledgebase functionality, which provides detailed insights about how your systems are communicating with the 
Internet and your internal devices. For more information about use cases, explore our 
[Use Cases](https://bayse.io/resources/use-cases) online.  

## Installation
Note that this package requires libpcap-dev to be installed on your system. Please use your system's package manager
(such as apt on Ubuntu) to install libpcap-dev:

`sudo apt-get install libpcap-dev`

While Windows isn't yet supported due to issues with underlying libraries (specifically pcapy), we'd welcome anyone 
who wants to document the steps to make it work. At the very minimum, you will need the following:

* A C++ compiler. Microsoft Visual Studio Build Tools is known to work.
* Npcap's SDK, which is a replacement for the WinPCAP developer's kit.

To install the bayse-tools package, simply type the following:

`pip install bayse-tools`


## Available Modules
There are two modules available within this package. Details about each are below.

### A. Standalone Converter
module name: `converter`

This module allows you to convert captured network traffic from any of our [supported formats](#supported-file-formats) 
into unlabeled 
BayseFlows (BayseFlows without BayseFlow category labels). This is useful if you 
already have network telemetry that you'd like to enrich with our knowledgebase and labeling.

To import this module into your project, type the following:

`from bayse_tools.converter import convert`

#### Supported File Formats
Bayse currently supports conversion into BayseFlow format from the following formats:

| Format         | Specific File Types                                |
|----------------|----------------------------------------------------|
 | Packet Capture | `CAP`, `PCAP`, and `PCAPNG`                        |
 | Zeek           | `conn.log` and `dns.log` in `TSV` or `JSON` format |
 | Interflow      | comma-separated list of `JSON` records             |
| Netflow        | `CSV` of unidirectional Netflow v9 records           |

`?` If you have a format that you'd like us to support, please contact support at bayse [.] io.

#### Usage
To convert a Packet Capture into an unlabeled BayseFlow file, simply enter the following:

`convert.convert_pcap(<path_to_packet_capture>)`

To convert a Zeek `conn.log` into an unlabeled BayseFlow file, simply enter the following:

`convert.convert_zeek(<path_to_conn_log>, zeek_dnsfile_location=<optional_path_to_dns_log>)`

If you have (and would like to include) DNS information that was captured by Zeek, provide the `dns.log` in addition 
to the `conn.log`. Naming will be much enhanced by doing so.

To convert an Interflow log into an unlabeled BayseFlow file, simply enter the following:

`convert.convert_interflow(<path_to_interflow_log>)`

Any DNS Interflows should also be passed in within the Interflow log.

Note that there are MANY fields in Interflow that will not be a part of this converter, as they are not necessary to 
convert to BayseFlow. The log file is expected to contain a comma-separated list of Interflow records in JSON format.
For non-DNS records, below is an example Interflow record that includes only the fields we need:

    [{"timestamp": 1656517273641, "duration": 401, "_id": "6c0liABC8qtQm3loQr7H", "msg_class": "interflow_traffic", "srcip": "172.18.40.120", "srcport": 55503,"dstip": "142.251.40.65", "dstip_host": "ci3.googleusercontent.com", "dstport": 80, "proto_name": "tcp", "outbytes_total": 0, "inpkts_delta": 5, "outpkts_delta": 0, "inbytes_total": 17765}]

To convert a Netflow CSV into an unlabeled BayseFlow file, simply enter the following:

`convert.convert_netflow(<path_to_Netflow_csv>)`

Similar to Interflow, there are MANY fields in Netflow v9 that are not used.


#### Plaintext Formats

Zeek and Interflow data is supported as JSON files in two formats:

1. Comma-Separated List of JSON records (**preferred**):

`[{json_record_1},{json_record_2}]`

2. One JSON record per line:

```
{json_record_1}
{json_record_2}
```

Moreover, Zeek also supports plaintext `TSV` (i.e. the OG). If you are converting plaintext logs, please make sure 
that your `conn.log` files have the following fields in the following order (Field # lines are annotations for 
clarity below and should **NOT** be included in the file):

```     
        Field #  1       2        3                 4               5
                ts      uid     id.orig_h       id.orig_p       id.resp_h
        Field #    6             7        8        9               10
                id.resp_p       proto   service duration        orig_bytes
        Field #     11              12              13              14
                resp_bytes      conn_state      local_orig      local_resp
        Field #     15             16      17               18             19
                missed_bytes    history orig_pkts       orig_ip_bytes  resp_pkts
        Field #       20              21
                resp_ip_bytes   tunnel_parents
```

An example of a flow line from the `conn.log` is as follows ("field #" lines are our annotation):
```
        Field #       1                        2                      3
                1601060272.439360       CC9S3G178KjzSMTGRk      192.168.100.224
        Field #   4             5        6       7       8          9
                 137    192.168.100.255 137     udp     dns     12.114023
        Field #  10     11    12      13     14      15      16       17
                1186    0     S0      -       -       0       D       23
        Field #  18     19      20      21
                1830    0       0       -
```

For dns.log files (again, if you're not using JSON), make sure that the file matches the following:
```
        Field #       1       2          3               4               5               6             7         8
                      ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       proto   trans_id
        Field #       9       10        11               12             13         14           15        16
                     rtt     query    qclass         qclass_name       qtype   qtype_name      rcode   rcode_name
        Field #      17      18      19      20      21        22         23        24
                     AA      TC      RD      RA      Z       answers     TTLs    rejected
```

Note that all field names are not used. If you do not have a field, please give it an appropriate default
value as defined by the [Zeek format](https://docs.zeek.org/en/master/logs/dns.html).


### B. Streaming BayseFlow Collector
module name: `streaming`

This module allows you to directly capture network traffic as unlabeled BayseFlows (BayseFlows without flow category 
labels). This is beneficial for a number of reasons:

BayseFlows are extremely lightweight, so you can actually do this continuously in the background without affecting 
system performance (i.e. you can collect network telemetry continuously on your endpoints)! BayseFlows have no 
identifying data, so you can avoid worrying about accidentally leaking information (URIs, passwords, keys, etc...)

To import this module into your project, type the following:

`from bayse_tools.streaming import streaming`

#### Usage
To capture BayseFlows continuously using this module from your project, enter the following (note that you'll need
to be root by default to capture packets):

`streaming.start(<interface_name>, <duration_in_seconds>, <optional_verbosity>)`

Note that the above will run in perpetuity, capturing packets from the interface you specified (such as *enp0s3* on
Ubuntu systems). Every time the specified duration you provided is reached (if you provide no value, it defaults to
300 seconds), a sample will be created, processed, and summarized. Providing the optional `is_verbose` value will 
print a small amount of information about the number of BayseFlows and its UUID.

## BayseFlow Format
The BayseFlow format is saved as a `.bf` file and is comprised of records containing that look like the following:

```
{
  "hash": "6d8091a34ae3427bbffa5c19e5d3391c",
  "trafficDate": "1654204899.899094",
  "fileName": "testCase24_ipv6_withDNS_and_icmpv6.conn.bf",
  "BayseFlows": [
    {
      "src": "192.168.12.164:62239",
      "dst": "9.9.9.9:53",
      "destinationNameSource": "original",
      "srcPkts": 1,
      "srcBytes": 32,
      "dstPkts": 1,
      "dstBytes": 155,
      "relativeStart": 0.0,
      "protocolInformation": "UDP",
      "identifier": "CvJxCegvPKSYQpTeh",
      "duration": 0.055491
    },
    {
      "src": "2607:fb90:d726:7b19:5cc5:a122:d8ae:24e6:58740",
      "dst": "r3.o.lencr.org:80",
      "destinationNameSource": "passive",
      "srcPkts": 84,
      "srcBytes": 14526,
      "dstPkts": 47,
      "dstBytes": 35524,
      "relativeStart": 0.112856,
      "protocolInformation": "TCP",
      "identifier": "COVJH14OZAUWEf1Vd",
      "duration": 48.68848
    },
    ...
  ]
}
```

Details about each field are as follows:

| Field Name                | Purpose                                                                                      | Derived How                                                                                                                                                                                                                          |
|---------------------------|----------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | hash                      | helps to identify the BayseFlow file                                                         | randomly-generated                                                                                                                                                                                                                   |
| trafficDate               | identifies the time (as UTC epoch time in seconds.ms) the traffic capture started            | timestamps from initial file                                                                                                                                                                                                         |
| fileName                  | name of the file                                                                             | based on name of input file                                                                                                                                                                                                          |
| BayseFlows                | mapping of each flow to the BayseFlow format (fields below here are subfields of BayseFlows) | Combines packets/flows based on their 4-tuples                                                                                                                                                                                       |
| --> src                   | identify the source of the flow                                                              | attempts to identify the true source based on how the flow is communicating; will add a DNS name if known                                                                                                                            |
| --> dst                   | identify the destination of the flow                                                         | attempts to identify the true destination based on how the session is communicating; will add a DNS name if known                                                                                                                    |
| --> destinationNameSource | identify how the destination was named                                                       | names are derived (in order) by direct knowledge of the name (in the flow; `session`), the closest DNS lookup (`passive`), a short-term local cache (`cache`), a long-term local cache (`cache`), or they stay as an IP (`original`) |
| --> srcPkts               | number of packets from the source                                                            | counts how many packets seen in the input file from this source in this flow                                                                                                                                                         |
| --> srcBytes              | number of bytes from the source                                                              | counts how many bytes were seen in the transport layer payloads (or ICMP data) from the source in this flow                                                                                                                          | 
| --> dstPkts               | number of packets from the destination                                                       | counts how many packets seen in the input file from this destination in this flow                                                                                                                                                    |
| --> dstBytes              | number of bytes from the destination                                                         | counts how many bytes were seen in the transport layer payloads (or ICMP data) from the destination in this flow                                                                                                                     | 
| --> relativeStart         | how far into this file this flow began                                                       | time (as seconds.ms) from `trafficDate`                                                                                                                                                                                              | 
| --> protocolInformation   | communicate any information about the transport layer protocol (or ICMP) seen                | transport layer reported by network layer                                                                                                                                                                                            |
| --> identifier            | a way to tie the BayseFlow back to a flow in the original input format                       | unique_id for Zeek, TCP/UDP stream for Packet Captures                                                                                                                                                                               |
| --> duration              | identify how long (seconds.ms) a flow lasted                                                 | start of first packet to end of last packet in flow                                                                                                                                                                                  |


## Labeling
BayseFlows can also be labeled with information about characteristics of the flow that are interesting. These labels 
are useful for identifying activity that is (ab)normal, understanding how data is flowing locally or to/from the 
Internet, and so on. Doing so adds the `label` field to the `.bf` file. Take the following flow for example:

```
    {
      "src": "192.168.1.133:59893",
      "dst": "54.37.70.105:8080",
      "destinationNameSource": "original",
      "srcPkts": 218,
      "srcBytes": 1540,
      "dstPkts": 1801,
      "dstBytes": 2497139,
      "relativeStart": 17.916785,
      "protocolInformation": "TCP",
      "identifier": "115",
      "duration": 5.62408,
      "label": "floodOfXLFilelikeDownloaded"
    }
```

The `floodOfXLFilelikeDownloaded` shows us that something that looks like a large file was downloaded in just 
seconds. Depending on whether or not that activity is expected from that destination can help you to determine if 
you should care.

To label your data, you'll first need to register for an account on [Bayse](https://www.bayse.io/register), request 
a free API key, and reach out to hello at bayse [.] io. We'd love to help you get started!

## Troubleshooting

If you are having issues installing `bayse-tools`, here are some common issues:
* Your system does not have libpcap-dev installed. Please refer to the [Installation](#installation) steps above for 
  details.
* You have `pcapy` installed and are trying to use Python 3.10 or greater. In v1.0.1 of `bayse-tools`, we have moved 
  away from `pcapy` in favor of `pcapy-ng`. Please make sure to uninstall the former if it exists.

If nothing above helps or you have other questions, please reach out to us at support at bayse [.] io

## License
This software is provided under the MIT Software License. See the accompanying LICENSE file for more information.