django-ssf-mitre 2.7.2
Shadowserver MITRE ATT&CK and MBC clone
Requires Python: >=3.10

# MITRE ATT&CK

<!-- original development issues: -->
<!-- - https://git.shadowserver.org/contracted-developers/outputcattle/-/issues/1825 -->
<!-- - https://git.shadowserver.org/contracted-developers/outputcattle/-/issues/1910 -->

## Description

The general description as described by MITRE:

> MITRE ATT&CK® is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. The ATT&CK knowledge base is used as a foundation for the development of specific threat models and methodologies in the private sector, in government, and in the cybersecurity product and service community.

> With the creation of ATT&CK, MITRE is fulfilling its mission to solve problems for a safer world — by bringing communities together to develop more effective cybersecurity. ATT&CK is open and available to any person or organization for use at no charge.

We utilize their open data to incorporate their knowledge base into our own. This is done through providing a data structure very similar to what is displayed on [MITRE ATT&CK (attack.mitre.org)](https://attack.mitre.org/).

This project syncs with the open data provided by MITRE in a git repository. We consume that data into database models and slightly modify the human readible data for better represenation (e.g. rewrite internal links). We are then able to use the data for relating objects in our own knowledge base.


## Installation in a Django Project (e.g. outputcattle)

Add `django_ssf_core` & `django_filtering_ui` to the installed apps:

```
INSTALLED_APPS = [
    ...
    "django_ssf_core",
    "django_filtering_ui",
]
```

Include the `GITHUB_ACCESS_TOKEN` in your settings file with either `None` as the value or a GitHub access token. Without a valid access token the synchronization of the data may fail because of API limitations.

Either enable `TEMPLATES['APP_DIRS'] = True` or have the app directories loader enabled (i.e. `django.template.loaders.app_directories.Loader`).

There is an assumption that `jquery` is globally available in some base template. If this is not the case, you'll need to add a jquery script tag to the templates.


## Usage

### Data import procedure

The [MITRE ATT&CK STIX 2.1 formatted data](https://github.com/mitre-attack/attack-stix-data) is the source of the data. We get updates from this source when MITRE pushes a release. This gives us a stable source of the data MITRE ATT&CK produces.

The data is imported using the `ingest_attack_data` management command.

    python manage.py ingest_attack_data

To ingest a specific version (latest is default) use `--version-to-ingest <version>`. For example:

    python manage.py ingest_attack_data --version-to-ingest 9.0

The output is verbose by default. To quiet the output use `--quiet`.

    python manage.py ingest_attack_data --quiet

<details>
<summary> Developer notes (click to expand)</summary>

Acquire the data:

    git clone git@github.com:mitre-attack/attack-stix-data.git
    cd attack-stix-data

The data is arranged by three main catagories that are versioned collections (in the STIX sense).

Start by parsing the collections index:

```python
import json
from pathlib import Path

with Path('index.json').open('r') as fb:
    index = json.load(fb)
```

The catagories are:

```python
print(', '.join([c['name'] for c in index['collections']]))
```

To get a list of paths to our local copy of the data use the following line:

```python
collections = [Path(c['versions'][0]['url'].split('/master/')[-1]) for c in index['collections']]
[x.exists() for x in collections]
```

For version 10.1 these are the object counts by type:

- x-mitre-collection: 1
- attack-pattern: 707
- relationship: 14467
- course-of-action: 284
- identity: 1
- intrusion-set: 136
- malware: 475
- tool: 73
- x-mitre-tactic: 14
- x-mitre-matrix: 1
- x-mitre-data-source: 38
- x-mitre-data-component: 109
- marking-definition: 1

This data was obtained using:

```python
ent = json.load(collections[0].open('r'))
by_type = {t:[y for y in ent['objects'] if y['type'] == t] for t in set([x['type'] for x in ent['objects']])}
{k: len(v) for k, v in by_type.items()}
```

</details>


## Developer guidance

Use `pre-commit` when developing on this project. To use it run: `pip install pre-commit && pre-commit install`. This will help your development by enforcing and correcting code lint and code-style problems on commit.


### Using the Example Project for development and testing

There is an `example_project` directory located in this repository that contains an example django project with the `django_ssf_core` app loaded within it. This can be used for development and testing. The recommendation would be to use it in conjunction with the docker-compose setup (see the `docker-compose.yml` configuration file).

Use the following to start the project:
```
docker compose up
```
Use either `^C` or `docker compose down` to stop the project.

The project is viewable at http://localhost:8000


## Developer notes


### Note on versions

Versions of the data are essentially updates and additions to entries. We can therefore assume with some certainty that relationships built in one version are retained in the next.

Subtraction looks to be done through the use of the STIX common property: `revoked`. This means the data continues to live on, though for all intensive purposes it is abandoned by the author. Our usage of this data has been to append "(revoked)" to the name of the item. The `revoked-by` reference relationship should indicate what if anything has replaced it.

The data contains two identifiers: STIX ID and MITRE ID. These appear to be unchanged between versions. This allows us to consistently sync the data on version changes. However, in practice there is at least one known MITRE ID collision; exceptions have been put in the code to address this.


### Modeling

The Mitre terminology doesn't cleanly match with the STIX terminology. This is an attempt to map that terminology and model the items that are different and/or do not exist.

- Intrusion Set :: Groups: STIX 2.1 `intrusion-set`
  - Most likely an exception to the rule, but `G0058` has a duplicate. It's not a problem as long as we address data marked with `x_mitre_deprecated`, which it is.
- Malware :: Software: STIX 2.1 `malware`
- Tool :: Software: STIX 2.1 `tools`
- _ :: Tactics: Custom `x-mitre-tactic`
  - These are children of a Matrix item
  - These are one parent (in a multi parent relationship) to Techniques
  - These are linked to Attack Patterns / Techniques through the Technique's `kill_chain_phases` (array of mappings keyed by `phase_name` in lowercase). `x_mitre_shortname` is a short identifier used specifically for the stix data. `x_mitre_shortname` on the Tactic matches the value of `phase_name` in the Technique.
- Attack Pattern :: Techniques: STIX 2.1 `attack-pattern`
  - Some of these records have a `revoked` property (e.g. `attack-pattern--9b99b83a-1aac-4e29-b975-b374950551a3`). According to the STIX spec, this means this object is "no longer considered valid by the object creator." Interestly, Mitre has redirects for the IDs associated with these. Though our source of data doesn't show how those redirects would be built.
  - Techniques related to other Techniques as sub-techniques.
- _ :: Data Sources: Custom `x-mitre-data-source` & `x-mitre-data-component` (source is parent, component is child)
- Course of Action :: Mitigations: STIX 2.1 `course-of-action`
- Relationship :: References (integrated into other models): STIX 2.1 `relationship`
- _ :: Matrices: Custom `x-mitre-matrix` (visual catagorization of Tactics to relate/contain techniques)
- Identity :: _ :STIX 2.1 `identity` (only one entry for MITRE)
- _ :: Collection: Custom `x-mitre-collection` used to version a set of data

We do little to nothing with the following typed data:
- `x-mitre-collection`: The versioned collection itself. Mostly used in the import process. Also used as a way to know the currently imported version.
- `identity`: This is mostly used to make the STIX data complete. There is only one record for Mitre.
- `x-mitre-matrix`: Basically a container for `x-mitre-tactic`.
- `marking-definition`: unknown.

### Adding a new model

In the instance of a new data-type appearing in the data you'll need to create a new model for it. There are a few steps that will allow the model to be used during ingestion.

1. Create the model for the new data-type
1. Create the identification pattern for the new data-type and link it to the model

   This involves defining a model pattern pair in `django_ssf_mitre.<app>.utils` and placing that defined pair in the `MATCHABLE_MODEL_PATTERNS` of that module.

1. Register the urls for the index and details views

   Include the model pattern pair in the view module's `VIEWABLE_MODELS_AND_PK_PATTERNS`. This will by default generate an index view for the model. You may need to explicitly define an index view to override base behavior. You will also need to define a detail view for the model.

1. Run the tests and correct errors in `utils` and `views` tests
1. Create the ingestion form
1. Create navigation
1. Run the tests and correct errors


## Developer Debug Notes

<details>
<summary> ... (click to expand)</summary>

List the references for generic content-type:

```python
from django.contrib.contenttypes.models import ContentType
from django_ssf_mitre.attack.models import *
refs = Reference.objects.filter(source_ref_id=41, source_ref_content_type=ContentType.objects.get_for_model(Malware))
refs.count()
[x for x in refs]
```


GitHub PAT:

    export GITHUB_ACCESS_TOKEN=<your-token>
    # Sync the latest version using the specified GitHub token
    python manage.py ingest_attack_data


Print out the Mitre ID that have duplicate records:

```python
from django.db.models import Count
from django_ssf_mitre.attack.models import MitreIdentifiableMixIn
from django_ssf_mitre.attack.models import ALL_MODELS

for model in ALL_MODELS:
    if not issubclass(model, MitreIdentifiableMixIn):
        continue
    for obj in model.objects.values('mitre_id').annotate(count=Count('mitre_id')
).filter(count__gte=2).all():
        print(model.__name__)
        print(f"- {obj}")
```


Find all reference types associated with a model:

```python
from django_ssf_mitre.attack.models import *
rel_target_types = set([])
rel_source_types = set([])
model = Malware
for tech in model.objects.filter(revoked=False, deprecated=False).all():
    for ref in tech.target_refs.all():
        rel_target_types.add(ref.relationship_type)
    for ref in tech.source_refs.all():
        rel_source_types.add(ref.relationship_type)
print(f"target and source relationship types: {rel_target_types} and {rel_source_types}")
```


List each tactic with the number of techniques and techiques with subtechniques:

```python
from django_ssf_mitre.attack.models import *
col = Collection.objects.first()
tactics = Tactic.objects.filter(collection=col).all()
[{tac.name: [tac.techniques.get_active().filter(is_subtechnique=False).count(), {mtec.name: [stec.name for stec in mtec.subtechniques.get_active().all()] for mtec in tac.techniques.get_active().filter(is_subtechnique=False).all()}] for tac in mat.tactic_set.order_by('order_weight').all()} for mat in Matrix.objects.all()]
```

</details>