sat-automations 0.26.5
Contains a collection celery based apps
Requires Python: >=3.9

# SAT Automations

This repository contains a collection of Django apps intended to
run automation tasks that support the operations of SAT.

## Environment Variables

- PEOPLESOFT_PROXY_URL
- CLEARANCE_SERVICE_URL
- AUTH_SERVICE_URL
- CCURE_SERVER
- CCURE_PORT
- CCURE_USERNAME
- CCURE_PASSWORD
- GOLD_DB_USERNAME
- GOLD_DB_PASSWORD
- GOLD_DB_DSN
- FEED_SERVER
- FEED_PORT
- FEED_USERNAME
- FEED_PASSWORD
- SENDGRID_API_KEY
- IMAGE_VOLUME_BASE_PATH
- GRAVITY_FORMS_CONSUMER_KEY
- GRAVITY_FORMS_CONSUMER_SECRET
- GRAVITY_FORMS_BASE_URL

## Installation

## Usage

Add the automations you want to run in your Django APPS

## ACS Journal Automation

An automation that runs hourly to pull in Journal data from the CCURE database.
This is now and should always be a read-only operation.

### Tasks

#### Process Access Control Journal Task

This task runs hourly and pulls in journal data from the CCURE database. It will pull in all journal data starting from the last
record it finds in the local database. It will then process the journal data up to the time that the process was started.

## Demo Automation

This is exactly what it sounds like. It is a demo automation that can be used to test the automation framework.

## Disablement Automation

The disablement automation is designed to run daily and disable users in CCURE that are no longer active in the system.
If a Student or a Faculty/Staff member is no longer active in the system, they should be disabled in CCURE.
The default rules are that when a student is determined inactive they will be disabled in CCURE 180 days from the day they
were marked as inactive in Peoplesoft. For Faculty/Staff, they will be disabled in CCURE 30 days from the day they were marked
as inactive in Peoplesoft.

### Tasks

#### Populate Tracker Task

This task runs daily. It reconciles the CCURE list of users with the list of users from Peoplesoft. If a user comes back as
inactive in Peoplesoft, they will have a disable_on date set in the tracker. This date is used to determine when the user should
be disabled in CCURE.

#### Process Tracker Task

This task runs daily. It will disable users in CCURE that:

1. Are marked as INACTIVE
2. Are not already disabled in CCURE.
3. And have a `disable_on` date that is less than or equal to the current date.

This task can also be triggered as an Action from the Admin interface. In which case it will process the user that have been
selected. It does not do any sanity checks on the users selected, so it is important to only select users that should be disabled.

#### Send Emails Task

This task runs daily. It sends emails to users that are marked for disablement.

Our current process is to send emails to the user and their supervisor (if they have one).

Emails are sent on the following schedule:

1. The day the user is marked for disablement.
2. 7 days before the user is disabled.

## Gold Feed Automation

The gold feed automation is designed to run daily and update the UserFeed table.
Future work will add the information directly to CCURE via the API.

## Housing Automation

The housing automation is designed to run daily and process housing assignments for students.

### Tasks

#### Housing Populate Tracker Task

This task runs daily and pulls in housing assignments and housing revocations from the
`ps_nc_his_bed_proc` view in Peoplesoft.

#### Housing Process Move Ins Task

Runs daily in the morning and processes move ins for students.

#### Housing Process Move Outs Task

Runs daily in the afternoon and processes move outs for students.

## Image Update Automation

An automation that runs daily to pull images from the GOLD database. Currently, this automation just stores the images, but
future work will include adding/updating the image for a user in CCURE.

Matches are performed based on `campus_id`.

### Tasks

#### Daily Image Update Task

Checks gold for images added over the past 1 day, and stores the image on an NFS filesystem. Currently this is mounted at
`/app/images`.

#### All Images Task

Should be run once to populate the image store with all images from the GOLD database.
CAUTION: This task will take a long time to run and should only be run once.

## Affiliate Automation

This automation tracks details of OneCard requests. Stores data in tables for Sponsor, Affiliate, and CardRequest.
Data is gathered and updated on a set schedule.

### Tasks

#### Get Form Entries Task

Gathers all card request forms submitted over a given time interval

#### Save Card Requests Task

Processes the data from the gathered card request forms and persists the data in Sponsor, Affiliate, and CardRequest tables

## Manage Automations

This app is currently used to manage the service accounts in the authentication process for automations.
Generally this would be the place to put models and or processes that are required to manage the automations themselves.

## API

The api app is used to provide a central location for the API endpoints that are used to interact with the automations.

## Development

### Setup

Ensure you are in a virtual environment with Python 3.9.6 or higher.

```shell
> make setup
```

Ensure pre-commit is installed.

```shell
> pre-commit install
```

### Add dependencies

#### Updating Requirements

This project uses `pip-tools` to manage requirements. To update the requirements add your requirement
to the `pyproject.toml` file.

For dependencies required to run the app in production, add them to the `pyproject.toml` file under the `[project]` section.

```toml
[project]
...
dependencies = [
    "fastapi>=0.95.1, <1.0.0",
    "pyjwt>=2.6.0, <3.0.0",
    "...",
    "<YOUR NEW REQUIREMENT HERE>",
    "...",
]
```

For developer dependencies required or nice to have for development, add them to the `pyproject.toml` file under the `[project.optional-dependencies]` section.

```toml
[project.optional-dependencies]
dev = [
    "pytest>=6.2.5, <7.0.0",
    "...",
    "<YOUR NEW DEV REQUIREMENT HERE>",
    "...",
]
```

When you have added the dependency run:

```shell
$> make update-requirements
```

### Create an automation app

To create an automation app execute the following command:

NOTE: Using `test_app` for example, substitute the app name you desire.

```shell
$> python startapp.py test_app
```

This will create a new app in the `sat_automations` directory.
You will need to edit the `apps.py` file and add `sat_automations` to the name.

The below boilerplate should also be in place.

Example:

```python
from logging import DEBUG

from django.apps import AppConfig
from django.conf import settings
from sat.logs import SATLogger

if settings.DEBUG:
    logger = SATLogger(__name__, level=DEBUG)
else:
    logger = SATLogger(__name__)
...
class Automation(AppConfig):
    name = "sat_automations.test_app"
```

## Build and Publish during development

On your branch, you can build and publish by updating the version in `pyproject.toml`
For example, if the version for your feature is say 0.1.0, you would update the version in `pyproject.toml`
to `0.1.0`.

Rules:

- Major features like adding a new app or breaking changes to an existing app require a major version change.
- New feature work to an existing app that will not break existing APIs requires a minor version change.
- Bug fixes and other minor changes require a patch version change.

Large work that should be tested on staging before merging to main should be done in a feature branch, and use
a release candidate version. For example: `1.23.4rc1`. This will allow you to test your work on staging before merging to main.

In django-automation, you would use the just pushed version in it's `pyproject.toml` file.

## CI/CD

When pushing or merging to the main branch, a new build will automatically be generated and published to our PyPI at https://pypi.ehps.ncsu.edu.
This will be based on the updated `pyproject.toml`, so be sure to update it before pushing or merging main. This will also create a new release
on Github which can be found at https://github.ncsu.edu/SAT/sat-automations/releases.

### Build

If you are testing before merging to `main`, you can build the package locally with the following command:

```shell
> git checkout <your-branch>
> git pull
... update the version in pyproject.toml with release candidate version ...
> flit build
```

### Publish

Currently, this is handled via the CI/CD pipeline, but if you need to do it manually, you can do so with the following command:

As long as your PyPI credentials are set up correctly, you can publish to PyPI with the following command:

```shell
> flit publish
```

## Testing

Testing requires a running postgres installation in order to accommodate the triggers and
indexes used in the `acs_journal` app.

Before you run tests locally you will need to make sure it is up and running.

```shell
> make run-db
...
> pytest
```