dbt-glue 1.10.19
dbt adapter for AWS Glue
Requires Python: >=3.9

<p align="center">
  <img src="/etc/dbt-logo-full.svg" alt="dbt logo" width="500"/>
</p>

**[dbt](https://www.getdbt.com/)** enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.

# dbt-glue

The `dbt-glue` package implements the [dbt adapter](https://docs.getdbt.com/docs/contributing/building-a-new-adapter) protocol for AWS Glue's Spark engine. 
It supports running dbt against Spark, through the new Glue Interactive Sessions API.

To learn how to deploy a data pipeline in your modern data platform using the `dbt-glue` adapter, please read the following blog post: [Build your data pipeline in your AWS modern data platform using AWS Lake Formation, AWS Glue, and dbt Core](https://aws.amazon.com/blogs/big-data/build-your-data-pipeline-in-your-aws-modern-data-platform-using-aws-lake-formation-aws-glue-and-dbt-core/)

## Installation

The package can be installed from PyPI with:

```bash
$ pip3 install dbt-glue
```
For further (and more likely up-to-date) info, see the [README](https://github.com/aws-samples/dbt-glue#readme)


## Connection Methods


### Configuring your AWS profile for Glue Interactive Session
There are two IAM principals used with interactive sessions.
- Client principal: The princpal (either user or role) calling the AWS APIs (Glue, Lake Formation, Interactive Sessions)
from the local client. This is the principal configured in the AWS CLI and likely the same.
- Service role: The IAM role that AWS Glue uses to execute your session. This is the same as AWS Glue
ETL.

Read [this documentation](https://docs.aws.amazon.com/glue/latest/dg/glue-is-security.html) to configure these principals.

You will find bellow a least privileged policy to enjoy all features of **`dbt-glue`** adapter.

Please to update variables between **`<>`**, here are explanations of these arguments:

|Args	|Description	| 
|---|---|
|region|The region where your Glue database is stored |
|AWS Account|The AWS account where you run your pipeline|
|dbt output database|The database updated by dbt (this is the schema configured in the profile.yml of your dbt environment)|
|dbt source database|All databases used as source|
|dbt output bucket|The bucket name where the data will be generate dbt (the location configured in the profile.yml of your dbt environment)|
|dbt source bucket|The bucket name of source databases (if they are not managed by Lake Formation)|

```yaml
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Read_and_write_databases",
            "Action": [
                "glue:SearchTables",
                "glue:BatchCreatePartition",
                "glue:CreatePartitionIndex",
                "glue:DeleteDatabase",
                "glue:GetTableVersions",
                "glue:GetPartitions",
                "glue:DeleteTableVersion",
                "glue:UpdateTable",
                "glue:DeleteTable",
                "glue:DeletePartitionIndex",
                "glue:GetTableVersion",
                "glue:UpdateColumnStatisticsForTable",
                "glue:CreatePartition",
                "glue:UpdateDatabase",
                "glue:CreateTable",
                "glue:GetTables",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:GetDatabase",
                "glue:GetPartition",
                "glue:UpdateColumnStatisticsForPartition",
                "glue:CreateDatabase",
                "glue:BatchDeleteTableVersion",
                "glue:BatchDeleteTable",
                "glue:DeletePartition",
                "glue:GetUserDefinedFunctions",
                "lakeformation:ListResources",
                "lakeformation:BatchGrantPermissions",
                "lakeformation:ListPermissions", 
                "lakeformation:GetDataAccess",
                "lakeformation:GrantPermissions",
                "lakeformation:RevokePermissions",
                "lakeformation:BatchRevokePermissions",
                "lakeformation:AddLFTagsToResource",
                "lakeformation:RemoveLFTagsFromResource",
                "lakeformation:GetResourceLFTags",
                "lakeformation:ListLFTags",
                "lakeformation:GetLFTag",
            ],
            "Resource": [
                "arn:aws:glue:<region>:<AWS Account>:catalog",
                "arn:aws:glue:<region>:<AWS Account>:table/<dbt output database>/*",
                "arn:aws:glue:<region>:<AWS Account>:database/<dbt output database>"
            ],
            "Effect": "Allow"
        },
        {
            "Sid": "Read_only_databases",
            "Action": [
                "glue:SearchTables",
                "glue:GetTableVersions",
                "glue:GetPartitions",
                "glue:GetTableVersion",
                "glue:GetTables",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:GetDatabase",
                "glue:GetPartition",
                "lakeformation:ListResources",
                "lakeformation:ListPermissions"
            ],
            "Resource": [
                "arn:aws:glue:<region>:<AWS Account>:table/<dbt source database>/*",
                "arn:aws:glue:<region>:<AWS Account>:database/<dbt source database>",
                "arn:aws:glue:<region>:<AWS Account>:database/default",
                "arn:aws:glue:<region>:<AWS Account>:database/global_temp"
            ],
            "Effect": "Allow"
        },
        {
            "Sid": "Storage_all_buckets",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<dbt output bucket>",
                "arn:aws:s3:::<dbt source bucket>"
            ],
            "Effect": "Allow"
        },
        {
            "Sid": "Read_and_write_buckets",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<dbt output bucket>"
            ],
            "Effect": "Allow"
        },
        {
            "Sid": "Read_only_buckets",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<dbt source bucket>"
            ],
            "Effect": "Allow"
        }
    ]
}
```

### Configuration of the local environment

Because **`dbt`** and **`dbt-glue`** adapter are compatible with Python versions 3.7, 3.8, and 3.9, check the version of Python:

```bash
$ python3 --version
```

Configure a Python virtual environment to isolate package version and code dependencies:

```bash
$ python3 -m venv dbt_venv
$ source dbt_venv/bin/activate
$ python3 -m pip install --upgrade pip
```

Configure the last version of AWS CLI

```bash
$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ sudo ./aws/install
```

Install boto3 package

```bash
$ sudo yum install gcc krb5-devel.x86_64 python3-devel.x86_64 -y
$ pip3 install --upgrade boto3
```

Install the package:

```bash
$ pip3 install dbt-glue
```

### Example config

```yml
type: glue
query-comment: This is a glue dbt example
role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole
region: us-east-1
workers: 2
worker_type: G.1X
idle_timeout: 10
schema: "dbt_demo"
session_provisioning_timeout_in_seconds: 120
location: "s3://dbt_demo_bucket/dbt_demo_data"
```

The table below describes all the options.

| Option	                                 | Description	                                                                                                                                                                                                                                                                                      | Mandatory |
|-----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
| project_name	                           | The dbt project name. This must be the same as the one configured in the dbt project.	                                                                                                                                                                                                            | yes       |
| type	                                   | The driver to use.	                                                                                                                                                                                                                                                                               | yes       |
| query-comment	                          | A string to inject as a comment in each query that dbt runs. 	                                                                                                                                                                                                                                    | no        |
| role_arn	                               | The ARN of the glue interactive session IAM role.	                                                                                                                                                                                                                                                | yes       |
| region	                                 | The AWS Region were you run the data pipeline.	                                                                                                                                                                                                                                                   | yes       |
| workers	                                | The number of workers of a defined workerType that are allocated when a job runs.	                                                                                                                                                                                                                | yes       |
| worker_type	                            | The type of predefined worker that is allocated when a job runs. Accepts a value of Standard, G.1X, or G.2X.	                                                                                                                                                                                     | yes       |
| schema	                                 | The schema used to organize data stored in Amazon S3.Additionally, is the database in AWS Lake Formation that stores metadata tables in the Data Catalog.	                                                                                                                                        | yes       |
| session_provisioning_timeout_in_seconds | The timeout in seconds for AWS Glue interactive session provisioning.	                                                                                                                                                                                                                            | yes       |
| location	                               | The Amazon S3 location of your target data.	                                                                                                                                                                                                                                                      | yes       |
| query_timeout_in_minutes	               | The timeout in minutes for a signle query. Default is 300                                                                                                                                                                                                                                         | no        |
| idle_timeout	                           | The AWS Glue session idle timeout in minutes. (The session stops after being idle for the specified amount of time)	                                                                                                                                                                              | no        |
| glue_version	                           | The version of AWS Glue for this session to use. Currently, the only valid options are 2.0, 3.0 and 4.0. The default value is 4.0.	                                                                                                                                                               | no        |
| security_configuration	                 | The security configuration to use with this session.	                                                                                                                                                                                                                                             | no        |
| connections	                            | A comma-separated list of connections to use in the session.	                                                                                                                                                                                                                                     | no        |
| conf	                                   | Specific configuration used at the startup of the Glue Interactive Session (arg --conf)	                                                                                                                                                                                                          | no        |
| extra_py_files	                         | Extra python Libs that can be used by the interactive session.                                                                                                                                                                                                                                    | no        |
| delta_athena_prefix	                    | A prefix used to create Athena compatible tables for Delta tables	(if not specified, then no Athena compatible table will be created)                                                                                                                                                             | no        |
| tags	                                   | The map of key value pairs (tags) belonging to the session. Ex: `KeyName1=Value1,KeyName2=Value2`                                                                                                                                                                                                 | no        |
| seed_format	                            | By default `parquet`, can be Spark format compatible like `csv` or `json`                                                                                                                                                                                                                         | no        |
| seed_mode	                              | By default `overwrite`, the seed data will be overwritten, you can set it to `append` if you just want to add new data in your dataset                                                                                                                                                            | no        |
| default_arguments	                      | The map of key value pairs parameters belonging to the session. More information on [Job parameters used by AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html). Ex: `--enable-continuous-cloudwatch-log=true,--enable-continuous-log-filter=true` | no        |
| glue_session_id                         | re-use a glue-session to run multiple dbt run commands. Will create a new glue-session using glue_session_id if it does not exists yet.                                                                                                                                                           | no        | 
| glue_session_reuse                      | re-use the glue-session to run multiple dbt run commands: If set to true, the glue session will not be closed for re-use. If set to false, the session will be closed. The glue session will close after idle_timeout time is expired after idle_timeout time                                     | no        | 
| group_session_id                      | [**Model Level Meta Setting**] Set a specific glue session suffix id to group sets of models together to a specific session id.  Good for models that have chained dependencies in a larger dag and you want to save on session startup times.                                      | no        | 
| datalake_formats	                       | The ACID datalake format that you want to use if you are doing merge, can be `hudi`, `iceberg` or `delta`                                                                                                                                                                                         |no|
| use_arrow	                           | (experimental) use an arrow file instead of stdout to have better scalability.                                                                                                                                                                                                                    |no|
| enable_spark_seed_casting	              | Allows spark to cast the columns depending on the specified model column types. Default `False`.        |no|

## Configs

### Configuring tables

When materializing a model as `table`, you may include several optional configs that are specific to the dbt-spark plugin, in addition to the standard model configs.

| Option                | Description                                                                                                                                                                                                                                                  | Required?                               | Example                                           |
|-----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|---------------------------------------------------|
| file_format           | The file format to use when creating tables (`parquet`, `csv`, `json`, `text`, `jdbc`, `orc`, `delta`, `iceberg`, `hudi`, or `s3tables`).                                                                                                                   | Optional                                | `parquet`                                         |
| partition_by          | Partition the created table by the specified columns. A directory is created for each partition.                                                                                                                                                             | Optional                                | `date_day`                                        |
| clustered_by          | Each partition in the created table will be split into a fixed number of buckets by the specified columns.                                                                                                                                                   | Optional                                | `country_code`                                    |
| buckets               | The number of buckets to create while clustering                                                                                                                                                                                                             | Required if `clustered_by` is specified | `8`                                               |
| custom_location       | By default, the adapter will store your data in the following path: `location path`/`schema`/`table`. If you don't want to follow that default behaviour, you can use this parameter to set your own custom location on S3                                   | No                                      | `s3://mycustombucket/mycustompath`                |
| hudi_options          | When using file_format `hudi`, gives the ability to overwrite any of the default configuration options.                                                                                                                                                      | Optional                                | `{'hoodie.schema.on.read.enable': 'true'}`        |
| meta                  | Spawns isolated Glue session with different session configuration. Use Case: When specific models require configurations different from the default session settings. For example, a particular model might require more Glue workers or larger worker type. | Optional                                | `meta = { "workers": 50, "worker_type": "G.1X" }` |
| add_iceberg_timestamp | Add `update_iceberg_ts` column on Iceberg tables. (default: false)                                                                                                                                                                                           | Optional                                | `true`                                            |
| use_iceberg_temp_views | Use Spark temporary views when using Iceberg targets instead of physical tables to store intermediate results. (default: false)                                                                                                                                                                                           | Optional                                | `true`                                            |
| purge_dropped_iceberg_data | Purge Iceberg and S3 Tables underlying data S3 object during drop table. (default: false)                                                                                                                                                                                           | Optional                                | `true`                                            |

## Amazon S3 Tables Support (Experimental)

**WARNING: Experimental Feature**: Amazon S3 Tables support is currently experimental and may have limitations or breaking changes in future versions.

dbt-glue supports [Amazon S3 Tables](https://aws.amazon.com/s3/features/tables/), a new table type for analytics workloads that provides Apache Iceberg compatibility with automatic optimization and management.

### Configuration

To use S3 Tables, set `file_format='s3tables'` in your model configuration:

```sql
{{ config(
    materialized='table',
    file_format='s3tables'
) }}

select 
    id,
    name,
    created_at
from {{ ref('source_table') }}
```

### Profile Configuration

S3 Tables require specific Spark configurations in your `profiles.yml`:

```yaml
your_profile:
  target: dev
  outputs:
    dev:
      type: glue
      # ... other configurations ...
      conf: >-
        spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
        --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog
        --conf spark.sql.defaultCatalog=glue_catalog
        --conf spark.sql.catalog.glue_catalog.warehouse=s3://your-warehouse-path
        --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
        --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
        --conf spark.sql.catalog.glue_catalog.glue.id=YOUR_ACCOUNT_ID:s3tablescatalog/your-s3-tables-bucket
      datalake_formats: iceberg
```

### Example Model

```sql
{{ config(
    materialized='table',
    file_format='s3tables',
    partition_by=['year', 'month']
) }}

select 
    customer_id,
    order_date,
    extract(year from order_date) as year,
    extract(month from order_date) as month,
    total_amount
from {{ ref('raw_orders') }}
where order_date >= '2024-01-01'
```

### Requirements

- **AWS Glue 4.0 or later** (recommended)
- **S3 Tables bucket** configured in your AWS account
- **Proper IAM permissions** for S3 Tables operations
- **Iceberg Spark extensions** configured in your profile

## Python models (Experimental)

**WARNING: Experimental Feature**: Python model support is currently experimental and may have limitations or breaking changes in future versions.

dbt-glue supports [Python models](https://docs.getdbt.com/docs/build/python-models) that allow you to apply transformations to your data using Python code and libraries, rather than SQL. This enables more complex data transformations, statistical analysis, and machine learning workflows within your dbt project.

### Requirements

- **AWS Glue version 4.0 or later** (recommended for best Python support)
- **Iceberg file format** (required for Python models)
- **Proper Iceberg configuration** in your profile (see configuration example below)

### Configuration

Python models require Iceberg file format and specific Spark configurations. Add the following to your `profiles.yml`:

```yaml
your_profile:
  target: dev
  outputs:
    dev:
      type: glue
      # ... other configurations ...
      datalake_formats: iceberg
      conf: >-
        --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
        --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog
        --conf spark.sql.catalog.glue_catalog.warehouse=s3://your-warehouse-path
        --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
        --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
```

### Basic Usage

Create a Python model by adding a `.py` file to your `models/` directory:

```python
def model(dbt, spark):
    # Configure the model
    dbt.config(materialized='python_model', file_format='iceberg')
    
    # Create your DataFrame using Spark
    data = [
        (1, 'Alice', 100),
        (2, 'Bob', 200),
        (3, 'Charlie', 300)
    ]
    columns = ['id', 'name', 'value']
    
    # Return a Spark DataFrame
    return spark.createDataFrame(data, columns)
```

### Referencing Other Models

You can reference other dbt models and sources in your Python models:

```python
def model(dbt, spark):
    dbt.config(materialized='python_model', file_format='iceberg')
    
    # Reference another dbt model
    customers_df = dbt.ref('customers')
    
    # Reference a source
    orders_df = dbt.source('raw_data', 'orders')
    
    # Perform transformations
    result_df = customers_df.join(orders_df, 'customer_id')
    
    return result_df
```

### Incremental Python Models

Python models support incremental materialization with merge strategy:

```python
def model(dbt, spark):
    dbt.config(
        materialized='incremental',
        file_format='iceberg',
        incremental_strategy='merge',
        unique_key='id'
    )
    
    # Get source data
    source_df = dbt.ref('raw_events')
    
    if dbt.is_incremental():
        # Only process new records for incremental runs
        max_date = spark.sql(f"SELECT MAX(event_date) FROM {dbt.this}").collect()[0][0]
        source_df = source_df.filter(source_df.event_date > max_date)
    
    # Apply transformations
    transformed_df = source_df.groupBy('user_id').agg(
        count('*').alias('event_count'),
        max('event_date').alias('last_event_date')
    )
    
    return transformed_df
```

### Configuration Options

Python models support the same configuration options as regular models, plus some Python-specific ones:

| Option | Description | Example |
|--------|-------------|---------|
| `materialized` | Materialization type (`python_model` or `incremental`) | `python_model` |
| `file_format` | File format (must be `iceberg` for Python models) | `iceberg` |
| `incremental_strategy` | Strategy for incremental models | `merge` |
| `unique_key` | Unique key for merge operations | `['id']` |
| `partition_by` | Partition columns | `['date_column']` |

### Limitations

- **File format**: Only Iceberg file format is supported for Python models
- **Glue version**: Requires AWS Glue 4.0 or later for optimal support
- **Session configuration**: Requires proper Iceberg Spark extensions configuration
- **Return type**: Model function must return exactly one Spark DataFrame
- **Performance**: Python models may have longer execution times compared to SQL models

### Best Practices

1. **Use appropriate file formats**: Always use `file_format='iceberg'` for Python models
2. **Optimize DataFrame operations**: Use Spark DataFrame operations efficiently
3. **Handle incremental logic**: Use `dbt.is_incremental()` for conditional processing
4. **Test thoroughly**: Python models are experimental, so test extensively
5. **Monitor performance**: Python models may require more resources than SQL models

### Example: Data Science Workflow

```python
def model(dbt, spark):
    dbt.config(
        materialized='python_model',
        file_format='iceberg',
        partition_by=['analysis_date']
    )
    
    # Import required libraries (available in Glue environment)
    from pyspark.sql.functions import col, when, avg, stddev
    from datetime import datetime
    
    # Get source data
    sales_df = dbt.ref('sales_data')
    
    # Perform statistical analysis
    stats_df = sales_df.groupBy('product_category').agg(
        avg('sales_amount').alias('avg_sales'),
        stddev('sales_amount').alias('stddev_sales'),
        count('*').alias('transaction_count')
    )
    
    # Add analysis metadata
    result_df = stats_df.withColumn('analysis_date', lit(datetime.now().date()))
    
    return result_df
```

## Incremental models

dbt seeks to offer useful and intuitive modeling abstractions by means of its built-in configurations and materializations.

For that reason, the dbt-glue plugin leans heavily on the [`incremental_strategy` config](configuring-incremental-models#about-incremental_strategy). This config tells the incremental materialization how to build models in runs beyond their first. It can be set to one of three values:
 - **`append`** (default): Insert new records without updating or overwriting any existing data.
 - **`insert_overwrite`**: If `partition_by` is specified, overwrite partitions in the table with new data. If no `partition_by` is specified, overwrite the entire table with new data.
 - **`merge`** (Apache Hudi and Apache Iceberg only): Match records based on a `unique_key`; update old records, insert new ones. (If no `unique_key` is specified, all new data is inserted, similar to `append`.)
 
Each of these strategies has its pros and cons, which we'll discuss below. As with any model config, `incremental_strategy` may be specified in `dbt_project.yml` or within a model file's `config()` block.

**Notes:**
The default strategy is **`insert_overwrite`**

### The `append` strategy

Following the `append` strategy, dbt will perform an `insert into` statement with all new data. The appeal of this strategy is that it is straightforward and functional across all platforms, file types, connection methods, and Apache Spark versions. However, this strategy _cannot_ update, overwrite, or delete existing data, so it is likely to insert duplicate records for many data sources.

#### Source code
```sql
{{ config(
    materialized='incremental',
    incremental_strategy='append',
) }}

--  All rows returned by this query will be appended to the existing table

select * from {{ ref('events') }}
{% if is_incremental() %}
  where event_ts > (select max(event_ts) from {{ this }})
{% endif %}
```
#### Run Code
```sql
create temporary view spark_incremental__dbt_tmp as

    select * from analytics.events

    where event_ts >= (select max(event_ts) from {{ this }})

;

insert into table analytics.spark_incremental
    select `date_day`, `users` from spark_incremental__dbt_tmp
```

### The `insert_overwrite` strategy

This strategy is most effective when specified alongside a `partition_by` clause in your model config. dbt will run an [atomic `insert overwrite` statement](https://spark.apache.org/docs/latest/sql-ref-syntax-dml-insert-overwrite-table.html) that dynamically replaces all partitions included in your query. Be sure to re-select _all_ of the relevant data for a partition when using this incremental strategy.

If no `partition_by` is specified, then the `insert_overwrite` strategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances, since it minimizes downtime while the table contents are overwritten. The operation is comparable to running `truncate` + `insert` on other databases. For atomic replacement of Delta-formatted tables, use the `table` materialization (which runs `create or replace`) instead.

#### Source Code
```sql
{{ config(
    materialized='incremental',
    partition_by=['date_day'],
    file_format='parquet'
) }}

/*
  Every partition returned by this query will be overwritten
  when this model runs
*/

with new_events as (

    select * from {{ ref('events') }}

    {% if is_incremental() %}
    where date_day >= date_add(current_date, -1)
    {% endif %}

)

select
    date_day,
    count(*) as users

from events
group by 1
```

#### Run Code

```sql
create temporary view spark_incremental__dbt_tmp as

    with new_events as (

        select * from analytics.events


        where date_day >= date_add(current_date, -1)


    )

    select
        date_day,
        count(*) as users

    from events
    group by 1

;

insert overwrite table analytics.spark_incremental
    partition (date_day)
    select `date_day`, `users` from spark_incremental__dbt_tmp
```

Specifying `insert_overwrite` as the incremental strategy is optional, since it's the default strategy used when none is specified.

### The `merge` strategy

**Compatibility:**
- Hudi : OK
- Delta Lake : OK
- Iceberg : OK
- Lake Formation Governed Tables : On going

NB: 

- For Glue 3: you have to setup a [Glue connectors](https://docs.aws.amazon.com/glue/latest/ug/connectors-chapter.html).

- For Glue 4: use the `datalake_formats` option in your profile.yml

When using a connector be sure that your IAM role has these policies:
```
{
    "Sid": "access_to_connections",
    "Action": [
        "glue:GetConnection",
        "glue:GetConnections"
    ],
    "Resource": [
        "arn:aws:glue:<region>:<AWS Account>:catalog",
        "arn:aws:glue:<region>:<AWS Account>:connection/*"
    ],
    "Effect": "Allow"
}
```
and that the managed policy `AmazonEC2ContainerRegistryReadOnly` is attached. 
Be sure that you follow the getting started instructions [here](https://docs.aws.amazon.com/glue/latest/ug/setting-up.html#getting-started-min-privs-connectors).


This [blog post](https://aws.amazon.com/blogs/big-data/part-1-integrate-apache-hudi-delta-lake-apache-iceberg-datasets-at-scale-aws-glue-studio-notebook/) also explain how to setup and works with Glue Connectors

#### Hudi

**Usage notes:** The `merge` with Hudi incremental strategy requires:
- To add `file_format: hudi` in your table configuration
- To add a datalake_formats in your profile : `datalake_formats: hudi`
  - Alternatively, to add a connections in your profile : `connections: name_of_your_hudi_connector`
- To add Kryo serializer in your Interactive Session Config (in your profile):  `conf: spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.hive.convertMetastoreParquet=false`

dbt will run an [atomic `merge` statement](https://hudi.apache.org/docs/writing_data#spark-datasource-writer) which looks nearly identical to the default merge behavior on Snowflake and BigQuery. If a `unique_key` is specified (recommended), dbt will update old records with values from new records that match on the key column. If a `unique_key` is not specified, dbt will forgo match criteria and simply insert all new records (similar to `append` strategy).

#### Profile config example
```yaml
test_project:
  target: dev
  outputs:
    dev:
      type: glue
      query-comment: my comment
      role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole
      region: eu-west-1
      glue_version: "4.0"
      workers: 2
      worker_type: G.1X
      schema: "dbt_test_project"
      session_provisioning_timeout_in_seconds: 120
      location: "s3://aws-dbt-glue-datalake-1234567890-eu-west-1/"
      conf: spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.hive.convertMetastoreParquet=false
      datalake_formats: hudi
```

#### Source Code example
```sql
{{ config(
    materialized='incremental',
    incremental_strategy='merge',
    unique_key='user_id',
    file_format='hudi',
    hudi_options={
        'hoodie.datasource.write.precombine.field': 'eventtime',
    }
) }}

with new_events as (

    select * from {{ ref('events') }}

    {% if is_incremental() %}
    where date_day >= date_add(current_date, -1)
    {% endif %}

)

select
    user_id,
    max(date_day) as last_seen

from events
group by 1
```

#### Delta

You can also use Delta Lake to be able to use merge feature on tables.

**Usage notes:** The `merge` with Delta incremental strategy requires:
- To add `file_format: delta` in your table configuration
- To add a datalake_formats in your profile : `datalake_formats: delta`
  - Alternatively, to add a connections in your profile : `connections: name_of_your_delta_connector`
- To add the following config in your Interactive Session Config (in your profile):  `conf: "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog`

**Athena:** Athena is not compatible by default with delta tables, but you can configure the adapter to create Athena tables on top of your delta table. To do so, you need to configure the two following options in your profile:
- `delta_athena_prefix: "the_prefix_of_your_choice"`
- If your table is partitioned, then the add of new partition is not automatic, you need to perform an `MSCK REPAIR TABLE your_delta_table` after each new partition adding

#### Profile config example
```yaml
test_project:
  target: dev
  outputs:
    dev:
      type: glue
      query-comment: my comment
      role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole
      region: eu-west-1
      glue_version: "4.0"
      workers: 2
      worker_type: G.1X
      schema: "dbt_test_project"
      session_provisioning_timeout_in_seconds: 120
      location: "s3://aws-dbt-glue-datalake-1234567890-eu-west-1/"
      datalake_formats: delta
      conf: "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"
      delta_athena_prefix: "delta"
```

#### Source Code example
```sql
{{ config(
    materialized='incremental',
    incremental_strategy='merge',
    unique_key='user_id',
    partition_by=['dt'],
    file_format='delta'
) }}

with new_events as (

    select * from {{ ref('events') }}

    {% if is_incremental() %}
    where date_day >= date_add(current_date, -1)
    {