Repository: aws/aws-sdk-pandas

Issue #2378: `wr.s3.to_deltalake` throwing TypeError about storage_options
Created: 2023-07-03T22:05:52Z
Describe the bug
Calling we.s3.to_deltalake() throws the following error:
self._table = RawDeltaTable(
                  ^^^^^^^^^^^^^^
TypeError: argument 'storage_options': 'NoneType' object cannot be converted to 'PyString'

How to Reproduce
wr.s3.to_deltalake(
            df=data,
            path="s3://bucket/delta",
            index=False,
            partition_cols=["a", "b"],
            overwrite_schema=True,
            s3_additional_kwargs={
                "AWS_ACCESS_KEY_ID": "...",
                "AWS_SECRET_ACCESS_KEY": "...",
                "AWS_REGION": "eu-west-1",
            },
            s3_allow_unsafe_rename=True,
        )
Expected behavior
I'd expect awswrangler to connect to S3 and write the delta table.
Your project
No response
Screenshots
No response
OS
Mac
Python version
3.11.4
AWS SDK for pandas version
3.2.1
Additional context
No response

Duplicate issue #2484: wr.s3.to_deltalake throw error: argument 'storage_options': 'NoneType' object cannot be converted to 'PyString'
Created: 2023-10-06T06:07:23Z
Describe the bug
When I try save a data by wr.s3.to_deltalake I recive the error: argument 'storage_options': 'NoneType' object cannot be converted to 'PyString'
How to Reproduce
import os
import awswrangler as wr
import boto3
import pandas
from datetime import datetime
import traceback
from config_old_tables import list_source_database, partition_cols, aws_profile_source, aws_profile_target, dbs_to_buckets, list_tables

session_source= boto3.Session(profile_name=aws_profile_source)
session_target = boto3.Session(profile_name=aws_profile_target)

poor_log_path =  os.path.join(os.path.dirname(os.path.abspath(__file__)), 'poor_log.txt') 

with open(poor_log_path , "w") as log:
    for source_database in list_source_database:
        for table_name in list_tables:
            sql_query = f"select * from {source_database}.{table_name}"
            print(f"Reading {table_name} from {source_database}")
            print(f"Executing query: {sql_query}")
            try:
                df = wr.athena.read_sql_query(
                    sql=sql_query,
                    database=source_database,
                    boto3_session=session_source,
                    ctas_approach=False
                )
            except Exception as e:
                print(f"Error while reading {table_name} from {source_database}")
                print(e)
                log.write(f"Error while reading {table_name} from {source_database}\n")
                log.write(f"{e}\n")
                continue
            for target_database, bucket in dbs_to_buckets.items():
                s3_path = f"{bucket}/{table_name}/".replace("_", "-")
                try:
                    if "raw" in target_database:
                        df_raw = df.copy()
                        df_raw['date_proc_de'] = datetime.now().strftime("%Y-%m-%d")
                        wr.s3.to_parquet(
                            df=df_raw,
                            path=s3_path,
                            dataset=True,
                            database=target_database,
                            table=table_name,
                            mode="overwrite",
                            partition_cols = partition_cols,
                            boto3_session=session_target
                        )
                        print(f"{table_name} saved into {target_database}")
                    else:
                        df_delta = df.copy()
                        df_delta['date_proc_de'] = datetime.now().strftime("%Y-%m-%d")

                        ###########  Here the error ###########  
                        wr.s3.to_deltalake(
                            df=df_delta,
                            path=s3_path,
                            mode="overwrite",
                            partition_cols = partition_cols,
                            index=False,
                            overwrite_schema=True,
                            s3_allow_unsafe_rename=True
                        )
                        print(f"{table_name} saved into {target_database}")
                except Exception as e:
                    traceback.print_exc()
                    print(f"Error while saving {target_database}.{table_name}")
                    print(e)                    
                    log.write(f"Error while saving {target_database}.{table_name}\n")
                    log.write(f"{e}\n")
                    continue
Expected behavior
Save with sucess like I get when use the wr.s3.to_parquet
Your project
No response
Screenshots
No response
OS
Windows 11
Python version
3.10
AWS SDK for pandas version
1.27.1
Additional context
awswrangler 3.3.0

Closing comment (2023-10-06T10:47:53Z): This is a duplicate of #2378
Please try the suggestions here: #2378 (comment)