Overview

The system supports reading input data files in S3 buckets

File Path

The files shall be uploaded with the path prefix YYYY/MM/DD, e.g. for tables.parquet file, the full S3 path shall be s3://{bucket_name}/2025/09/01/tables.parquet where the date is the date of query execution.

Bucket Setup

There are 2 options for configuring bucket access

Option 1: Upload to Single Origin S3 bucket

📘

Pre-requisite: Permission to create or access to IAM roles

Bucket creation

On our end, we'll create a custom bucket named so-data-{your_company_name} and share the ARN

Setup IAM role

1. Log in to the AWS Management Console
2. Go to IAM > Roles and click Create role
   1. We recommend creating a new role for better permission control, but an existing role can be used as well
3. Share the role ARN with Single Origin, e.g. arn:aws:iam::{client_account_id}:role/{row_name}
4. On our end, we will grant the role access to the S3 bucket for uploading data

Option 2: Use your own S3 bucket

📘

Pre-requisite: Permission to create S3 bucket as well as permission to create or access to IAM users

Create an S3 bucket

1. Login to AWS Management Console
2. Create a S3 Bucket
   1. Navigate to the S3 service
   2. Click on Create bucket
3. Enter a unique bucket name (e.g., your-unique-bucket-name)
4. Select the AWS region where you want to create the bucket
5. Create bucket

Grant read access to Single Origin

1. Go to the IAM service in the AWS Management Console
2. Click on Users and then Add user
3. Enter a user name (e.g., S3ReadOnlyUser) and select Programmatic access
4. Click on Next: Permissions
5. Select Attach policies directly and search for AmazonS3ReadOnlyAccess
6. Select the policy and click Next: Tags, then Next: Review, and finally Create user
7. Note down the Access Key ID and Secret Access Key
8. Save Access Keys in Our System
   1. Log in to our system and navigate to the settings or configuration page for S3 access. Enter the Access Key ID and Secret Access Key in the respective fields. Save the configuration.