> ## Documentation Index
> Fetch the complete documentation index at: https://app.redbrickai.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Source Discovery

> A hands-on guide to connecting an Amazon S3 bucket, indexing DICOM studies, and keeping a RedBrick AI Workspace synchronized.

Source Discovery turns the DICOM files already stored in Amazon S3 into searchable Workspace datapoints. Your files stay in your bucket: RedBrick reads their DICOM headers, groups them into series and studies, and keeps the Workspace synchronized as the bucket changes.

> [card] No data migration
>
> Keep S3 as the source of truth. RedBrick stores the metadata needed for discovery and search.

> [card] One datapoint per study
>
> Instances are grouped by `SeriesInstanceUID`, then into a study by `StudyInstanceUID`.

> [card] Automatic updates
>
> Weekly scans reconcile the bucket. Optional S3 access logs provide daily change detection.

<video controls src="https://mintcdn.com/redbrickai-c2e4bc62/cstZHsBFobULR0NV/assets/images/dashboard/source-discovery/overview.mp4?fit=max&auto=format&n=cstZHsBFobULR0NV&q=85&s=783f821d75b9edeb66f0d58eaaee39ce" />

## What you will set up

This guide takes you through four steps:

1. Give RedBrick read-only access to your source bucket.
2. Create an S3 storage integration in RedBrick.
3. Connect that integration to a Workspace.
4. Optionally enable S3 access logs for daily updates.

You need Organization **Admin** or **Owner** access and an existing Workspace.

## 1. Give RedBrick access to the source bucket

RedBrick needs to list objects and read DICOM files. It does not need permission to upload, modify, or delete source data.

![AWS Role Permission](https://mintcdn.com/redbrickai-c2e4bc62/cstZHsBFobULR0NV/assets/images/dashboard/source-discovery/role-permission.png?fit=max\&auto=format\&n=cstZHsBFobULR0NV\&q=85\&s=898fc636169bf2459513ec87cca39c0d)

Add the following policy to the IAM role or IAM user used by your RedBrick S3 storage integration. Replace `YOUR-SOURCE-BUCKET` with the bucket name.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListDiscoverySource",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::YOUR-SOURCE-BUCKET"
    },
    {
      "Sid": "ReadDiscoveryObjects",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectVersion"],
      "Resource": "arn:aws:s3:::YOUR-SOURCE-BUCKET/*"
    }
  ]
}
```

> [!TIP]
>
> To scan only one folder, replace the object resource with a prefix such as
> `arn:aws:s3:::YOUR-SOURCE-BUCKET/dicom/*`. You can also restrict `s3:ListBucket` with an
> `s3:prefix` condition, but the prefix must match the folder entered in RedBrick.

### If the bucket uses SSE-KMS

If objects are encrypted with a customer-managed AWS KMS key, also allow the integration role to decrypt with that key. Replace the placeholders below and ensure the KMS key policy also permits the role.

```json theme={null}
{
  "Sid": "DecryptDiscoveryObjects",
  "Effect": "Allow",
  "Action": "kms:Decrypt",
  "Resource": "arn:aws:kms:AWS-REGION:AWS-ACCOUNT-ID:key/KMS-KEY-ID"
}
```

> [!WARNING]
>
> An IAM allow policy cannot override an explicit `Deny` in an IAM policy, bucket policy, service
> control policy, VPC endpoint policy, or KMS key policy. Review those policies if RedBrick receives
> `AccessDenied` for only some files or prefixes.

### Role-based access or access keys?

Role-based access is recommended because it uses temporary credentials.

> [tab] IAM role (recommended)
>
> 1. Create an IAM role in the AWS account that owns the bucket.
> 2. Attach the read-only policy above to the role.
> 3. In RedBrick, begin creating an AWS S3 storage integration and select role-based credentials.
> 4. Copy the **trust relationship** displayed by RedBrick into the role's **Trust relationships** tab in AWS.
> 5. Copy the role ARN into RedBrick and verify the integration.
>
> The generated trust policy contains RedBrick's AWS principal and your unique external ID. Keep the
> external-ID condition in place to prevent another AWS customer from assuming the role.

> [tab] Access and secret keys
>
> 1. Create an IAM user dedicated to RedBrick.
> 2. Attach the read-only policy above.
> 3. Create an access key for the IAM user.
> 4. Enter the access key and secret key when creating the RedBrick storage integration.
>
> Rotate the keys according to your organization's security policy. Do not use AWS root-user keys.

> [tab] Public bucket
>
> If the bucket permits unsigned `ListObjectsV2` and `GetObject` requests, enable **Public bucket** in
> the RedBrick storage form. No role or access keys are required.
>
> Public access can vary by prefix or object. Confirm that every object you want to discover is
> publicly readable; a successful bucket listing does not guarantee access to every file.

## 2. Create the S3 storage integration

**1. Open Integrations**

In RedBrick, open **Integrations**, select **Storage**, and choose **New storage method**.

**2. Choose Amazon S3**

Enter a recognizable name, the source bucket, its AWS Region, and the credentials configured in
the previous step.

**3. Verify access**

Save and verify the integration. If verification fails, confirm the bucket name, Region, role
trust policy, external ID, and `s3:GetObject` permission.

For general storage-integration setup, see [Configure AWS S3 storage for RedBrick AI](https://app.redbrickai.com/docs/importing-data/import-cloud-data/configuring-aws-s3.md).

## 3. Connect the bucket to a Workspace

**1. Open the Workspace**

Open the Workspace that should receive the studies and select the **Discovery** tab.

**2. Connect storage**

Select **Connect storage** and choose the S3 integration created above.

![Connect Source option](https://mintcdn.com/redbrickai-c2e4bc62/cstZHsBFobULR0NV/assets/images/dashboard/source-discovery/connect-source.png?fit=max\&auto=format\&n=cstZHsBFobULR0NV\&q=85\&s=ec37b76fbe9b9528f0195b0edd70659f)

**3. Choose an optional folder**

Leave the folder empty to scan the entire bucket, or enter a prefix such as `dicom/2026/`. Do
not include `s3://` or the bucket name.

![New Source option](https://mintcdn.com/redbrickai-c2e4bc62/cstZHsBFobULR0NV/assets/images/dashboard/source-discovery/new-source.png?fit=max\&auto=format\&n=cstZHsBFobULR0NV\&q=85\&s=ef902f3cc360255ce62877536d4ef5a2)

**4. Start discovery**

Create the source. RedBrick immediately schedules a full scan. Studies appear in the Workspace
**Data** tab as they finish processing; you do not need to wait for the entire bucket to finish.

![Workspace sync](https://mintcdn.com/redbrickai-c2e4bc62/cstZHsBFobULR0NV/assets/images/dashboard/source-discovery/workspace-sync.png?fit=max\&auto=format\&n=cstZHsBFobULR0NV\&q=85\&s=83c01f6392ea9158919edd504ac26fcb)

> [!TIP]
>
> Setup is working when the Discovery tab shows indexed item, series, and study counts. For a large
> bucket, these counts increase while the scan is running.

## 4. Enable daily updates with S3 access logs

This step is optional. Without access logs, the weekly full scan still detects additions, modifications, renames, and deletions. With access logs configured, Discovery also processes relevant changes daily.

### Create or choose a logs bucket

Use a separate bucket in the same AWS account and Region as the source bucket. Choose a dedicated prefix such as `discovery/`.

> [!WARNING]
>
> Do not enable server access logging on the destination logs bucket. Logging the logs bucket can
> create an ever-growing loop of log objects. AWS also does not support server access log
> destinations with S3 Object Lock default retention or Requester Pays enabled.

In the AWS S3 console:

1. Open the **source bucket**.
2. Select **Properties** → **Server access logging** → **Edit**.
3. Enable logging and select the destination logs bucket.
4. Enter a destination prefix, for example `discovery/`.
5. Save changes.

The AWS console normally adds the required destination bucket policy automatically. If you manage the policy yourself, add the following statement to the **logs bucket policy**. Replace all placeholders.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3ServerAccessLogDelivery",
      "Effect": "Allow",
      "Principal": {
        "Service": "logging.s3.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::YOUR-LOGS-BUCKET/discovery/*",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::YOUR-SOURCE-BUCKET"
        },
        "StringEquals": {
          "aws:SourceAccount": "YOUR-AWS-ACCOUNT-ID"
        }
      }
    }
  ]
}
```

> [!NOTE]
>
> AWS recommends a bucket policy instead of ACLs for log delivery. Use Amazon S3 managed encryption
> (SSE-S3) for the logs bucket; AWS warns that server access log delivery can be incompatible with
> default SSE-KMS encryption on the destination. See AWS's [server access logging
> guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html)
> for the current destination-bucket requirements.

### Let RedBrick read the logs

The same role or IAM user used by the source storage integration must be able to list and read the logs location. Add these statements to its IAM policy:

```json theme={null}
{
  "Sid": "ListDiscoveryLogs",
  "Effect": "Allow",
  "Action": "s3:ListBucket",
  "Resource": "arn:aws:s3:::YOUR-LOGS-BUCKET",
  "Condition": {
    "StringLike": {
      "s3:prefix": [
        "discovery",
        "discovery/*"
      ]
    }
  }
},
{
  "Sid": "ReadDiscoveryLogs",
  "Effect": "Allow",
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::YOUR-LOGS-BUCKET/discovery/*"
}
```

Then open the Workspace **Discovery** tab, select the pencil icon beside **Logs bucket**, and enter:

* **Bucket:** `YOUR-LOGS-BUCKET`
* **Prefix:** `discovery/`

You do not need a second RedBrick storage integration for the logs bucket.

## What happens when files change?

| S3 change | Discovery behavior |
| - | - |
| A file is added | Adds the instance to its series and study, then creates or updates the Workspace datapoint |
| A file is overwritten | Preserves the old discovery version, creates a new version, and republishes the study |
| A file is renamed | Treats the new key and old-key deletion as one study update when their DICOM UIDs match |
| A file is deleted | Removes it from the current study after a complete scan confirms the deletion, then republishes the datapoint |

Discovery never treats an access error as a deletion. A file is marked deleted only after a complete successful full scan confirms that the key is absent.

## Monitor the first scan

The Workspace **Discovery** tab shows:

* Current status and pending retries.
* Indexed item, series, and study counts.
* The last completed full scan.
* Whether daily log sync is configured.
* Actions to refresh status, start a full scan or log sync, and retry failed files.

The normal schedule is:

| Operation | Schedule |
| - | - |
| Initial full scan | Immediately after connection |
| Incremental log sync | Daily, when a logs bucket is configured |
| Reconciliation scan | Weekly |

> [!NOTE]
>
> Indexing and Workspace publication run independently. On a large source, the study count can grow
> faster than the Workspace datapoint count while publication catches up.

## Troubleshooting checklist

### Access denied for every object

* Confirm the role trust relationship contains the RedBrick principal and correct external ID.
* Confirm `s3:ListBucket` targets the bucket ARN without `/*`.
* Confirm `s3:GetObject` and `s3:GetObjectVersion` target the object ARN with `/*`.
* Check bucket policies, organization service control policies, and VPC endpoint policies for explicit denies.

### Access denied for only some objects

* Check whether the object keys fall outside the permitted prefix.
* Check whether those objects use a different KMS key.
* Check object ownership and any object-specific ACLs on older buckets.
* For public integrations, verify the failing object with an unsigned request, not with your local AWS credentials.

### No daily changes appear

* Confirm AWS is writing objects under the configured logs prefix.
* Confirm the integration role can list and read that prefix.
* Remember that S3 server access log delivery is best effort and can be delayed; the weekly full scan remains the final reconciliation mechanism.
