Managing Storage

In CloudSoda, a storage is a logical handle to a collection of data—whether that data lives in an object bucket (with or without a prefix), a network file share, or a local file path. The storage resource abstracts away protocols, credentials, and connection details so you can browse, analyze, and move files and objects through a single, consistent interface. Once a storage is defined, CloudSoda treats it as a self‑contained namespace that can be scanned, costed, or used as a source or target in any job.

Use the following procedures to manage storage:

• Creating and Configuring Storage
• Storage Status
• Editing Storage

Creating and Configuring Storage

To create new storage, follow these steps:

1. Click Orchestration > Storage in the left-hand navigation pane.
   The Storage page displays and defaults to the Storage tab.
2. Click Create Storage at the top-right of the page.
   The Create Storage dialog box opens, which enables you to name and configure key details for the new storage. The following sections describe configuration steps for various storage types.

To configure the new storage, go to the appropriate section for the storage type:

• File
• AWS S3
• Azure
• Google Cloud Platform (GCP)
• Custom S3
• Wasabi
• Dell PowerScale

File

Enter a name for the file storage, select the desired Price Book, and click Save.

NOTE: For important details on naming the storage file, refer to the File Naming Best Practices section of CloudSoda Best Practices. For guidance on configuring access to the new Storage, see Managing Accessors.

AWS S3

If you are using KMS for your encryption provider on your bucket(s), do not use the CloudFormation template, as it resets your bucket's encryption provider to the standard SSE-S3 encryption provided by Amazon. Using the Terraform provider to create your buckets is known as "AES256" (supplied by the sse_algorithm value). Use our CloudFormation template instead. The example below lists the resulting policy when using CloudFormation and the KMS policy objects we've determined work correctly with CloudSoda.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "kms:DescribeKey",
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:GenerateDataKey*",
                "kms:ReEncrypt*"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:kms:${REGION}:${ACCOUNT_NUMBER}:key/${KEY_HASH}",
            "Sid": "grantKmsPrivs"
        },
        {
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            },
            "Action": [
                "s3:ListBucket",
                "s3:*Object",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET_NAME}",
                "arn:aws:s3:::${BUCKET_NAME}/*"
            ],
            "Effect": "Deny",
            "Sid": "DenyNonSSL"
        },
        {
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "true"
                }
            },
            "Action": [
                "s3:ListBucket",
                "s3:*Object*",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET_NAME}",
                "arn:aws:s3:::${BUCKET_NAME}/*"
            ],
            "Effect": "Allow",
            "Sid": "PermitSSLoperationsSoDA"
        }
    ]
}

The policy provides the user with the ability to read and write objects to a KMS-encrypted bucket by scoping permissions to the bucket and its contents. Additionally, the policy permits KMS operations on the specific KMS key (ARN). You can broaden the policy scope to allow for multiple buckets and KMS keys. 

The "DenyNonSSL" statement ID is a denial policy that disallows HTTP requests to the bucket, requiring SSL-encrypted transactions. This identifier is optional, but it is commonly-used.

To complete this process, confirm that you have applied this IAM policy to a user and then generate a key pair for the user, to be entered into your Controller as Accessor information, discussed below.

It is also recommended to restart your agents after applying the policy due to potential caching issues.

Using CloudFormation to Configure AWS S3 Storage

CloudSoda provides CloudFormation templates to streamline the creation of IAM users and buckets. If you have already created your bucket and have an Access Key ID and a Secret Key ID, then proceed to the next section. If you have not completed this configuration, please leverage the CloudFormation templates to assist you. 

Enter your information into the following fields to set up AWS Storage:

• Name – Name of the storage
• Scope – Defines access control to the resource
• Storage Type – AWS
• Bucket – Name of an existing bucket to expose to the system
  Example: my-soda-bucket
• Prefix – Specifies objects that are visible; prepended to all objects transferred to the bucket
  NOTE: The default is to leave this field blank.
  Example: /project/example
• Storage Class – Default storage class used for all objects transferred to the bucket by CloudSoda.
• Region – Select your preferred region from the drop-down list.

Azure

Enter your information into the following fields to set up Azure Storage.

• Name – Name of the storage
• Storage Type – Azure Blob Storage
• Storage Account – Name of an existing storage account that provides a namespace for the container.
  Example: mystorageaccount
• Container – Name of an existing container to expose to the system.
  Example: mycontainer
• Region – Select your preferred region from the drop-down list.
• Prefix – Specifies objects that are visible; prepended to all objects transferred to the bucket
  NOTE: The default is to leave this field blank.
  Example: /project/example
• Access Tier – Default storage class used for all objects transferred to the bucket by the system. 

Google Cloud Platform (GCP)

Enter your information into the following fields to set up GCP Storage.

• Name – Name of the storage
• Storage Type – Google Cloud Storage
• Bucket – Name of an existing bucket to expose to the system
  Example: my-soda-bucket
• Prefix – Specifies objects that are visible; prepended to all objects transferred to the bucket
  NOTE: The default is to leave this blank.
  Example: /project/example
• Storage Class – Default storage class used for all objects transferred to the bucket by the system. Objects with different storage classes are visible, but not affected by this setting.
• Region – Select your preferred region from the drop-down list.

Wasabi 

Enter your information into the following fields to set up Wasabi Storage.

• Name – Name of the storage
• Storage Type – Wasabi
• Bucket – Name of an existing bucket to expose to the system
  Example: my-soda-bucket
• Prefix – Specifies objects that are visible; prepended to all objects transferred to the bucket
  NOTE: The default is to leave this blank.
  Example: /project/example
• Region – Select your preferred region from the drop-down list. 
  NOTE: Wasabi with ObjectScale backed storage regions (ending with dell-obs) are not supported using the Wasabi storage type. Please use "Custom S3" for the storage and accessor.
• Price Book – Select the appropriate price book from the drop-down menu.

Dell PowerScale

Enter your information into the following fields to set up Dell PowerScale Storage.

• Name – Name of the storage
• Access Point - Top level folder path on the cluster. We recommend /ifs , but you can use a lower level directory like /ifs/foo/bar

Custom S3

Use this selection to connect to a public or private cloud store that utilizes the S3 interface. Enter your information into the following fields to set up Custom S3 Storage.

• Name – Name of the storage card
• Storage Type – Custom S3
• Bucket – Name of an existing bucket to expose to the system
  Example: my-soda-bucket
• Prefix – Specifies objects that are visible; prepended to all objects transferred to the bucket.
  NOTE: The default is to leave this field blank.
  Example: /project/example
• Storage Class – Default storage class used for all objects transferred to the bucket by the system. Only the Standard option is available for Custom S3.
• Price Book – Select the appropriate price book from the drop-down menu.

Storage Status

Use the visual indicators on the Storage page to quickly determine the status of different storages (either enabled or disabled):

• Enabled (blue icon) Indicates the storage is active, but the Agent attachment needs to be connected to use the storage.
• Disabled (grey icon) Indicates the storage is disabled and cannot be used for jobs or storage management.

Editing Storage

You can edit any storage you've created, but it must be disabled first. Once you've made your changes, re-enable the storage before using it again.

To enable or disable storage, follow these steps:

1. Click Orchestration > Storage in the left-hand navigation pane.
   The Storage page displays and defaults to the Storage tab.
   
   
2. Select the storage you want to edit. 
   
   
3. Click the ellipsis (…) at the top-right of the page and Disable to toggle the storage's status.
   
   
4. Click the ellipsis (…) again and Edit to update the storage's settings.
   
   
5. Update the settings as needed and click Update Storage to save the changes. 
   
   
6. Click the ellipsis (…) again and Enable to reactivate the storage.