Managing Quick Transfers

The Quick Transfer feature allows you to create and execute data transfers between any configured storage location. For more fine-grained control, you can customize the transfer with advanced settings to:

• Define rule-based criteria
• Manage source and target conflict handling
• Set conditions for job failure
• Enable group and user tracking

These options can be configured prior to starting the transfer job to ensure it meets your specific requirements.

• Create a Quick Transfer
  • Define Rules
  • Define Conflict Handling
  • Define Job Failures
• Add Group and User Tracking
• Add a Tag
• Run the Quick Transfer

Create a Quick Transfer

A Quick Transfer operation in CloudSoda involves three phases: Scanning, Analysis, and Transfer.

• Scanning: CloudSoda scans all files and folders in the selected data source and creates an index of the items to be transferred.
• Analysis: CloudSoda analyzes the indexed data and applies any configured rule, such as conflict handling. 
• Transfer: CloudSoda transfers the files and folders from the data source to the selected target.

To create a quick transfer, follow these steps:

1. Click Orchestration > Dashboard in the left-hand navigation panel.
   The Dashboard page displays. 
   
   
2. Click Quick Transfer at the top-right of the page.
   The Transfer page displays.
   
   
3. Enter the name of the Quick Transfer in the Add a name…field. If left blank, CloudSoda automatically generates a name, which displays on the Jobs page once the transfer starts.
   
   
4. Select the data source from the Source drop-down, click the / field to browse, and click Select to choose the source folder.
   NOTE: Only the folder contents are transferred, not the folder itself).
   
   
5. Select the target from the Target drop-down, click the / field to browse, and click either Select for an existing target folder or Create New Folder to define a new target folder.
   
   
6. Select the file transfer method: Copy, Move or Sync:
   
   • Copy – Transfers files without deleting anything on the target
   • Move – Transfers files and deletes them from the source
   • Sync – Makes the target match the source exactly by copying new files and deleting those not present in the source
     
     NOTE: When using Sync, you must specify a conflict handling rule. Sync transfer can also be configured with file rules to selectively sync specific files. Use caution when creating rules, as incorrect settings may result in unintended data deletion. It's recommended to perform a Dry Run and review the job log before executing the full transfer. 

At this point, you can configure additional settings. Before starting the transfer, you can apply additional options such as:  

• Rule-based criteria
• Conflict handling preferences
• Failure conditions
• Group and user tracking

Define Rules

Rules enable you to specify criteria for any data transfer within CloudSoda, including Quick Transfers. Available rules allow you to define transfer policies based on different file attributes.

Note: You do not have to define any rules to start a Quick Transfer.

1. Click Orchestration > Dashboard in the left-hand navigation panel.
   The Dashboard page displays. 
   
   
2. Click Quick Transfer > Rules at the top-right of the page.
   The Rules tab displays.
   
   
3. Click Add a Rule.
   The Rule dialog box displays.
   
   
4. Select a file attribute for the rule. 
   
   • File name
   • File path
   • File size
   • File accessed
   • File accessed (relative)
   • File modified
   • File modified (relative)
     
     
5. Select a rule for the selected attribute and click Save. The following table lists available rules. 
   
   

   File Name	File Path	File Size	File Accessed	File Modified
   contains	contains	equals	after	after
   does not contain	does not contain	not equal to	before	before
   equals	glob	greater than	newer than	newer than
   does not equal	does not glob	greater than or equal to	older than	older than
   ends with	matches	less than
   does not end with	does not match	less than or equal to
   starts with
   does not start with
   matches
   does not match

   NOTE: The matches and does not match attributes allow for regular expressions, e.g. to find a .wav, .WAV or .Wav file you could use the expression (?i)\.wav$

6. To create additional rules, repeat Steps 1 to 5.
   
   
7. If multiple rules has been defined, use the Rules Selector to specify how rules are applied to files based on include/exclude options. In the Rules Selector drop-down, select one of the following options:
   
   1. • Include if all – Files must match ALL rules to be included in the transfer 
      • Include if any – Files can match ANY rule to be included in the transfer
      • Exclude if all – Files must match ALL rules to be excluded from the transfer
      • Exclude if any – Files can match ANY rule to be excluded from the transfer
8. Click Set Ignore.
   
   Use the Set Ignore setting to prevent CloudSoda from scanning specific files and folders. Ignore lists are applied before rules, which can decrease scan times. For more information, please see Creating an Ignore List.

Define Conflict Handling

Conflict handling policies determine how identically-named files on the data source and the target will be managed. For more fine-grained conflict handling, you can define additional conditions based on the size of the source file or its modified time.

1. Click Orchestration > Dashboard in the left-hand navigation panel.
   The Dashboard page displays. 
   
   
2. Click Quick Transfer > Conflicts at the top-right of the page.
   The Conflicts tab displays.
   
   
3. In the Conflicts Handling drop-down, select one of the following options:
   • Overwrite Target File – Overwrites the target file
   • Rename Transferred File – Creates a new target file with the name of the source file and adds the copy-1 suffix. For example, the Test1.txt file on the source is renamed as the Test1_copy-1.txt file on the target. As the policy continues to run, the file name is incremented, i.e. copy-1, copy-2, copy-3, etc.
   • Skip Source File – Skips the source file during the data transfer and keeps the target file
     
     
4. To define a conflict handling condition, click Add a Condition.
   The Conflict Criterion dialog box displays.
   
   
5. Select a conflict attribute for the condition from these options:
   
   • Source file modified time
   • Source file size
     
     
6. Select a conflict criterion for the selected attribute and click Save. The following table lists available criteria for each conflict attribute.
   
   

   Source File Modified Time	Source File Size
   not equal to Target	not equal to Target
   after Target	less than Target
   after or equal to Target	less than or equal to Target
   before Target	greater than Target
   before or equal to Target	greater than or equal to Target

   
   NOTE: The matches and does not match attributes allow for regular expressions, e.g. to find a .wav, .WAV or .Wav file you could use the expression (?i)\.wav$
   
   

7. To define additional conflict handling conditions, repeat Steps 3 to 6. Multiple conditions are handled with AND operators, meaning all conditions must be true for the conflict handling action to be performed. 

Define Job Failures

Certain conditions may indicate a problem with the Quick Transfer (too many file transfer errors or a very long run time). In these situations, you may want to terminate the operation and fail the job. Use the Failure option to specify failure conditions and thresholds for the data transfer.

1. Click Orchestration > Dashboard in the left-hand navigation panel.
   The Dashboard page displays. 
   
   
2. Click Quick Transfer > Failure at the top-right of the page.
   The Failure tab displays.
   
   
3. Define the job failure conditions. The default condition is Transfer Error Limit, which is set to 10 failures. To change this threshold, click the Edit icon, specify a new value, and click Save.
   
   
4. To define additional failure conditions, click Add a Condition.
   The Failure Condition dialog box displays.
   
   
5. Select a failure condition from these options, specify a value and click Save:
   
   • Transfer Error Percentage
   • Transfer Run Time Limit
     
     
6. To define additional failure conditions, repeat Steps 4 to 6. Multiple conditions are handled with OR operators, meaning that any of the conditions can be true for the job to be terminated.

Add Group and User Tracking

Use the Tracking option to assign specific groups and users to the Quick Transfer, which enables them to track job status.

1. Click the Tracking option.
   
   
2. In the Group drop-down, select a group to assign to the data transfer.
   Refer to Adding Groups for information on adding Groups to this list.
   
   
3. In the User drop-down, select a user to assign to the data transfer. 
   Refer to Adding Users for information on adding Users to this list.
   

Add a Tag

Object tagging gives you a way to categorize Amazon S3 storage. Each tag is a key-value pair. You can associate up to 10 tags with an object. Tags associated with an object must have unique tag keys.

1. Click Add Tag.
   The Add Tag window displays.
   
   
2. Enter a Key, a Value, and click Save.

Advanced Options

The Advanced tab provides tunable options to adjust advanced properties of the job. This section should be ignored unless you know why you want to override the default behavior.

Validation Method 

The Validation Method drop-down menu allows you to select your preferred approach to validate files on the storage target. Not all storages offer the same validation methods, for example, validation options for objects and files differ. The available validation methods are listed below:

• Content - Validation method used depends on the file contents, e.g. MD5, ETag or CRC32C
• Recommended - CloudSoda identifies the most efficient data transfer method based on speed and file efficacy
• Size - Validation method used is based on file size
• None - Validation is not performed

Windows Security Information

The Windows Security Information allows the user to tune the Windows Security Descriptors when transferring files between Windows file storages. This feature contains some constraints to ensure the job’s success. The agent running the job must be a Windows agent, and the accessor for both the source and the target must be the type Local with a path provided. Currently, the SMB accessor is not supported for replicating Windows Security Descriptors. If the agent is unable to modify the permissions, the file transfer will fail. Therefore, ensure the agent running the transfer has the necessary permissions to modify the Windows Security Descriptors.

Note:System Access Control Lists (SACLs) are often not set and will result in a permission error. Therefore, they should be ignored in most cases unless you are sure you need to copy them.

Run the Quick Transfer

After you have created the Quick Transfer, including any additional configuration options, select one of the following options: 

• Dry Run - previews details of the configured Quick Transfer before it starts
• Transfer Now - runs the data transfer without group or user tracking assigned