Data Transfer: Jobs and Agent Clustering

Data transfer jobs consist of four key phases: Scanning, Analysis, Transfer, and Validation. Understanding how these phases work with agent clustering is essential to optimize the execution of data transfer jobs.

Job Phases

1. Scanning

The first phase of a data transfer job is Scanning. During this phase, the system scans the file system to identify files and directories in the specified storage. Only one agent can perform scanning for a given job. However, this agent can operate in a multi-threaded mode to improve efficiency.

2. Analysis

After scanning, the job enters the Analysis phase. The system checks the job rules to determine which files need to be transferred based on user-specified criteria. Similar to the Scanning phase, Analysis is handled by a single agent, ensuring consistency in rules applied across the transfer job.

3. Transfer

Once the relevant files have been identified, the job moves to the Transfer phase, in which multiple agents work in parallel to move the data. Transfer benefits significantly from agent clustering, as the use of multiple agents drastically reduces the time required to transfer large volumes of data.

4. Validation

The final phase is Validation, when the system checks the integrity and completeness of the transferred file data and confirm it matches the source. Validation occurs immediately after files are transferred. Multiple agents can participate in this phase, ensuring that data has been transferred accurately, without corruption. 

NOTE: In file-to-file transfers, the Validation phase can be skipped by setting the validation method to None.

Agent Clustering

Agent clustering functionality makes CloudSoda controller delegate files available to all agents that can perform the data transfer job. If you have several transfer jobs running at the same time, one agent can move data for multiple jobs simultaneously. The controller will not delegate files for agent-to-agent transfers unless there are no other options for a job to run.

Transfer Agents

In the Transfer phase, multiple agents can be used to move data concurrently. This also applies to agent-to-agent transfers. Agent clustering enables more efficient data transfers, particularly when managing large datasets or executing complex transfers.

Agent Selection and Restriction

During the Transfer and Validation phases, users can select specific agents to move data in a transfer job, which restricts other agents from being used in the operation. Managing agents is particularly important if certain agents do not have access to the required storage locations. Selecting an agent that cannot access the target storage will cause the job to become stuck, as the agent cannot perform its assigned task.

Selecting Agents for a Data Transfer Job 

Ensure that only agents with access to the necessary storage locations are selected for the transfer job.

1. Click Orchestration > Jobs in the left-hand navigation panel. 
   The Jobs page displays.
2. In the jobs list, click on the running job.
3. Open the options menu (represented by the ellipse (...) symbol).
4. Click Agent Override.
5. Select the agents to use for the data transfer job.

Carefully selection of the appropriate agents ensures that your data transfer job runs smoothly, without unnecessary delays. You can only select agents after the transfer job starts. If you perform the Agent Override action during the Scanning phase, then the scan will restart.

Best Practices

• Let CloudSoda Pick the Agent: CloudSoda tracks the available agent resources and uses heuristics to determine which agent is the best option to perform a data transfer. Unless there is a compelling reason to limit a data transfer to a specific agent, it is best to let CloudSoda broker the job.
  
  
• Choose Agents with Access: If you want to restrict the agents that can run a job, only select agents that have continuous access to the required storage locations. This prevents jobs from becoming stuck.