Security Whitepaper

Data Integrity

CloudSoda is responsible for moving and copying data, whether it is file-to-file or file-to-object. To ensure data integrity during transfers, CloudSoda employs two distinct methods. Firstly, when a file-to-file transfer is initiated, an MD5 checksum (1) is generated on the source file. The file is then transferred to a temporary file on the target system to avoid overwriting an existing file. A MD5 checksum is then generated on the target temp file, and the two checksums are compared. If they match, CloudSoda will rename and update the attributes on the target file, ensuring a complete transfer without data corruption. If they do not match, the operation is retried up to three times before being marked as failed. When a file is transferred to an S3 object, the same process is attempted, but the validation is based on the S3 ETag (2) mechanism. AWS or S3-based object storage generates the Etag when the file is uploaded and completed, and CloudSoda can validate the resulting object Etag by calculating it using the source data and the number of parts used in the upload. If the Etag does not match or the upload fails, CloudSoda will retry up to 40 times with an exponential backoff before marking the operation as failed. When an object is downloaded, the object’s Etag is compared to the local temp file's MD5 before CloudSoda completes the object-to-file transfer. This guarantees that the files and objects are identical and that there is no data corruption during transfers.

For Google Cloud Storage, the validation process is identical to S3, but it uses a CRC32C hash instead of an Etag. However, Azure Blob is different in that it does not provide a hash unless it is a single-part upload. Therefore, CloudSoda validates the upload for Azure Blob by verifying that the correct number of bytes is written to the blob container. At the end of the upload, CloudSoda sets the MD5 of the object as metadata, making it available for future use.

Data Movement

The CloudSoda Controller orchestrates the data movement while the Agent performs the data movement. The Agent can be internal or external to the Controller. The internal Agent supports mounting NFS/SMB shares directly. While it is possible for an external Agent to mount NFS/SMB storage, it is not supported because it can result in unexpected behavior. Therefore, it is recommended that if an external agent needs to access data over NFS/SMB that the end user directly mounts the storage on the host.

There is no encryption when moving data via the SMB/NFS protocols as they do not support it by default. For cloud or object transfers CloudSoda uses the cloud vendors provided SDK. The AWS, Azure SDK transfers the data using TLS v1.2 (4) (5) (6) over the WAN to ensure data is encrypted in flight. The Google SDK uses TLS 1.3. to transfer data into their object storage. (7) (8) (9)

The external Agent can be installed on Linux, Windows, or MacOS. The Agent has the potential to access any files that the host operating system can access, which includes Local Volumes, Direct Attached Storage, or SMB/NFS mounted storage.

Software Access and Firewalls 

The CloudSoda Controller can be deployed on-prem, in a customer cloud, or cloud hosted by CloudSoda. Fig 1 refers to the ports and software packages CloudSoda uses when talking to file-based storage, cloud provider storage, CloudSoda Agents, and our managed software control plane.

Figure 1

The CloudSoda Controller uses ports and connections to perform four primary tasks:

• Administration: This involves deploying, upgrading, and monitoring CloudSoda.
• Data Movement: The Controller facilitates file/object-based data movement through the Agent or SMB/NFS.
• Management: This includes facilitating the creation of the mesh network, as well as upgrading and monitoring the status of the Agents.
• Web UI and API: The Controller offers both a graphical user interface and an application programming interface for utilizing the platform.

Administering CloudSoda offers many benefits to customers, such as easy installation, real-time software patching and updates, and simplified troubleshooting. Additionally, it's necessary to gather information about the health of CloudSoda software and hardware. However, no customer data is collected during this process.

Data Movement:

If you use the Controller's internal Agent for NFS/SMB, the Controller needs access to all the ports indicated on the diagram to perform file operations successfully.

In contrast, when accessing cloud storage, CloudSoda does not require any firewall exceptions. Ports 80/443 are used by CloudSoda to establish a connection with cloud storage targets. After the connection is established, all data sent to cloud storage targets is encrypted as per the aforementioned section.

Communication:

CloudSoda uses agents to move or copy data. To enable communication with the CloudSoda Controller, the agent requires two outbound UDP Ports 7498/7499. The agent can be installed on Linux, Windows, or MacOS operating systems and can access any files accessible to the host operating system, including local volumes, directly attached storage, or SMB/NFS-mounted storage.