This reference architecture document provides guidance for CloudSoda deployments. Included is the reference architecture for a CloudSoda controller and CloudSoda agents. For specific use cases please work with the CloudSoda team on specifics for your solution.
CloudSoda Agents
The tables below outline the recommended architecture for servers and VMs,
and include cloud templates that can be used. These values should only serve as a guideline. The proper specification should be adjusted based on the dataset, storage type, and network speed involved in the data transfer.
| CloudSoda Agent | ||||
| Minimum | Small | Recommended | Large | |
| CPU | 2 Cores | 8 Cores | 16 Cores | 48 Cores |
| Memory | 4 GB | 16 GB | 32 GB | 96 GB |
| Storage | 200 GB | 200 GB | 200 GB | 200 GB |
| Estimated Performance | 500 Mbps-1 Gbps | 1-3 Gbps | 3-10 Gbps | 10+ Gbps |
|
Agent OS Support Matrix |
||
| Linux | RHEL 8 and later & Ubuntu 18.04 and later | x86/arm |
| MacOS | 10.15 Catalina or later | Intel/Apple Silicon |
| Windows | Windows 10 and later & Server 2016 and later | x86 |
Things to know:
- An agent can run on almost any architecture to enable data movement including laptops, servers, VMs, and other platforms.
- Estimated speed may vary based on latency and file numbers/sizes
- The performance of the data movement is dependent on the number of cores available to the agent. The more cores you have available the faster the agent will perform.
- We recommend a two-to-one memory-to-core ratio for cloud transfers.
- For agents doing data movement to and from the cloud or an object store, using a processor that has integrated SHA extension will give approximately a 30% increase in performance. For a list of processors, click on the links below:
https://en.wikipedia.org/wiki/Intel_SHA_extensions
https://en.wikipedia.org/wiki/Ice_Lake_(microprocessor)
https://en.wikipedia.org/wiki/Zen_3
CloudSoda Controller
The tables below outline the recommended architecture for servers and VMs and include cloud templates that can be used.
| CloudSoda Controller | |||||
| Non-Production/Sandbox | Minimum | Recommended | Cloud Template | ||
| CPU | 8 Cores | 16 Cores | 24 Cores | AWS | t3a.2xlarge |
| Memory | 8 GB | 16 GB | 48 GB | Azure | 1 Dv v2 |
| Storage (SSD) | 200 GB (Single Partition) | 500 GB (Single Partition) | 2 TB (Single partition) | GCP | e2-standard-8 |
| Controller OS Support Matrix | |
| Linux | Ubuntu 22.04 LTS Server |
Things to know:
- The CloudSoda controller requires a minimum 500GB of storage on the primary partition for production controllers.
- The CloudSoda controller should only have a single IP address (other than the loopback).
- 100 million files or scanning hourly should consider more storage on their controller node, please contact the CloudSoda team for details.
Comments
0 comments
Article is closed for comments.