This reference architecture document provides guidance for the Data Intelligence platform deployments. Included is the reference architecture for a controller, an ElasticSearch cluster and an agents. For specific use cases please work with the CloudSoda team on specifics for your solution.
Single-Instance Deployment for the Data Intelligence Controller
The single instance deployment is for customer with less than 250 million files or if you want to run DataIntell as a POC.
| Small | Recommended | AWS | |
| CPU | 12 Cores | 24 Cores | m6i.4xlarge (16 Cores) |
| Memory | 32 GB | 76 GB | m6i.4xlarge (64 GB) |
| Storage | 500 GB SSD (500 MB/s) | 2 TB NVMe (1000 MB/s) |
500-2 TB GP3
|
| Network* | 1 GB Ethernet Connection | 10 GB Ethernet Connection |
1 GB |
| OS | Ubuntu 22.04 LTS or RHEL 9 | Ubuntu 22.04 LTS or RHEL 9 |
Ubuntu 22.04 LTS or RHEL 9 |
| Estimated indexing performance | up to 18 000 files/second | up to 25 000 files/second |
up to 25 000 files/second |
| Maximum number of files | 100 million files | 250 million files |
200 million files |
Things to Know:
-
A single instance deployment can host all necessary components, including the scanning agent. However, it is recommended to have a 10 GB Ethernet connection if you plan to run the scanning agent on the same machine as the controller.
-
Indexing performance depends on the speed of the scanned storage, the folder tree structure, the number of scanning agents, and the total amount of storage being scanned.
-
DataIntell will still work for customer with more than 250 million files if it’s only for a single scan(assessment), but performance and features could be limited.
Cluster Deployment for the DataIntell Controller
The cluster deployment is for customers with more than 250 million files or for customers that require high redundancy in their infrastructure.
| Controller | ElasticSearch | |
| CPU | 12 Cores | 16 Cores |
| Memory | 32 GB | 64 GB |
| Storage | 50 GB SSD (500 MB/s) | 1TB SSD (500 MB/s) |
| Network | 1 GB Ethernet Connection | 1 GB Ethernet Connection |
| OS | Ubuntu 22.04 LTS or RHEL 9 | Ubuntu 22.04 LTS or RHEL 9 |
|
Estimated indexing performance |
- | up to 40 000 files/s |
| Number of VM (Between 250 million files and 1 billion files) | 1 | 3 |
| Number of VM (Between 1 billion files and 1.5 billion files) | 1 | 4 |
| Number of VM (Between 1.5 billion files and 2 billion files) | 1 | 5 |
Comments
0 comments
Article is closed for comments.