CloudSoda Release 26.1

Release Highlights

This release focuses on three major themes: storage intelligence, observability, and data protection.

On the storage intelligence side, PowerScale incremental index updates deliver a major performance improvement by only processing changed files after the initial scan, significantly reducing load on storage systems. A new Force Fresh Index option complements this by giving administrators manual control when a full re-index is needed. Extended forecasting horizons in Data Intelligence insights further strengthen long-term planning capabilities.

For integration and observability, a new File Content Delivery API (Public Preview) enables on-demand file retrieval with Snowflake and other platform integrations in mind, while a new Prometheus metrics endpoint allows teams to monitor agent performance within their existing observability stack.

Finally, Strict Scanning Mode adds an important data protection safeguard, ensuring incomplete scans cannot trigger sync operations that might inadvertently delete target files.

New Features & Enhancements

Modern SSO Authentication with PKCE

CloudSoda now supports the OAuth 2.0 Authorization Code flow with PKCE (Proof Key for Code Exchange) for OpenID Connect identity providers such as Okta and Azure AD. This replaces the legacy Implicit grant as the recommended authentication method, aligning with current industry security standards (RFC 9700).

Key improvements:

• Authorization Code Flow with PKCE: Resolves compatibility issues with identity providers that require or enforce PKCE, including Okta configurations that previously could not authenticate.
• Flow Selection in UI: Administrators can now choose between Authorization Code and Implicit flows when configuring identity providers, with Authorization Code as the new recommended default.
• Client Secret Support: Added support for confidential OIDC client configurations, enabling the use of client secrets at the token endpoint for identity providers that require them.
• Backward Compatibility: Existing identity provider configurations using the Implicit flow continue to work unchanged. PKCE parameters are safely ignored by providers that do not require them.

PowerScale Incremental Index Updates

Data Intelligence scanning for Dell PowerScale storage systems now supports incremental index updates using the PowerScale snapshot diff API. After the initial full scan, subsequent index updates only process changed files rather than re-walking the entire file system. This results in significantly faster index refreshes and reduced load on the storage system.

Force Fresh Index for Data Intelligence Scans

A new “Force Fresh Index” option is now available when launching manual scan jobs in Data Intelligence. This allows users to trigger a complete re-index of their storage, bypassing incremental updates when a full refresh is needed, such as after storage migrations or to resolve data discrepancies.

Enhanced Data Intelligence Insights

The Data Intelligence insights cards for both volumes and projects now include extended time range options. Users can now view cost projections and data growth trends across 2-year, 3-year, and 5-year horizons, providing better long-term planning and forecasting capabilities.

File Content Delivery API

A new REST API endpoint enables on-demand retrieval of file content from managed storage systems. The API supports HTTP range requests for partial file retrieval, includes proper content-length and content-type headers, and supports large file streaming. This capability enables integration scenarios such as ingesting file content alongside metadata into platforms like Snowflake.

Agent Prometheus Metrics Endpoint

The CloudSoda agent now exposes a Prometheus-compatible metrics endpoint (/metrics), enabling customers to monitor agent performance and storage system interactions using their existing observability stack. Initial metrics cover SMB and PowerScale/OneFS API requests and responses, including latency, success rates, and request types, providing valuable diagnostics for performance troubleshooting.

Strict Scanning Mode

A new strict scanning mode has been added to the agent and conductor to prevent incomplete scans from being used in sync-type transfers. Previously, if a source scan encountered errors and only partially completed, a sync transfer could inadvertently delete files on the target that appeared missing from the incomplete source scan. Strict mode ensures scan integrity before any sync operations proceed.

Performance Enhancements

• NAS Deployment Optimization: Significant performance improvements for deployments using network-attached storage (NFS). Job logging and statistics operations have been optimized to eliminate I/O bottlenecks, ensuring consistent job performance regardless of storage type.
• S3 Connection Management: The agent now limits the number of concurrent connections per S3 host, preventing resource exhaustion in high-throughput environments.
• PowerScale Error Handling: Improved retry logic for PowerScale (OneFS) API requests, adding resilience against transient storage errors.

Security Enhancements

• Updated Data Intelligence libraries to address known CVEs.
• Upgraded platform infrastructure components, including nginx-ingress (5.3.2) and Go runtime (1.25.7).
• Updated the AWS SDK
• Bundled x.509 trusted root certificates with the Windows agent to avoid resources exhaustion

Bug Fixes

Data Intelligence

• Fixed an issue where fetching projects and reports would fail for all users when a deleted storage was present.
• Resolved a bug where freshly scanned storage would not appear in File Management.
• Fixed search PDF values not reflecting the newly applied filters.
• Resolved an issue where a cancelled task could cause a job to enter a stuck state.

Data Orchestration

• Fixed an issue where PowerScale directory listings would not retrieve pages beyond the first, causing incomplete folder views.
• Fixed a very rare bug that could cause a directory to be skipped during scanning.
• Fixed preview jobs that could sometimes fail to terminate properly.
• Resolved a GCS storage configuration issue where storages could not be created or updated when disabled or when a region was not specified.
• Google Cloud Platform egress price data now supports the new InterConnect Egress fee schedule Google announced.