CloudSoda Release 25.8

Major Features

Storage Ignore Functionality (Data Orchestration & Data Intelligence)

• Storage-Level Ignore Rules: Each storage can now have an optional list of directory paths or file names that will be ignored during scanning, walking, browsing, and data movement operations. This feature allows administrators to exclude sensitive directories, system folders, or performance-intensive paths (such as .snapshot directories on PowerScale) from all CloudSoda operations for both security and efficiency reasons. These ignore rules cannot be overridden at the job level, ensuring consistent security enforcement across the platform.

Data Orchestration Enhancements

Enhanced Job Error Handling

• Resilient Data Movement Job Processing: Changed the behavior of jobs to not immediately fail when an error is encountered during the scanning phase. Instead, errors will be evaluated according to the job's failure conditions just as they would be if they had occurred during the transfer phase. This addresses issues with large dataset migrations where entire jobs would fail due to single file issues like illegal characters, permission problems, or corrupted files.
• Warning Status Support: Introduced new job completion status "finished with errors" that allows jobs to complete successfully while maintaining visibility into problematic files. Jobs now use failure rules as thresholds to determine when a job should actually fail versus complete with warnings.

PowerScale Performance & Stability

• Connection Handling: Resolved critical issue where PowerScale clusters would reset connections when handling high volumes of simultaneous write operations. Implemented throttling to eliminate connection resets while maintaining data transfer reliability.
• Symbolic Link Handling: On PowerScale, due to API limitations, symbolically linked files and directories may still appear in browsing views but will be excluded from job execution to prevent unintended behavior. Symbolic link targets are still correctly identified and treated as directories or files based on their actual targets when resolution is possible.

Data Intelligence Enhancements

Performance Optimizations

• Deep Pagination Optimization: Replaced legacy pagination system with modern approach, significantly improving performance for large dataset queries, report generation, and CSV exports. This eliminates timeout issues and reduces processing time for extensive data analysis operations.
• Report Engine: Dramatically improved duplicate file report performance by optimizing database queries and implementing parallel processing. This results in significantly faster report generation for large datasets.
• Project Management: Redesigned project data handling to improve performance on deployments with many projects, preventing slowdowns and timeout issues.

Enhanced Scan Reporting & Error Handling

• Improved Data Intelligence Scan Reporting: Significantly enhanced the accuracy and reliability of scan reporting with better error tracking, more detailed scan statistics, and improved visibility into scan operations across all storage types. Reports now provide more accurate error counts that match the actual number of problematic files encountered during scanning operations.
• Better Error Reporting: Enhanced scan job status display with detailed, actionable error messages when scan tasks fail due to storage conflicts. Error messages now provide clear guidance on next steps, such as reviewing previous scan jobs, adjusting scan frequency, or canceling stuck scans.

User Interface Enhancements

• Folder Details: Expanded folder metadata display to include previously missing information: number of files, number of folders, recursive and folder-specific last modified dates, and recursive and folder-specific last access dates. Added helpful tooltips to explain the difference between recursive and folder-only timestamps.
• Localization: Standardized date formatting throughout the Data Intelligence module to match Data Orchestration's localization standards. This ensures consistent user experience and proper regional formatting based on browser locale settings.
• UI Fixes: Resolved multiple interface issues including card spacing inconsistencies in Dashboard, table formatting problems in storage and schedule settings, and various alignment issues throughout the platform. Fixed scrolling behavior in project sections and resolved page refresh issues for specific project and report pages.

Session & Authentication

Session Management: Fixed session handling bug where expired sessions would show generic error messages instead of properly signing out users and redirecting to login. Users are now automatically signed out when sessions expire and prompted to re-authenticate.

Token Refresh: Improved session handling to prevent timeout issues during long-running operations like job monitoring.

Bug Fixes

Data Orchestration

• Date Transfer Rules: Fixed issue where date picker selections for transfer rules (file accessed/modified criteria) were not properly setting values. Also resolved "criterion is required field" errors when manually entering relative date criteria.
• Job Status Display: Resolved issue where newly started jobs would remain stuck showing "Scanning" status despite actually progressing to transfer phases. Status updates now properly reflect real-time job state without requiring page refreshes.
• Directory Browsing: Improved error handling for directory browsing operations to provide better error information to users and prevent interface failures when storage connectivity issues occur.

Data Intelligence

• Volume History: Fixed storage timeline view limitation that was artificially restricting historical data to only the 10 most recent entries. Timeline now displays complete historical data for the selected time period, providing better visibility into storage growth trends.
• Scan Task Timing: Corrected timestamp handling bug where some scan tasks would show incorrect timing information, resulting in confusing scan duration displays.