Migrating Data from Google Workspace with CloudSoda

Many organizations require a streamlined and efficient process for migrating data from the Google Workspace platform (formerly G Suite). By using tools such as CloudSoda in combination with middleware solutions such as rclone, administrators can manage cloud-based storage files and simplify the data migration workflow. Using a Google Workspace admin account, data can be transferred and converted into more universally compatible formats.

Key Considerations for Google Workspace Migration

When planning a Google data migration, keep these considerations in mind:

Agent Types

When using rclone mounts, we recommend using Linux-based agents for optimal compatibility and performance.

Sizing the Linux Agent

Before migrating data from Google Workspace, files must be staged locally. This staging should occur in a virtual file system (VFS) cache with a predefined size. Opting for a larger-than-standard cache can enhance performance for some workloads. Additionally, setting a non-standard buffer size may boost throughput, though it can increase memory consumption. Refer to the Reference Architecture Guide for baseline specifications to support this setup.

Google Accounts

Service accounts with appropriate permissions are essential for migrating data across from multiple Google Workspace users. For large-scale migrations, consider using multiple service accounts with domain-wide delegation to enable parallel transfers. This approach provides greater flexibility, load distribution, and better handling of Drive API rate limits. 

When migrating data from shared drives (team drives), ensure the email address of the service account is added to the member list for the team drive. For individual drives (my drives), domain delegation typically grants the required access. 

Throttling Considerations

Google may throttle API requests during large data migrations. To reduce the impact of throttling, distribute the load across multiple service accounts to ensure consistent throughput. 

Permissions Management

CloudSoda accesses and migrates data using the permissions of the CloudSoda Accessor, applying them during both read and write operations. When writing to the destination storage, CloudSoda uses the Accessor's write permissions. Complex environments may require additional post-migration configuration within CloudSoda. 

Mount Options

Rclone offers several mount options that can influence performance depending on the workload. Key flags include: 

Buffer Size

This flag specifies the amount of memory used to pre-buffer data per open file. Increasing the value, e.g. 512M allows large files to be cached entirely in memory, improving performance for read-heavy operations. Be careful, larger buffers can lead to high memory consumption over time. 

--buffer-size X

Dir Cache Time

This flag controls how long directory structures are cached. The default is 5 minutes. For larger directory trees, increasing this value can reduce the overhead of frequent directory lookups. 

--dir-cache-time X

VFS Cache Mode

This flag enables both read and write operations on the mounted storage. Writes is the only mode that fully supports both reads and writes. It is essential for environments that require file modification or creation. 

--vfs-cache-mode writes

VFS Cache Max Age

This flag determines how long an object stays in the cache since its last access (default is 1 hour). Reducing the value increases the cache turnover, which may benefit environments with frequent file changes. For large files, increasing the max age helps retain them in cache longer, for better performance. 

--vfs-cache-max-age X

VFS Read Chunk Size

This flag specifies the size of data chunks read from the source (default is 128M). Decrease this value for workloads with many small files. Increase it for large files to reduce the number of API requests, which helps avoid hitting Google's request rate limits. 

--vfs-read-chunk-size X  

Read Only

This flag prevents additional modifications to the source during the mount session. It is ideal for migrations or backup scenarios where source integrity is critical. This flag can also be configured via the rclone configuration file. 

--read-only       

Other mount options are available to fine-turn rclone behavior based on your specific workload. For a comprehensive list of options and their default values, refer to Rclone - Mount.

Configuring the Migration Environment and Moving Google Data

Data migration from Google Workspace involves several key steps. First, install and configure rclone to connect to the appropriate Google Drive within your Workspace environment. Next, set up a CloudSoda Accessor and define the destination storage location for the migrated data. Once configured, initiate the transfer by creating either a migration policy or executing a Quick Transfer to move the data to the new storage environment. 

Install and Configure Rclone

1. Install rclone on the CloudSoda Agent. 
   

   sudo -v ; curl https://rclone.org/install.sh | sudo bash

   
   
   For more information, refer to the Rclone - Install documentation.
   
   
2. Configure rclone by connecting a Google Drive to a mount point in the local file system.
   
   1. Make a directory under /mnt to mount the Google drive.
      
      
      
      
   2. Start a rclone configuration session.
      

       rclone config

       
   3. Create a new remote (enter n) and assign a name to the remote.  
      
      
      
      
   4. Select the option that corresponds to Google Drive (or enter drive).
      
      
      
      
      
      
   5. To enhance workload performance, we highly recommend you create your own Client ID instead of using the default rclone Client ID. While not mandatory, this step can significantly improve stability and throughput. For guidance, refer to Making your Own Client ID.
      
      
   6. If you created a custom Client ID, enter it and the associated Secret ID.
      
      
      
      
   7. Assign the drive scope to grant full access the all files in the connected Google account.
      
      
      
      
   8. Optionally, enter the path to the service account file.
      
      
      
      
   9. Exit from advanced configuration (enter n). 
      
      
       
   10. Open the redirect URL in a web browser and log into your Google Workspace account.
       
       
       
       

Mount the Google Drive

Mount the Google Drive containing the data to be migrated.

rclone --vfs-cache-mode writes --vfs-cache-max-size X --vfs-cache-max-age Y 
--buffer-size Z --read-only --OTHER_FLAGS --mount DRIVE: PATH 

Replace the X, Y, Z, OTHER_FLAGS, DRIVE, and PATH entries with appropriate values for your workload.

Create a CloudSoda Accessor

Create a new CloudSoda Accessor to support the data migration, see Creating Accessors.

Create Storage to Access the Google Data

Create new storage to support the data migration, see Creating Storage.

Create a Policy or Quick Transfer to Manage the Data Migration

Create either a new CloudSoda policy or a Quick Transfer to manage data migration from the Google Workspace to the new storage. See Creating a New Policy and Managing Quick Transfers.