Transfer data between file systems

This page shows you how to transfer data between two POSIX file systems. Common use cases include:

  • Burst to cloud and Hybrid HPC: Quickly transfer large data sets from on-premises to the cloud for processing.
  • Migration and sync to Filestore: Migrate or sync data from an on-premises file system to Filestore.
  • Managed file transfer: Securely and reliably transfer data between data centers or between two in-cloud file systems.

Transfer performance guidelines

The following guidelines can help to maximize performance during file system to file system transfers.

Agent deployment

In general, we recommend using three agents in each of the source and destination agent pools. Monitor the transfer and add more agents if required. Each agent needs 4 vCPU and 8 GiB RAM.

If you are migrating to a Filestore instance, Filestore recommends using the n2-standard-8 instance type for each agent. Specify nconnect=2 when you mount the instance to a Compute Engine VM. Refer to Filestore's performance guidelines for more information on how to optimize and test instance performance.

Transferring large numbers of small files

For better performance when transferring a large number of small files, we recommend splitting the files into multiple directories and avoiding a single directory with millions of files.

Before you begin

Before you can perform the tasks described on this page, complete the prerequisite steps.

Create agent pools and install agents

For file system to file system transfers, you need to create agent pools and agents for both the source and destination file systems. Agents for the source agent pool need to be installed on machines or VMs that have access to the source file system. Agents for the destination agent pool need to be installed on machines or VMs that have access to the destination file system.

Don't include sensitive information such as personally identifiable information (PII) or security data in your agent ID prefix or agent pool name. Resource names may be propagated to the names of other Google Cloud resources and may be exposed to Google-internal systems outside of your project.

Create a source agent pool

Create a source agent pool using one of the following methods:

gcloud CLI

Create a source agent pool by running:

gcloud transfer agent-pools create SOURCE_AGENT_POOL

Replace SOURCE_AGENT_POOL with the name that you want to give to the source agent pool.

Google Cloud console

  1. In the Google Cloud console, go to the Agent pools page.

    Go to Agent pools

    The Agent pools page is displayed, listing your existing agent pools.

  2. Click Create another pool.

  3. Enter a name for the pool.

  4. Click Create.

Install agents for the source agent pool

Install agents for the source agent pool on a machine or VM that has access to the source file system:

gcloud CLI

Install agents for the source agent pool by running:

gcloud transfer agents install --pool=SOURCE_AGENT_POOL --count=NUMBER_OF_AGENTS \
  --mount-directories=MOUNT_DIRECTORIES

Replace the following:

  • SOURCE_AGENT_POOL with the name of the source agent pool.
  • NUMBER_OF_AGENTS with the number of agents that you want to install for the source agent pool. To determine the optimal number of agents for your environment, see Agent requirements and best practices.
  • MOUNT_DIRECTORIES with a comma-separated list of directories on the source file system from which to copy. Omitting this flag mounts the entire file system, which could present a security risk.

Google Cloud console

  1. In the Google Cloud console, go to the Agent pools page.

    Go to Agent pools

    The Agent pools page is displayed, listing your existing agent pools.

  2. Click the name of the source agent pool that you just created.

  3. Under the Agents tab, click Install agent.

  4. Follow the instructions in Google Cloud console to install Docker and start the agent.

Create a destination agent pool and install agents

Repeat the preceding steps to create a destination agent pool and install agents.

Create a Cloud Storage bucket as an intermediary

File system to file system transfers require a Cloud Storage bucket as an intermediary for the data transfer.

  1. Create a Cloud Storage Standard class bucket with the following settings:

    • Encryption: You can specify a customer-managed encryption key (CMEK). Otherwise, a Google-owned and Google-managed encryption key is used.
    • Object Versioning, Bucket Lock, and default object holds: Keep these features disabled.

  2. Grant permissions and roles using one of the following methods:

    • Grant the Storage Transfer Service service account the Storage Admin role (roles/storage.admin) for the bucket.

    • Use gcloud transfer authorize to authorize your account for all Storage Transfer Service features. This command grants project-wide Storage Admin permissions: