Use Cloud Storage FUSE file caching

The Cloud Storage FUSE file cache feature is a client-based read cache that lets repeat file reads to be served from a faster cache storage of your choice. This page describes how to enable and use Cloud Storage FUSE file caching. For an overview of file caching, stat caching, or type caching, see Overview of caching.

Before you begin

The file cache requires a directory path to be used to cache files. You can create a new directory on an existing file system or create a new file system on provisioned storage. If you are provisioning new storage to be used, use the following instructions to create a new file system:

  1. For Google Cloud Hyperdisk, see Create a new Google Cloud Hyperdisk volume.

  2. For Persistent Disk, see Create a new Persistent Disk volume.

  3. For Local SSDs, see Add a Local SSD to your VM.

  4. For in-memory RAM disks, see Creating in-memory RAM disks.

Enable and configure caching behavior

  1. Enable and configure file caching by using the file-cache field in a Cloud Storage FUSE configuration file and specify the cache directory you want to use in the cache-dir field. The file cache is disabled by default. Note that you enable file caching by passing a directory to cache-dir field.

  2. Optional: configure stat caching and type caching by using the metadata-cache field in a configuration file. To learn more about stat and type caches, see Overview of type caching or Overview of stat caching.

  3. Optional: increase the TTL of cached entries by setting the ttl-secs option to a value based on the expected time between repeat reads while balancing consistency needs. We recommend that you set the ttl-secs value to as high as your workload lets you. You can configure the TTL in a Cloud Storage FUSE configuration file. For more information about setting a TTL for cached entries, see Time to live.

    For example, the following configuration file enables file caching, stat caching, and type caching with a TTL of 3600 seconds and the cache directory set to /path/to/a/directory/. Note that max-size-mb is set to -1, which configures the file cache to use all available capacity.

    file-cache:
  max-size-mb: -1
  cache-file-for-range-read: false

metadata-cache:
  stat-cache-max-size-mb: 32
  ttl-secs: 3600
  type-cache-max-size-mb: 4

cache-dir: /path/to/a/directory

  4. Optional: accelerate large file reads, including first-time reads, by enabling the enable-parallel-downloads property which uses multiple workers to download a large file in parallel using the file cache directory as a prefetch buffer. For more information about parallel downloads and how to configure its supporting properties, see Improve read performance using parallel downloads.

  5. Manually run the ls -R command on your mounted bucket before you run your workload to pre-populate metadata to ensure the type cache populates ahead of the first read in a faster, batched method. For more information about how to improve first time read performance, see Improve first-time reads.

Download multiple files in parallel using parallel downloads

You can improve read performance by enabling and configuring the parallel download feature, which uses multiple workers to download a file in parallel using the file cache directory as a prefetch buffer. We recommend using parallel downloads for single-threaded read scenarios that load large files such as model serving and checkpoint restores.

Before you enable parallel downloads, consider the following:

  • If your application does high read parallelism over eight threads, you might experience a slight performance degradation.

  • We don't recommend using parallel downloads for training workloads because of their high read parallelism.

  • To use parallel downloads, you must first enable and configure the file cache.

  • The file being read must fit within the file cache directory's available capacity which can be controlled using the max-size-mb property.

Configure parallel downloads

  1. In a Cloud Storage FUSE configuration file, set the enable-parallel-downloads property to true and optionally configure the following supporting settings:

    • parallel-downloads-per-file: the number of maximum workers that can be spawned per file to download the object from Cloud Storage into the file cache. The default value is 16.

    • max-parallel-downloads: the number of maximum workers that can be spawned at any given time across all file download jobs. The default is set to twice the number of CPU cores on your machine. To specify no limit, enter a value of -1.

    • download-chunk-size-mb: the size of each read request in MiB that each worker makes to Cloud Storage when downloading the object into the file cache. The default size is 50 MiB. Note that a parallel download is only triggered if the file being read is the specified size.

What's next