Use Cloud Storage FUSE file caching

The Cloud Storage FUSE file cache is a client-based read cache that lets repeat file reads to be served from a faster cache storage of your choice. This page describes how to enable and use Cloud Storage FUSE file caching.

File caching behavior

The file cache is disabled by default and can be enabled and configured using a Cloud Storage FUSE configuration file. You can control caching behavior using the following fields:

  • max-size-mb: controls the maximum capacity in your cache directory that cached data can occupy. By default, the max-size-mb field is set to let cached data grow until it occupies all the available capacity in your cache directory. For more information about the eviction of cached data, see Eviction.

  • cache-dir: specifies a directory for storing file cache data. Note that specifying a cache directory is a prerequisite for enabling the file cache.

  • ttl-secs: determines when cached data becomes stale and needs to be refreshed from Cloud Storage. By default, the ttl-secs field is set to expire and refresh from Cloud Storage after 60 seconds of inactivity. We recommend increasing this value. To learn how to control cache data invalidation, see Configuring cache data invalidation.

  • enable-parallel-downloads: is enabled automatically when file caching is enabled. Parallel downloads improve read performance by using multiple workers to download multiple parts of a file in parallel using the file cache directory as a prefetch buffer. To learn more about parallel downloads and its supporting settings, see Parallel downloads.

Before you begin

The file cache requires a directory path to be used to cache files. You can create a new directory on an existing file system or create a new file system on provisioned storage. If you are provisioning new storage to be used, use the following instructions to create a new file system:

  1. For Google Cloud Hyperdisk, see Create a new Google Cloud Hyperdisk volume.

  2. For Persistent Disk, see Create a new Persistent Disk volume.

  3. For Local SSDs, see Add a Local SSD to your VM.

  4. For in-memory RAM disks, see Creating in-memory RAM disks.

Enable and configure file caching behavior

Enable and configure file caching using either a Cloud Storage FUSE configuration file or Cloud Storage FUSE CLI options:

  1. Specify the cache directory you want to use using the file-cache:cache-dir field or the --cache-dir option, which are used to enable the file cache.

  2. Optional: if you want to limit the total capacity the Cloud Storage FUSE cache can use within its mounted directory, adjust the max-size-mb property, which is automatically set to -1 when you set the cache-dir property.

  3. Optional: increase the TTL of cached entries by setting the ttl-secs option to a value based on the expected time between repeat reads while balancing consistency needs. We recommend that you set the ttl-secs value to as high as your workload lets you. For more information about setting a TTL for cached entries, see Time to live.

    For example, the following configuration file enables file caching, stat caching, and type caching with a TTL of 3600 seconds and the cache directory set to /path/to/a/directory/.

    file-cache:
  max-size-mb: -1
  cache-file-for-range-read: false

metadata-cache:
  stat-cache-max-size-mb: 32
  ttl-secs: 3600
  type-cache-max-size-mb: 4

cache-dir: /path/to/a/directory

  4. Optional: configure stat caching and type caching using the metadata-cache. To learn more about stat and type caches, see Overview of type caching or Overview of stat caching.

  5. Manually run the ls -R command on your mounted bucket before you run your workload to pre-populate metadata to ensure the type cache populates ahead of the first read in a faster, batched method. For more information about how to improve first time read performance, see Improve first-time reads.

Once you enable file caching, parallel downloads are enabled automatically. To learn more about parallel downloads, which lets you use multiple workers to download multiple parts of a file in parallel, see Parallel downloads.

Parallel downloads

Parallel downloads can improve read performance by using multiple workers to download multiple parts of a file in parallel using the file cache directory as a prefetch buffer. The enable-parallel-downloads property is automatically set to true when you enable file caching. We recommend using parallel downloads for read scenarios that load large files such as model serving, checkpoint restores, and training on large objects.

Configure supporting properties for parallel downloads

You can optionally configure the following supporting properties for parallel downloads:

  • parallel-downloads-per-file: the number of maximum workers that can be spawned per file to download the object from Cloud Storage into the file cache. The default value is 16.

  • max-parallel-downloads: the number of maximum workers that can be spawned at any given time across all file download jobs. The default is set to twice the number of CPU cores on your machine. To specify no limit, enter a value of -1.

  • download-chunk-size-mb: the size of each read request in MiB that each worker makes to Cloud Storage when downloading the object into the file cache. The default size is 200 MiB. Note that a parallel download is only triggered if the file being read is the specified size.

Disable parallel downloads

To disable parallel downloads, set the enable-parallel-downloads property to false in your Cloud Storage FUSE configuration file.

What's next