This document describes how to automatically set default Cloud Storage FUSE values used for high-performance Compute Engine machine types, which are designed to optimize performance for demanding, high-throughput workloads. Values that are manually set at the time of mount will override these defaults.
Machine types
Configuration values are automated for the following high-performance Compute Engine machine types:
| Series type | Machine type |
|---|---|
| A2 machine series | |
a2-megagpu-16g |
|
a2-ultragpu-8g |
|
| A3 machine series | |
a3-edgegpu-8g |
|
a3-highgpu-8g |
|
a3-megagpu-8g |
|
a3-ultragpu-8g |
|
| A4 machine series | |
4-highgpu-8g-lowmem |
|
| TPU v5e | |
ct5l-hightpu-8t |
|
ct5lp-hightpu-8t |
|
| TPU v5p | |
ct5p-hightpu-4t |
|
ct5p-hightpu-4t-tpu |
|
| TPU v6e (Trillium) | |
ct6e-standard-4t |
|
ct6e-standard-4t-tpu |
|
ct6e-standard-8t |
|
ct6e-standard-8t-tpu |
Automated configuration values
When a supported machine type is detected, Cloud Storage FUSE automatically applies the following configuration values:
| Cloud Storage FUSE configuration file field | Cloud Storage FUSE CLI option | Automated configuration value |
|---|---|---|
metadata-cache.negative-ttl-secs |
--metadata-cache-negative-ttl-secs |
0 |
metadata-cache.ttl-secs1 |
--metadata-cache-ttl-secs1 |
|
metadata-cache.stat-cache-max-size-mb |
--stat-cache-max-size-mb |
1024 |
metadata-cache.type-cache-max-size-mb |
--type-cache-max-size-mb |
128 |
implicit-dirs |
--implicit-dirs |
true |
file-system.rename-dir-limit |
--rename-dir-limit |
200000 |
1Setting this configuration to -1 significantly boosts
performance by always serving files from the cache. Be aware that this
configuration bypasses consistency checks, which can lead to serving
outdated data. For details on managing data consistency, refer to
Overview of caching in Cloud Storage FUSE.
Further performance tuning
When you use a high-performance Google Cloud machine type, the configuration values detailed on this page are automatically applied. However, you can further fine-tune your machine for optimal performance using the Performance tuning best practices guide.
If you're running training, serving, or checkpointing and JIT cache workloads on Google Kubernetes Engine clusters that use Cloud GPUs or Cloud TPU to access large datasets in Cloud Storage, you can streamline your setup by utilizing pre-configured YAML files to mount your Cloud Storage buckets directly into your pods more efficiently. For more information and instructions on how to use pre-configured GKE YAML files, see Use pre-configured GKE YAML files to optimize Cloud Storage FUSE performance.
What's next
Learn how to tune Cloud Storage FUSE for optimal performance.
Use a pre-configured GKE YAML file to configure tuning best practices.