Upload all files in a directory
Stay organized with collections
Save and categorize content based on your preferences.
Use Transfer Manager to upload all of the files in a directory with concurrency.
Explore further
For detailed documentation that includes this code sample, see the following:
Code sample
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],[],[],[],null,["# Upload all files in a directory\n\nUse Transfer Manager to upload all of the files in a directory with concurrency.\n\nExplore further\n---------------\n\n\nFor detailed documentation that includes this code sample, see the following:\n\n- [Host a static website](/storage/docs/hosting-static-website)\n- [Hosting a static website using HTTP](/storage/docs/hosting-static-website-http)\n- [Upload objects from a file system](/storage/docs/uploading-objects)\n\nCode sample\n-----------\n\n### Java\n\n\nFor more information, see the\n[Cloud Storage Java API\nreference documentation](https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/overview).\n\n\nTo authenticate to Cloud Storage, set up Application Default Credentials.\nFor more information, see\n\n[Set up authentication for client libraries](/storage/docs/authentication#client-libs).\n\n import com.google.cloud.storage.transfermanager.https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.ParallelUploadConfig.html;\n import com.google.cloud.storage.transfermanager.https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.TransferManager.html;\n import com.google.cloud.storage.transfermanager.https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.TransferManagerConfig.html;\n import com.google.cloud.storage.transfermanager.https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.UploadResult.html;\n import java.io.IOException;\n import java.nio.file.Files;\n import java.nio.file.Path;\n import java.util.ArrayList;\n import java.util.List;\n import java.util.stream.Stream;\n\n class UploadDirectory {\n\n public static void uploadDirectoryContents(String bucketName, Path sourceDirectory)\n throws IOException {\n https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.TransferManager.html transferManager = https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.TransferManagerConfig.html.newBuilder().build().https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.TransferManagerConfig.html#com_google_cloud_storage_transfermanager_TransferManagerConfig_getService__();\n https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.ParallelUploadConfig.html parallelUploadConfig =\n https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.ParallelUploadConfig.html.newBuilder().setBucketName(bucketName).build();\n\n // Create a list to store the file paths\n List\u003cPath\u003e filePaths = new ArrayList\u003c\u003e();\n // Get all files in the directory\n // try-with-resource to ensure pathStream is closed\n try (Stream\u003cPath\u003e pathStream = Files.walk(sourceDirectory)) {\n pathStream.filter(Files::isRegularFile).forEach(filePaths::add);\n }\n List\u003cUploadResult\u003e results =\n transferManager.https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.TransferManager.html#com_google_cloud_storage_transfermanager_TransferManager_uploadFiles_java_util_List_java_nio_file_Path__com_google_cloud_storage_transfermanager_ParallelUploadConfig_(filePaths, parallelUploadConfig).https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.UploadJob.html#com_google_cloud_storage_transfermanager_UploadJob_getUploadResults__();\n for (https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.transfermanager.UploadResult.html result : results) {\n System.out.println(\n \"Upload for \"\n + result.getInput().getName()\n + \" completed with status \"\n + result.getStatus());\n }\n }\n }\n\n### Node.js\n\n\nFor more information, see the\n[Cloud Storage Node.js API\nreference documentation](https://cloud.google.com/nodejs/docs/reference/storage/latest).\n\n\nTo authenticate to Cloud Storage, set up Application Default Credentials.\nFor more information, see\n\n[Set up authentication for client libraries](/storage/docs/authentication#client-libs).\n\n /**\n * TODO(developer): Uncomment the following lines before running the sample.\n */\n // The ID of your GCS bucket\n // const bucketName = 'your-unique-bucket-name';\n\n // The local directory to upload\n // const directoryName = 'your-directory';\n\n // Imports the Google Cloud client library\n const {Storage, TransferManager} = require('https://cloud.google.com/nodejs/docs/reference/storage/latest/overview.html');\n\n // Creates a client\n const storage = new Storage();\n\n // Creates a transfer manager client\n const transferManager = new https://cloud.google.com/nodejs/docs/reference/storage/latest/storage/transfermanager.html(storage.bucket(bucketName));\n\n async function uploadDirectoryWithTransferManager() {\n // Uploads the directory\n await transferManager.https://cloud.google.com/nodejs/docs/reference/storage/latest/storage/transfermanager.html(directoryName);\n\n console.log(`${directoryName} uploaded to ${bucketName}.`);\n }\n\n uploadDirectoryWithTransferManager().catch(console.error);\n\n### Python\n\n\nFor more information, see the\n[Cloud Storage Python API\nreference documentation](https://cloud.google.com/python/docs/reference/storage/latest).\n\n\nTo authenticate to Cloud Storage, set up Application Default Credentials.\nFor more information, see\n\n[Set up authentication for client libraries](/storage/docs/authentication#client-libs).\n\n def upload_directory_with_transfer_manager(bucket_name, source_directory, workers=8):\n \"\"\"Upload every file in a directory, including all files in subdirectories.\n\n Each blob name is derived from the filename, not including the `directory`\n parameter itself. For complete control of the blob name for each file (and\n other aspects of individual blob metadata), use\n transfer_manager.upload_many() instead.\n \"\"\"\n\n # The ID of your GCS bucket\n # bucket_name = \"your-bucket-name\"\n\n # The directory on your computer to upload. Files in the directory and its\n # subdirectories will be uploaded. An empty string means \"the current\n # working directory\".\n # source_directory=\"\"\n\n # The maximum number of processes to use for the operation. The performance\n # impact of this value depends on the use case, but smaller files usually\n # benefit from a higher number of processes. Each additional process occupies\n # some CPU and memory resources until finished. Threads can be used instead\n # of processes by passing `worker_type=transfer_manager.THREAD`.\n # workers=8\n\n from pathlib import Path\n\n from google.cloud.storage import https://cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.client.Client.html, https://cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.transfer_manager.html\n\n storage_client = Client()\n bucket = storage_client.https://cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.client.Client.html#google_cloud_storage_client_Client_bucket(bucket_name)\n\n # Generate a list of paths (in string form) relative to the `directory`.\n # This can be done in a single list comprehension, but is expanded into\n # multiple lines here for clarity.\n\n # First, recursively get all files in `directory` as Path objects.\n directory_as_path_obj = Path(source_directory)\n paths = directory_as_path_obj.rglob(\"*\")\n\n # Filter so the list only includes files, not directories themselves.\n file_paths = [path for path in paths if path.is_file()]\n\n # These paths are relative to the current working directory. Next, make them\n # relative to `directory`\n relative_paths = [path.relative_to(source_directory) for path in file_paths]\n\n # Finally, convert them all to strings.\n string_paths = [str(path) for path in relative_paths]\n\n print(\"Found {} files.\".format(len(string_paths)))\n\n # Start the upload.\n results = https://cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.transfer_manager.html.https://cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.transfer_manager.html(\n bucket, string_paths, source_directory=source_directory, max_workers=workers\n )\n\n for name, result in zip(string_paths, results):\n # The results list is either `None` or an exception for each filename in\n # the input list, in order.\n\n if isinstance(result, Exception):\n print(\"Failed to upload {} due to exception: {}\".format(name, result))\n else:\n print(\"Uploaded {} to {}.\".format(name, bucket.name))\n\nWhat's next\n-----------\n\n\nTo search and filter code samples for other Google Cloud products, see the\n[Google Cloud sample browser](/docs/samples?product=storage)."]]