Create virtual repositories

This page describes how to create Artifact Registry virtual repositories.

Virtual repositories act as a single access point to download, install, or deploy artifacts in the same format from one or more upstream repositories. An upstream repository can be a Artifact Registry standard or remote repository.

The other repository modes are:

• Standard: The default repository mode. You upload or publish artifacts such as private packages directly to standard repositories. Although you can download directly from individual standard repositories, accessing groups of repositories with a virtual repository simplifies tool configuration.
• Remote (language package repositories only): A pull through cache for artifacts in public repositories such as Maven Central or PyPI. It acts as a proxy for the public repositories so that you have more control over your external dependencies.

For more information on how virtual repositories work, see the Virtual repository overview.

Before you begin

Enable Artifact Registry, including enabling the Artifact Registry API and installing the latest version of Google Cloud CLI.

Required roles

To get the permissions that you need to manage repositories, ask your administrator to grant you the following IAM roles:

• Create virtual repositories and grant access to individual repositories: Artifact Registry Administrator (roles/artifactregistry.admin) on the Google Cloud project
• Grant access to repositories at the project level: Project IAM Admin (roles/resourcemanager.projectIamAdmin) on the Google Cloud project
• Grant access to repositories at the folder level: Folder Admin (roles/resourcemanager.folderAdmin) on the folder
• Grant access to repositories at the organization level: Organization Admin (roles/resourcemanager.organizationAdmin) on the Google Cloud organization

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

If upstream repositories are in other projects, an administrator in the projects with upstream repositories must have permissions to manage access to those repositories.

Create upstream repositories

Before setting up a virtual repository, create the upstream repositories that you want to make available with the virtual repository.

1. Create the upstream repositories in the same Google Cloud region where you plan to create the virtual repository.

   • Create a standard repository
   • Create a remote repository

2. Virtual repositories use the Artifact Registry Service Agent to authenticate to upstream repositories. You must grant the service agent read access to upstream repositories.

   You can grant the Artifact Registry Reader role to the Artifact Registry service agent at the project level so that you do not need to grant access to individual upstream repositories.

   In each project with upstream repositories, run the following command:

   gcloud projects add-iam-policy-binding UPSTREAM_PROJECT_ID \
       --member serviceAccount:service-VIRTUAL-REPO-PROJECT_NUMBER@gcp-sa-artifactregistry.iam.gserviceaccount.com \
       --role roles/artifactregistry.reader
   

   Where

   • UPSTREAM_PROJECT_ID is the ID of the Google Cloud project with the upstream repository.
   • VIRTUAL_REPO_PROJECT_NUMBER is the project number of the Google Cloud project for your virtual repository.

   Alternatively, if you want to grant more narrowly-scoped role to the service agent, add the permission artifactregistry.repositories.readViaVirtualRepository to a custom role and then grant that role to the service agent.

   For more information about granting permissions, see the access control documentation.

Create a virtual repository

When you create a repository, you must configure the following settings that cannot be changed after the repository is created:

• Artifact format.
• Repository mode, if multiple modes are available for the selected format.
• Repository location.
• Encryption with Google-owned and Google-managed keys or customer-managed encryption keys. Artifact Registry uses Google-owned and Google-managed keys by default.

Artifact Registry enforces organization policy constraints that require CMEK to encrypt resources or limit which Cloud KMS keys can be used for CMEK protection.

Create a virtual repository using Google Cloud console

1. Open the Repositories page in the Google Cloud console.

   Open the Repositories page

2. Click Create Repository.

3. Specify the repository name. For each repository location in a project, repository names must be unique.

4. Select the repository format.

5. Under Repository mode, select Virtual.

6. Under Virtual Repository Upstreams, click Add Upstream to add an upstream repository.

   • An upstream repository can be a standard, remote, or virtual repository.
   • An upstream repository must be in the same location as the virtual repository, but can be in a different Google Cloud project.
   • An upstream repository must be the same artifact format as the virtual repository.

   Specify a priority and policy name for each upstream repository. Higher priority values take precedence over lower values. Leave space between priority values so that there is flexibility to add upstream repositories with priorities that are in between existing values.

7. Under Location Type, choose the location for the repository:

   1. Choose the location type: Region or Multi-Region. The list of locations changes to reflect your selection.

   2. In the Region or Multi-region list, select a location.

   For information about location types and supported locations, see Repository locations

8. Add a description for the repository. Descriptions help to identify the purpose of the repository and the kind of artifacts it contains.

   Do not include sensitive data, since repository descriptions are not encrypted.

9. If you want to use labels to organize your repositories, click Add Label and enter the key-value pair for the label. You can add, edit, or remove labels after you create the repository.

10. In the Encryption section, choose the encryption mechanism for the repository.

    • Google-managed key - Encrypt repository content with a Google-owned and Google-managed key.
    • Customer-managed key - Encrypt repository content with a key that you control through Cloud Key Management Service. For key setup instructions, see Setting up CMEK for repositories.

11. Click Create.

Artifact Registry creates the repository and adds it to the list of repositories.

After you have created the repository:

• Grant access to the repository.

• Configure Docker, package managers, and other third-party clients to authenticate to repositories.

  • Container images: Docker, Helm
  • Language packages: Java, Node.js, Python, Go
  • Other: Kubeflow Pipeline templates

Create a virtual repository using gcloud CLI

To create a virtual repository, you define your upstream repositories in a policy file and then create the repository using the policy file.

1. Create an upstream policy file in JSON format with the following settings:

   [{
   "id" : "POLICY_ID",
   "repository" : "UPSTREAM_REPOSITORY_RESOURCE_NAME",
   "priority" : PRIORITY
   }, {...}]
   

   • POLICY_ID is a name for the policy. Each upstream repository in the policy must have a unique policy ID.

   • UPSTREAM_REPO_RESOURCE_NAME is the name of the upstream repository in the format:

     projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY
     

     • An upstream repository can be a standard, remote, or virtual repository.
     • An upstream repository must be in the same location as the virtual repository, but can be in a different Google Cloud project.
     • An upstream repository must be the same artifact format as the virtual repository.

   • PRIORITY is an integer that indicates the priority for the upstream repository. Higher priority values take precedence over lower values.

   The following example shows a policy file named policies.json with four upstream repositories. upstream1 has the highest value for priority, so Artifact Registry will search it first. upstream-test has the lowest priority value, so Artifact Registry will search it last. upstream2 and upstream3 have the same priority. If Artifact Registry finds a requested package in repositories with the same priority, it can serve the package from either repository.

   [{
   "id" : "upstream1",
   "repository" : "projects/myproject/locations/us-west1/repositories/python-repo1",
   "priority" : 100
   }, {
   "id" : "upstream2",
   "repository" : "projects/myproject/locations/us-west1/repositories/python-repo2",
   "priority" : 80
   }, {
   "id" : "upstream3",
   "repository" : "projects/myproject/locations/us-west1/repositories/python-repo2",
   "priority" : 80
   }, {
   "id" : "upstream-test",
   "repository" : "projects/myproject/locations/us-west1/repositories/python-test-repo",
   "priority" : 20
   }]
   

   This example also leaves space between the priority values. These gaps provide flexibility to add upstream repositories with priorities that are in between existing values. For example, you can add an upstream-3 repository with a priority of 70 so that Artifact Registry searches it before upstream-test.

   Consider storing your upstream policy file in your source control system so that you can reuse it later to make changes to upstream repository settings and track historical changes.

2. In the directory with your upstream policy file, run the following command to create the virtual repository:

   gcloud artifacts repositories create VIRTUAL-REPOSITORY-NAME \
       --project=PROJECT_ID \
       --repository-format=FORMAT \
       --mode=virtual-repository \
       --location=LOCATION \
       --description="DESCRIPTION" \
       --upstream-policy-file=UPSTREAM_POLICY_FILE
   

   Replace the following:

   • VIRTUAL-REPOSITORY-NAME is the name of the virtual repository.
   • PROJECT_ID is the ID of the project where you are creating the virtual repository. If this flag is omitted, the current or default project is used.
   • FORMAT is the format of the repository. The following values are available:
     • apt (private preview)
     • docker
     • maven
     • npm
     • python
     • yum (private preview)

   • LOCATION is the regional or multi-regional location for the repository. You can omit this flag if you set a default. To view a list of supported locations, run the command:

     gcloud artifacts locations list
     

   • DESCRIPTION is an optional description of the repository. Do not include sensitive data, since repository descriptions are not encrypted.

   • UPSTREAM_POLICY_FILE is a path to the upstream policy file.

   For example the following command creates a virtual repository named virtual-repo for Python packages in the region us-west1 with settings in policies.json.

   gcloud artifacts repositories create virtual-repo \
   --repository-format=python \
   --mode=virtual-repository \
   --location=us-west1 \
   --description="Python virtual repository" \
   --upstream-policy-file=policies.json
   

Artifact Registry creates the virtual repository.

Create a virtual repository using Terraform

Use the google_artifact_registry_repository resource to create repositories. terraform-provider-google version 5.0.0 or newer is required.

If you are new to using Terraform for Google Cloud, see the Get Started - Google Cloud page on the HashiCorp website.

The following example defines the provider, a virtual repository with the Terraform resource name my-repo and an upstream repository with the Terraform resource name my-repo-upstream.

provider "google" {
    project = "PROJECT-ID"
}
resource "google_artifact_registry_repository" "my-repo-upstream" {
  location      = "UPSTREAM_LOCATION"
  repository_id = "UPSTREAM_REPOSITORY_ID"
  description   = "UPSTREAM_REPOSITORY_DESCRIPTION"
  format        = "docker"
}

resource "google_artifact_registry_repository" "my-repo" {
  depends_on    = []
  location      = "LOCATION"
  repository_id = "REPOSITORY_ID"
  description   = "VIRTUAL_REPOSITORY_DESCRIPTION"
  format        = "docker"
  kms_key_name = "KEY"
  mode          = "VIRTUAL_REPOSITORY"
  virtual_repository_config {
    upstream_policies {
      id          = "UPSTREAM_REPOSITORY_ID"
      repository  = google_artifact_registry_repository.my-repo-upstream.id
      priority    = PRIORITY
    }
  }
}

provider "google" {
    project = "PROJECT-ID"
}
resource "google_artifact_registry_repository" "my-repo-upstream" {
  location      = "UPSTREAM_LOCATION"
  repository_id = "UPSTREAM_REPOSITORY_ID"
  description   = "UPSTREAM_REPOSITORY_DESCRIPTION"
  format        = "maven"
}

resource "google_artifact_registry_repository" "my-repo" {
  depends_on    = []
  location      = "LOCATION"
  repository_id = "REPOSITORY_ID"
  description   = "VIRTUAL_REPOSITORY_DESCRIPTION"
  format        = "maven"
  kms_key_name = "KEY"
  mode          = "VIRTUAL_REPOSITORY"
  virtual_repository_config {
    upstream_policies {
      id          = "UPSTREAM_REPOSITORY_ID"
      repository  = google_artifact_registry_repository.my-repo-upstream.id
      priority    = PRIORITY
    }
  }
}

provider "google" {
project = "my-project"
}

resource "google_artifact_registry_repository" "my-repo" {
provider = google-beta

location = "us-central1"
repository_id = "my-repo"
description = "Maven repository"
format = "MAVEN"
maven_config {
  version_policy = "RELEASE"
}
}

provider "google" {
    project = "PROJECT-ID"
}
resource "google_artifact_registry_repository" "my-repo-upstream" {
  location      = "UPSTREAM_LOCATION"
  repository_id = "UPSTREAM_REPOSITORY_ID"
  description   = "UPSTREAM_REPOSITORY_DESCRIPTION"
  format        = "npm"
}

resource "google_artifact_registry_repository" "my-repo" {
  depends_on    = []
  location      = "LOCATION"
  repository_id = "REPOSITORY_ID"
  description   = "VIRTUAL_REPOSITORY_DESCRIPTION"
  format        = "npm"
  kms_key_name = "KEY"
  mode          = "VIRTUAL_REPOSITORY"
  virtual_repository_config {
    upstream_policies {
      id          = "UPSTREAM_REPOSITORY_ID"
      repository  = google_artifact_registry_repository.my-repo-upstream.id
      priority    = PRIORITY
    }
  }
}

provider "google" {
    project = "PROJECT-ID"
}
resource "google_artifact_registry_repository" "my-repo-upstream" {
  location      = "UPSTREAM_LOCATION"
  repository_id = "UPSTREAM_REPOSITORY_ID"
  description   = "UPSTREAM_REPOSITORY_DESCRIPTION"
  format        = "python"
}

resource "google_artifact_registry_repository" "my-repo" {
  depends_on    = []
  location      = "LOCATION"
  repository_id = "REPOSITORY_ID"
  description   = "VIRTUAL_REPOSITORY_DESCRIPTION"
  format        = "python"
  kms_key_name = "KEY"
  mode          = "VIRTUAL_REPOSITORY"
  virtual_repository_config {
    upstream_policies {
      id          = "UPSTREAM_REPOSITORY_ID"
      repository  = google_artifact_registry_repository.my-repo-upstream.id
      priority    = PRIORITY
    }
  }
}

Artifact Registry creates your repository. Run the following command to view a description of the repository:

gcloud artifacts repositories describe REPOSITORY \
    --location=LOCATION

After you have created the repository:

• Grant access to the repository.

• Configure Docker, package managers, and other third-party clients to authenticate to repositories.

  • Container images: Docker, Helm
  • Language packages: Java, Node.js, Python, Go
  • OS packages: Debian, RPM
  • Other: Kubeflow Pipeline templates

Update upstream repositories

You can change the list of upstream repositories or adjust the priority of upstream repositories.

Edit repository descriptions

You can change the repository description from Google Cloud console or the gcloud CLI.

gcloud artifacts repositories update REPOSITORY \
    --project=PROJECT \
    --location=LOCATION \
    --description="DESCRIPTION"