This content was last updated in December 2023, and represents the status quo as of the time it was written. Google's security policies and systems may change going forward, as we continually improve protection for our customers.

This document describes the best practices that let you deploy a foundational set of resources in Google Cloud. A cloud foundation is the baseline of resources, configurations, and capabilities that enable companies to adopt Google Cloud for their business needs. A well-designed foundation enables consistent governance, security controls, scale, visibility, and access to shared services across all workloads in your Google Cloud environment. After you deploy the controls and governance that are described in this document, you can deploy workloads to Google Cloud.

The enterprise foundations blueprint (formerly known as the security foundations blueprint) is intended for architects, security practitioners, and platform engineering teams who are responsible for designing an enterprise-ready environment on Google Cloud. This blueprint consists of the following:

• A terraform-example-foundation GitHub repository that contains the deployable Terraform assets.
• A guide that describes the architecture, design, and controls that you implement with the blueprint (this document).

You can use this guide in one of two ways:

• To create a complete foundation based on Google's best practices. You can deploy all the recommendations from this guide as a starting point, and then customize the environment to address your business' specific requirements.
• To review an existing environment on Google Cloud. You can compare specific components of your design against Google-recommended best practices.

Supported use cases

The enterprise foundation blueprint provides a baseline layer of resources and configurations that help enable all types of workloads on Google Cloud. Whether you're migrating existing compute workloads to Google Cloud, building containerized web applications, or creating big data and machine learning workloads, the enterprise foundation blueprint helps you build your environment to support enterprise workloads at scale.

After you deploy the enterprise foundation blueprint, you can deploy workloads directly or deploy additional blueprints to support complex workloads that require additional capabilities.

A defense-in-depth security model

Google Cloud services benefit from the underlying Google infrastructure security design. It is your responsibility to design security into the systems that you build on top of Google Cloud. The enterprise foundation blueprint helps you to implement a defense-in-depth security model for your Google Cloud services and workloads.

The following diagram shows a defense-in-depth security model for your Google Cloud organization that combines architecture controls, policy controls, and detective controls.

The diagram describes the following controls:

• Policy controls are programmatic constraints that enforce acceptable resource configurations and prevent risky configurations. The blueprint uses a combination of policy controls including infrastructure-as-code (IaC) validation in your pipeline and organization policy constraints.
• Architecture controls are the configuration of Google Cloud resources like networks and resource hierarchy. The blueprint architecture is based on security best practices.
• Detective controls let you detect anomalous or malicious behavior within the organization. The blueprint uses platform features such as Security Command Center, integrates with your existing detective controls and workflows such as a security operations center (SOC), and provides capabilities to enforce custom detective controls.

Key decisions

This section summarizes the high-level architectural decisions of the blueprint.

The diagram describes how Google Cloud services contribute to key architectural decisions:

• Cloud Build: Infrastructure resources are managed using a GitOps model. Declarative IaC is written in Terraform and managed in a version control system for review and approval, and resources are deployed using Cloud Build as the continuous integration and continuous deployment (CI/CD) automation tool. The pipeline also enforces policy-as-code checks to validate that resources meet expected configurations before deployment.
• Cloud Identity: Users and group membership are synchronized from your existing identity provider. Controls for user account lifecycle management and single sign-on (SSO) rely on the existing controls and processes of your identity provider.
• Identity and Access Management (IAM): Allow policies (formerly known as IAM policies) allow access to resources and are applied to groups based on job function. Users are added to the appropriate groups to receive view-only access to foundation resources. All changes to foundation resources are deployed through the CI/CD pipeline which uses privileged service account identities.
• Resource Manager: All resources are managed under a single organization, with a resource hierarchy of folders that organizes projects by environments. Projects are labeled with metadata for governance including cost attribution.
• Networking: Network topologies use Shared VPC to provide network resources for workloads across multiple regions and zones, separated by environment, and managed centrally. All network paths between on-premises hosts, Google Cloud resources in the VPC networks, and Google Cloud services are private. No outbound traffic to or inbound traffic from the public internet is permitted by default.
• Cloud Logging: Aggregated log sinks are configured to collect logs relevant for security and auditing into a centralized project for long-term retention, analysis, and export to external systems.
• Organization Policy Service: Organization policy constraints are configured to prevent various high-risk configurations.
• Secret Manager: Centralized projects are created for a team responsible for managing and auditing the use of sensitive application secrets to help meet compliance requirements.
• Cloud Key Management Service (Cloud KMS): Centralized projects are created for a team responsible for managing and auditing encryption keys to help meet compliance requirements.
• Security Command Center: Threat detection and monitoring capabilities are provided using a combination of built-in security controls from Security Command Center and custom solutions that let you detect and respond to security events.

For alternatives to these key decisions, see alternatives.

What's next

• Read about authentication and authorization (next document in this series).

Authentication and authorization

This section introduces how to use Cloud Identity to manage the identities that your employees use to access Google Cloud services.

External identity provider as the source of truth

We recommend federating your Cloud Identity account with your existing identity provider. Federation helps you ensure that your existing account management processes apply to Google Cloud and other Google services.

If you don't have an existing identity provider, you can create user accounts directly in Cloud Identity.

The following diagram shows a high-level view of identity federation and single sign-on (SSO). It uses Microsoft Active Directory, located in the on-premises environment, as the example identity provider.

This diagram describes the following best practices:

• User identities are managed in an Active Directory domain that is located in the on-premises environment and federated to Cloud Identity. Active Directory uses Google Cloud Directory Sync to provision identities to Cloud Identity.
• Users attempting to sign in to Google services are redirected to the external identity provider for single sign-on with SAML, using their existing credentials to authenticate. No passwords are synchronized with Cloud Identity.

The following table provides links to setup guidance for identity providers.

We strongly recommend that you enforce multi-factor authentication at your identity provider with a phishing-resistant mechanism such as a Titan Security Key.

The recommended settings for Cloud Identity aren't automated through the Terraform code in this blueprint. See administrative controls for Cloud Identity for the recommended security settings that you must configure in addition to deploying the Terraform code.

Groups for access control

A principal is an identity that can be granted access to a resource. Principals include Google Accounts for users, Google groups, Google Workspace accounts, Cloud Identity domains, and service accounts. Some services also let you grant access to all users who authenticate with a Google Account, or to all users on the internet. For a principal to interact with Google Cloud services, you must grant them roles in Identity and Access Management (IAM).

To manage IAM roles at scale, we recommend that you assign users to groups based on their job functions and access requirements, then grant IAM roles to those groups. You should add users to groups using the processes in your existing identity provider for group creation and membership.

We don't recommend granting IAM roles to individual users because individual assignments can increase the complexity of managing and auditing roles.

The blueprint configures groups and roles for view-only access to foundation resources. We recommend that you deploy all resources in the blueprint through the foundation pipeline, and that you don't grant roles to users to groups to modify foundation resources outside of the pipeline.

The following table shows the groups that are configured by the blueprint for viewing foundation resources.

As you build your own workloads on top of the foundation, you create additional groups and grant IAM roles that are based on the access requirements for each workload.

We strongly recommend that you avoid basic roles (such as Owner, Editor, or Viewer) and use predefined roles instead. Basic roles are overly permissive and a potential security risk. Owner and Editor roles can lead to privilege escalation and lateral movement, and the Viewer role includes access to read all data. For best practices on IAM roles, see Use IAM securely.

Super admin accounts

Cloud Identity users with the super admin account bypass the organization's SSO settings and authenticate directly to Cloud Identity. This exception is by design, so that the super admin can still access the Cloud Identity console in the event of an SSO misconfiguration or outage. However, it means you must consider additional protection for super admin accounts.

To protect your super admin accounts, we recommend that you always enforce 2-step verification with security keys in Cloud Identity. For more information, see Security best practices for administrator accounts.

Issues with consumer user accounts

If you didn't use Cloud Identity or Google Workspace before you onboarded to Google Cloud, it's possible that your organization's employees are already using consumer accounts that are associated with their corporate email identities to access other Google services such as Google Marketing Platform or YouTube. Consumer accounts are accounts that are fully owned and managed by the individuals who created them. Because those accounts aren't under your organization's control and might include both personal and corporate data, you must decide how to consolidate these accounts with other corporate accounts.

We recommend that you consolidate existing consumer user accounts as part of onboarding to Google Cloud. If you aren't using Google Workspace for all your user accounts already, we recommend blocking the creation of new consumer accounts.

Administrative controls for Cloud Identity

Cloud Identity has various administrative controls that are not automated by Terraform code in the blueprint. We recommend that you enforce each of these best practice security controls early in the process of building your foundation.

What's next

• Read about organization structure (next document in this series).

Organization structure

The root node for managing resources in Google Cloud is the organization. The Google Cloud organization provides a resource hierarchy that provides an ownership structure for resources and attachment points for organization policies and access controls. The resource hierarchy consists of folders, projects, and resources, and it defines the structure and use of Google Cloud services within an organization.

Resources lower in the hierarchy inherit policies such as IAM allow policies and organization policies. All access permissions are denied by default, until you apply allow policies directly to a resource or the resource inherits the allow policies from a higher level in the resource hierarchy.

The following diagram shows the folders and projects that are deployed by the blueprint.

The following sections describe the folders and projects in the diagram.

Folders

The blueprint uses folders to group projects based on their environment. This logical grouping is used to apply configurations like allow policies and organization policies at the folder level and then all resources within the folder inherit the policies. The following table describes the folders that are part of the blueprint.

Projects

The blueprint uses projects to group individual resources based on their functionality and intended boundaries for access control. This following table describes the projects that are included in the blueprint.

Governance for resource ownership

We recommend that you apply labels consistently to your projects to assist with governance and cost allocation. The following table describes the project labels that are added to each project for governance in the blueprint.

Google might occasionally send important notifications such as account suspensions or updates to product terms. The blueprint uses Essential Contacts to send those notifications to the groups that you configure during deployment. Essential Contacts is configured at the organization node and inherited by all projects in the organization. We recommend that you review these groups and ensure that emails are monitored reliably.

Essential Contacts is used for a different purpose than the primarycontact and secondarycontact fields that are configured in project labels. The contacts in project labels are intended for internal governance. For example, if you identify non-compliant resources in a workload project and need to contact the owners, you could use the primarycontact field to find the person or team responsible for that workload.

What's next

• Read about networking (next document in this series).

Networking

Networking is required for resources to communicate within your Google Cloud organization and between your cloud environment and on-premises environment. This section describes the structure in the blueprint for VPC networks, IP address space, DNS, firewall policies, and connectivity to the on-premises environment.

Network topology

The blueprint repository provides the following options for your network topology:

• Use separate Shared VPC networks for each environment, with no network traffic directly allowed between environments.
• Use a hub-and-spoke model that adds a hub network to connect each environment in Google Cloud, with the network traffic between environments gated by a network virtual appliance (NVA).

Choose the dual Shared VPC network topology when you don't want direct network connectivity between environments. Choose the hub-and-spoke network topology when you want to allow network connectivity between environments that is filtered by an NVA such as when you rely on existing tools that require a direct network path to every server in your environment.

Both topologies use Shared VPC as a principal networking construct because Shared VPC allows a clear separation of responsibilities. Network administrators manage network resources in a centralized host project, and workload teams deploy their own application resources and consume the network resources in service projects that are attached to the host project.

Both topologies include a base and restricted version of each VPC network. The base VPC network is used for resources that contain non-sensitive data, and the restricted VPC network is used for resources with sensitive data that require VPC Service Controls. For more information on implementing VPC Service Controls, see Protect your resources with VPC Service Controls.

Dual Shared VPC network topology

If you require network isolation between your development, non-production, and production networks on Google Cloud, we recommend the dual Shared VPC network topology. This topology uses separate Shared VPC networks for each environment, with each environment additionally split between a base Shared VPC network and a restricted Shared VPC network.

The following diagram shows the dual Shared VPC network topology.

The diagram describes these key concepts of the dual Shared VPC topology:

• Each environment (production, non-production, and development) has one Shared VPC network for the base network and one Shared VPC network for the restricted network. This diagram shows only the production environment, but the same pattern is repeated for each environment.
• Each Shared VPC network has two subnets, with each subnet in a different region.
• Connectivity with on-premises resources is enabled through four VLAN attachments to the Dedicated Interconnect instance for each Shared VPC network, using four Cloud Router services (two in each region for redundancy). For more information, see Hybrid connectivity between on-premises environment and Google Cloud.

By design, this topology doesn't allow network traffic to flow directly between environments. If you do require network traffic to flow directly between environments, you must take additional steps to allow this network path. For example, you might configure Private Service Connect endpoints to expose a service from one VPC network to another VPC network. Alternatively, you might configure your on-premises network to let traffic flow from one Google Cloud environment to the on-premises environment and then to another Google Cloud environment.

Hub-and-spoke network topology

If you deploy resources in Google Cloud that require a direct network path to resources in multiple environments, we recommend the hub-and-spoke network topology.

The hub-and-spoke topology uses several of the concepts that are part of the dual Shared VPC topology, but modifies the topology to add a hub network. The following diagram shows the hub-and-spoke topology.

The diagram describes these key concepts of hub-and-spoke network topology:

• This model adds a hub network, and each of the development, non-production, and production networks (spokes) are connected to the hub network through VPC Network Peering. Alternatively, if you anticipate exceeding the quota limit, you can use an HA VPN gateway instead.
• Connectivity to on-premises networks is allowed only through the hub network. All spoke networks can communicate with shared resources in the hub network and use this path to connect to on-premises networks.
• The hub networks include an NVA for each region, deployed redundantly behind internal Network Load Balancer instances. This NVA serves as the gateway to allow or deny traffic to communicate between spoke networks.
• The hub network also hosts tooling that requires connectivity to all other networks. For example, you might deploy tools on VM instances for configuration management to the common environment.
• The hub-and-spoke model is duplicated for a base version and restricted version of each network.

To enable spoke-to-spoke traffic, the blueprint deploys NVAs on the hub Shared VPC network that act as gateways between networks. Routes are exchanged from hub-to-spoke VPC networks through custom routes exchange. In this scenario, connectivity between spokes must be routed through the NVA because VPC Network Peering is non-transitive, and therefore, spoke VPC networks can't exchange data with each other directly. You must configure the virtual appliances to selectively allow traffic between spokes.

Project deployment patterns

When creating new projects for workloads, you must decide how resources in this project connect to your existing network. The following table describes the patterns for deploying projects that are used in the blueprint.

IP address allocation

This section introduces how the blueprint architecture allocates IP address ranges. You might need to change the specific IP address ranges used based on the IP address availability in your existing hybrid environment.

The following table provides a breakdown of the IP address space that's allocated for the blueprint. The hub environment only applies in the hub-and-spoke topology.

The preceding table demonstrates these concepts for allocating IP address ranges:

• IP address allocation is subdivided into ranges for each combination of base Shared VPC, restricted Shared VPC, region, and environment.
• Some resources are global and don't require subdivisions for each region.
• By default, for regional resources, the blueprint deploys in two regions. In addition, there are unused IP address ranges so that you can can expand into six additional regions.
• The hub network is only used in the hub-and-spoke network topology, while the development, non-production, and production environments are used in both network topologies.

The following table introduces how each type of IP address range is used.

Centralized DNS setup

For DNS resolution between Google Cloud and on-premises environments, we recommend that you use a hybrid approach with two authoritative DNS systems. In this approach, Cloud DNS handles authoritative DNS resolution for your Google Cloud environment and your existing on-premises DNS servers handle authoritative DNS resolution for on-premises resources. Your on-premises environment and Google Cloud environment perform DNS lookups between environments through forwarding requests.

The following diagram demonstrates the DNS topology across the multiple VPC networks that are used in the blueprint.

The diagram describes the following components of the DNS design that is deployed by the blueprint:

• The DNS hub project in the common folder is the central point of DNS exchange between the on-premises environment and the Google Cloud environment. DNS forwarding uses the same Dedicated Interconnect instances and Cloud Routers that are already configured in your network topology.
  • In the dual Shared VPC topology, the DNS hub uses the base production Shared VPC network.
  • In the hub-and-spoke topology, the DNS hub uses the base hub Shared VPC network.
• Servers in each Shared VPC network can resolve DNS records from other Shared VPC networks through DNS forwarding, which is configured between Cloud DNS in each Shared VPC host project and the DNS hub.
• On-premises servers can resolve DNS records in Google Cloud environments using DNS server policies that allow queries from on-premises servers. The blueprint configures an inbound server policy in the DNS hub to allocate IP addresses, and the on-premises DNS servers forward requests to these addresses. All DNS requests to Google Cloud reach the DNS hub first, which then resolves records from DNS peers.
• Servers in Google Cloud can resolve DNS records in the on-premises environment using forwarding zones that query on-premises servers. All DNS requests to the on-premises environment originate from the DNS hub. The DNS request source is 35.199.192.0/19.

Firewall policies

Google Cloud has multiple firewall policy types. Hierarchical firewall policies are enforced at the organization or folder level to inherit firewall policy rules consistently across all resources in the hierarchy. In addition, you can configure network firewall policies for each VPC network. The blueprint combines these firewall policies to enforce common configurations across all environments using Hierarchical firewall policies and to enforce more specific configurations at each individual VPC network using network firewall policies.

The blueprint doesn't use legacy VPC firewall rules. We recommend using only firewall policies and avoid mixing use with legacy VPC firewall rules.

Hierarchical firewall policies

The blueprint defines a single hierarchical firewall policy and attaches the policy to each of the production, non-production, development, bootstrap, and common folders. This hierarchical firewall policy contains the rules that should be enforced broadly across all environments, and delegates the evaluation of more granular rules to the network firewall policy for each individual environment.

The following table describes the hierarchical firewall policy rules deployed by the blueprint.

Network firewall policies

The blueprint configures a network firewall policy for each network. Each network firewall policy starts with a minimum set of rules that allow access to Google Cloud services and deny egress to all other IP addresses.

In the hub-and-spoke model, the network firewall policies contain additional rules to allow communication between spokes. The network firewall policy allows outbound traffic from one to the hub or another spoke, and allows inbound traffic from the NVA in the hub network.

The following table describes the rules in the global network firewall policy deployed for each VPC network in the blueprint.

When you first deploy the blueprint, a VM instance in a VPC network can communicate with Google Cloud services, but not to other infrastructure resources in the same VPC network. To allow VM instances to communicate, you must add additional rules to your network firewall policy and tags that explicitly allow the VM instances to communicate. Tags are added to VM instances, and traffic is evaluated against those tags. Tags additionally have IAM controls so that you can define them centrally and delegate their use to other teams.

The following diagram shows an example of how you can add custom tags and network firewall policy rules to let workloads communicate inside a VPC network.

The diagram demonstrates the following concepts of this example:

• The network firewall policy contains Rule 1 that denies outbound traffic from all sources at priority 65530.
• The network firewall policy contains Rule 2 that allows inbound traffic from instances with the service=frontend tag to instances with the service=backend tag at priority 999.
• The instance-2 VM can receive traffic from instance-1 because the traffic matches the tags allowed by Rule 2. Rule 2 is matched before Rule 1 is evaluated, based on the priority value.
• The instance-3 VM doesn't receive traffic. The only firewall policy rule that matches this traffic is Rule 1, so outbound traffic from instance-1 is denied.

Private access to Google Cloud APIs

To let resources in your VPC networks or on-premises environment reach Google Cloud services, we recommend private connectivity instead of outbound internet traffic to public API endpoints. The blueprint configures Private Google Access on every subnet and creates internal endpoints with Private Service Connect to communicate with Google Cloud services. Used together, these controls allow a private path to Google Cloud services, without relying on internet outbound traffic or publicly advertised internet ranges.

The blueprint configures Private Service Connect endpoints with API bundles to differentiate which services can be accessed in which network. The base network uses the all-apis bundle and can reach any Google service, and the restricted network uses the vpcsc bundle which allows access to a limited set of services that support VPC Service Controls.

For access from hosts that are located in an on-premises environment, we recommend that you use a convention of custom FQDN for each endpoint, as described in the following table. The blueprint uses a unique Private Service Connect endpoint for each VPC network, configured for access to a different set of API bundles. Therefore, you must consider how to route service traffic from the on-premises environment to the VPC network with the correct API endpoint, and if you're using VPC Service Controls, ensure that traffic to Google Cloud services reaches the endpoint inside the intended perimeter. Configure your on-premise controls for DNS, firewalls, and routers to allow access to these endpoints, and configure on-premise hosts to use the appropriate endpoint. For more information, see access Google APIs through endpoints.

The following table describes the Private Service Connect endpoints created for each network.

To ensure that traffic for Google Cloud services has a DNS lookup to the correct endpoint, the blueprint configures private DNS zones for each VPC network. The following table describes these private DNS zones.

The blueprint has additional configurations to enforce that these Private Service Connect endpoints are used consistently. Each Shared VPC network also enforces the following:

• A network firewall policy rule that allows outbound traffic from all sources to the IP address of the Private Service Connect endpoint on TCP:443.
• A network firewall policy rule that denies outbound traffic to 0.0.0.0/0, which includes the default domains that are used for access to Google Cloud services.

Internet connectivity

The blueprint doesn't allow inbound or outbound traffic between its VPC networks and the internet. For workloads that require internet connectivity, you must take additional steps to design the access paths required.

For workloads that require outbound traffic to the internet, we recommend that you manage outbound traffic through Cloud NAT to allow outbound traffic without unsolicited inbound connections, or through Secure Web Proxy for more granular control to allow outbound traffic to trusted web services only.

For workloads that require inbound traffic from the internet, we recommend that you design your workload with Cloud Load Balancing and Google Cloud Armor to benefit from DDoS and WAF protections.

We don't recommend that you design workloads that allow direct connectivity between the internet and a VM using an external IP address on the VM.

Hybrid connectivity between an on-premises environment and Google Cloud

To establish connectivity between the on-premises environment and Google Cloud, we recommend that you use Dedicated Interconnect to maximize security and reliability. A Dedicated Interconnect connection is a direct link between your on-premises network and Google Cloud.

The following diagram introduces hybrid connectivity between the on-premises environment and a Google Virtual Private Cloud network.

The diagram describes the following components of the pattern for 99.99% availability for Dedicated Interconnect:

• Four Dedicated Interconnect connections, with two connections in one metropolitan area (metro) and two connections in another metro. Within each metro, there are two distinct zones within the colocation facility.
• The connections are divided into two pairs, with each pair connected to a separate on-premises data center.
• VLAN attachments are used to connect each Dedicated Interconnect instance to Cloud Routers that are attached to the Shared VPC topology.
• Each Shared VPC network has four Cloud Routers, two in each region, with the dynamic routing mode set to global so that every Cloud Router can announce all subnets, independent of region.

With global dynamic routing, Cloud Router advertises routes to all subnets in the VPC network. Cloud Router advertises routes to remote subnets (subnets outside of the Cloud Router's region) with a lower priority compared to local subnets (subnets that are in the Cloud Router's region). Optionally, you can change advertised prefixes and priorities when you configure the BGP session for a Cloud Router.

Traffic from Google Cloud to an on-premises environment uses the Cloud Router closest to the cloud resources. Within a single region, multiple routes to on-premises networks have the same multi-exit discriminator (MED) value, and Google Cloud uses equal cost multi-path (ECMP) routing to distribute outbound traffic between all possible routes.

On-premises configuration changes

To configure connectivity between the on-premises environment and Google Cloud, you must configure additional changes in your on-premises environment. The Terraform code in the blueprint automatically configures Google Cloud resources but doesn't modify any of your on-premises network resources.

Some of the components for hybrid connectivity from your on-premises environment to Google Cloud are automatically enabled by the blueprint, including the following:

• Cloud DNS is configured with DNS forwarding between all Shared VPC networks to a single hub, as described in DNS setup. A Cloud DNS server policy is configured with inbound forwarder IP addresses.
• Cloud Router is configured to export routes for all subnets and custom routes for the IP addresses used by the Private Service Connect endpoints.

To enable hybrid connectivity, you must take the following additional steps:

1. Order a Dedicated Interconnect connection.
2. Configure on-premises routers and firewalls to allow outbound traffic to the internal IP address space defined in IP address space allocation.
3. Configure your on-premises DNS servers to forward DNS lookups bound for Google Cloud to the inbound forwarder IP addresses that is already configured by the blueprint.
4. Configure your on-premises DNS servers, firewalls, and routers to accept DNS queries from the Cloud DNS forwarding zone (35.199.192.0/19).
5. Configure on-premise DNS servers to respond to queries from on-premises hosts to Google Cloud services with the IP addresses defined in private access to Cloud APIs.
6. For encryption in transit over the Dedicated Interconnect connection, configure MACsec for Cloud Interconnect or configure HA VPN over Cloud Interconnect for IPsec encryption.

For more information, see Private Google Access for on-premises hosts.

What's next

• Read about detective controls (next document in this series).

Detective controls

Threat detection and monitoring capabilities are provided using a combination of built-in security controls from Security Command Center and custom solutions that let you detect and respond to security events.

Centralized logging for security and audit

The blueprint configures logging capabilities to track and analyze changes to your Google Cloud resources with logs that are aggregated to a single project.

The following diagram shows how the blueprint aggregates logs from multiple sources in multiple projects into a centralized log sink.

The diagram describes the following:

• Log sinks are configured at the organization node to aggregate logs from all projects in the resource hierarchy.
• Multiple log sinks are configured to send logs that match a filter to different destinations for storage and analytics.
• The prj-c-logging project contains all the resources for log storage and analytics.
• Optionally, you can configure additional tooling to export logs to a SIEM.

The blueprint uses different log sources and includes these logs in the log sink filter so that the logs can be exported to a centralized destination. The following table describes the log sources.

The following table describes the log sinks and how they are used with supported destinations in the blueprint.

For guidance on enabling additional log types and writing log sink filters, see the log scoping tool.

Threat monitoring with Security Command Center

We recommend that you activate Security Command Center Premium for your organization to automatically detect threats, vulnerabilities, and misconfigurations in your Google Cloud resources. Security Command Center creates security findings from multiple sources including the following:

• Security Health Analytics: detects common vulnerabilities and misconfigurations across Google Cloud resources.
• Attack path exposure: shows a simulated path of how an attacker could exploit your high-value resources, based on the vulnerabilities and misconfigurations that are detected by other Security Command Center sources.
• Event Threat Detection: applies detection logic and proprietary threat intelligence against your logs to identify threats in near-real time.
• Container Threat Detection: detects common container runtime attacks.
• Virtual Machine Threat Detection: detects potentially malicious applications that are running on virtual machines.
• Web Security Scanner: scans for OWASP Top Ten vulnerabilities in your web-facing applications on Compute Engine, App Engine, or Google Kubernetes Engine.

For more information on the vulnerabilities and threats addressed by Security Command Center, see Security Command Center sources.

You must activate Security Command Center after you deploy the blueprint. For instructions, see Activate Security Command Center for an organization.

After you activate Security Command Center, we recommend that you export the findings that are produced by Security Command Center to your existing tools or processes for triaging and responding to threats. The blueprint creates the prj-c-scc project with a Pub/Sub topic to be used for this integration. Depending on your existing tools, use one of the following methods to export findings:

• If you use the console to manage security findings directly in Security Command Center, configure folder-level and project-level roles for Security Command Center to let teams view and manage security findings just for the projects for which they are responsible.

• If you use Google SecOps as your SIEM, ingest Google Cloud data to Google SecOps.

• If you use a SIEM or SOAR tool with integrations to Security Command Center, share data with Cortex XSOAR, Elastic Stack, ServiceNow, Splunk, or QRadar.

• If you use an external tool that can ingest findings from Pub/Sub, configure continuous exports to Pub/Sub and configure your existing tools to ingest findings from the Pub/Sub topic.

Custom solution for automated log analysis

You might have requirements to create alerts for security events that are based on custom queries against logs. Custom queries can help supplement the capabilities of your SIEM by analyzing logs on Google Cloud and exporting only the events that merit investigation, especially if you don't have the capacity to export all cloud logs to your SIEM.

The blueprint helps enable this log analysis by setting up a centralized source of logs that you can query using a linked BigQuery dataset. To automate this capability, you must implement the code sample at bq-log-alerting and extend the foundation capabilities. The sample code lets you regularly query a log source and send a custom finding to Security Command Center.

The following diagram introduces the high-level flow of the automated log analysis.

The diagram shows the following concepts of automated log analysis:

• Logs from various sources are aggregated into a centralized logs bucket with log analytics and a linked BigQuery dataset.
• BigQuery views are configured to query logs for the security event that you want to monitor.
• Cloud Scheduler pushes an event to a Pub/Sub topic every 15 minutes and triggers Cloud Run functions.
• Cloud Run functions queries the views for new events. If it finds events, it pushes them to Security Command Center as custom findings.
• Security Command Center publishes notifications about new findings to another Pub/Sub topic.
• An external tool such as a SIEM subscribes to the Pub/Sub topic to ingest new findings.

The sample has several use cases to query for potentially suspicious behavior. Examples include a login from a list of super admins or other highly privileged accounts that you specify, changes to logging settings, or changes to network routes. You can extend the use cases by writing new query views for your requirements. Write your own queries or reference security log analytics for a library of SQL queries to help you analyze Google Cloud logs.

Custom solution to respond to asset changes

To respond to events in real time, we recommend that you use Cloud Asset Inventory to monitor asset changes. In this custom solution, an asset feed is configured to trigger notifications to Pub/Sub about changes to resources in real time, and then Cloud Run functions runs custom code to enforce your own business logic based on whether the change should be allowed.

The blueprint has an example of this custom governance solution that monitors for IAM changes that add highly sensitive roles including Organization Admin, Owner, and Editor. The following diagram describes this solution.

The previous diagram shows these concepts:

• Changes are made to an allow policy.
• The Cloud Asset Inventory feed sends a real-time notification about the allow policy change to Pub/Sub.
• Pub/Sub triggers a function.
• Cloud Run functions runs custom code to enforce your policy. The example function has logic to assess if the change has added the Organization Admin, Owner, or Editor roles to an allow policy. If so, the function creates a custom security finding and sends it to Security Command Center.
• Optionally, you can use this model to automate remediation efforts. Write additional business logic in Cloud Run functions to automatically take action on the finding, such as reverting the allow policy to its previous state.

In addition, you can extend the infrastructure and logic used by this sample solution to add custom responses to other events that are important to your business.

What's next

• Read about preventative controls (next document in this series).

Preventative controls for acceptable resource configurations

We recommend that you define policy constraints that enforce acceptable resource configurations and prevent risky configurations. The blueprint uses a combination of organization policy constraints and infrastructure-as-code (IaC) validation in your pipeline. These controls prevent the creation of resources that don't meet your policy guidelines. Enforcing these controls early in the design and build of your workloads helps you to avoid remediation work later.

Organization policy constraints

The Organization Policy service enforces constraints to ensure that certain resource configurations can't be created in your Google Cloud organization, even by someone with a sufficiently privileged IAM role.

The blueprint enforces policies at the organization node so that these controls are inherited by all folders and projects within the organization. This bundle of policies is designed to prevent certain high-risk configurations, such as exposing a VM to the public internet or granting public access to storage buckets, unless you deliberately allow an exception to the policy.

The following table introduces the organization policy constraints that are implemented in the blueprint:

These policies are a starting point that we recommend for most customers and most scenarios, but you might need to modify organization policy constraints to accommodate certain workload types. For example, a workload that uses a Cloud Storage bucket as the backend for Cloud CDN to host public resources is blocked by storage.publicAccessPrevention, or a public-facing Cloud Run app that doesn't require authentication is blocked by iam.allowedPolicyMemberDomains. In these cases, modify the organization policy at the folder or project level to allow a narrow exception. You can also conditionally add constraints to organization policy by defining a tag that grants an exception or enforcement for policy, then applying the tag to projects and folders.

For additional constraints, see available constraints and custom constraints.

Pre-deployment validation of infrastructure-as-code

The blueprint uses a GitOps approach to manage infrastructure, meaning that all infrastructure changes are implemented through version-controlled infrastructure-as-code (IaC) and can be validated before deploying.

The policies enforced in the blueprint define acceptable resource configurations that can be deployed by your pipeline. If code that is submitted to your GitHub repository does not pass the policy checks, no resources are deployed.

For information on how pipelines are used and how controls are enforced through CI/CD automation, see deployment methodology.

What's next

• Read about deployment methodology (next document in this series)

Deployment methodology

We recommend that you use declarative infrastructure to deploy your foundation in a consistent and controllable manner. This approach helps enable consistent governance by enforcing policy controls about acceptable resource configurations into your pipelines. The blueprint is deployed using a GitOps flow, with Terraform used to define infrastructure as code (IaC), a Git repository for version control and approval of code, and Cloud Build for CI/CD automation in the deployment pipeline. For an introduction to this concept, see managing infrastructure as code with Terraform, Cloud Build, and GitOps.

The following sections describe how the deployment pipeline is used to manage resources in your organization.

Pipeline layers

To separate the teams and technology stack that are responsible for managing different layers of your environment, we recommend a model that uses different pipelines and different personas that are responsible for each layer of the stack.

The following diagram introduces our recommended model for separating a foundation pipeline, infrastructure pipeline, and application pipeline.

The diagram introduces the pipeline layers in this model:

• The foundation pipeline deploys the foundation resources that are used across the platform. We recommend that a single central team is responsible for managing the foundation resources that are consumed by multiple business units and workloads.
• The infrastructure pipeline deploys projects and infrastructure that are used by workloads, such as VM instances or databases. The blueprint sets up a separate infrastructure pipeline for each business unit, or you might prefer a single infrastructure pipeline used by multiple teams.
• The application pipeline deploys the artifacts for each workload, such as containers or images. You might have many different application teams with individual application pipelines.

The following sections introduce the usage of each pipeline layer.

The foundation pipeline

The foundation pipeline deploys the foundation resources. It also sets up the infrastructure pipeline that is used to deploy infrastructure used by workloads.

To create the foundation pipeline, you first clone or fork the terraform-example-foundation to your own Git repository. Follow the steps in the 0-bootstrap README file to configure your bootstrap folder and resources.

After you create the foundation pipeline in the 0-bootstrap stage, the following stages deploy resources on the foundation pipeline. Review the README directions for each stage and implement each stage sequentially.

The infrastructure pipeline

The infrastructure pipeline deploys the projects and infrastructure (for example, the VM instances and databases) that are used by workloads. The foundation pipeline deploys multiple infrastructure pipelines. This separation between the foundation pipeline and infrastructure pipeline allows for a separation between platform-wide resources and workload-specific resources.

The following diagram describes how the blueprint configures multiple infrastructure pipelines that are intended for use by separate teams.

The diagram describes the following key concepts:

• Each infrastructure pipeline is used to manage infrastructure resources independently of the foundation resources.
• Each business unit has its own infrastructure pipeline, managed in a dedicated project in the common folder.
• Each of the infrastructure pipelines has a service account with permission to deploy resources only to the projects that are associated with that business unit. This strategy creates a separation of duties between the privileged service accounts used for the foundation pipeline and those used by each infrastructure pipeline

This approach with multiple infrastructure pipelines is recommended when you have multiple entities inside your organization that have the skills and appetite to manage their infrastructure separately, particularly if they have different requirements such as the types of pipeline validation policy they want to enforce. Alternatively, you might prefer to have a single infrastructure pipeline managed by a single team with consistent validation policies.

In the terraform-example-foundation, stage 4 configures an infrastructure pipeline, and stage 5 demonstrates an example of using that pipeline to deploy infrastructure resources.

The application pipeline

The application pipeline is responsible for deploying application artifacts for each individual workload, such as images or Kubernetes containers that run the business logic of your application. These artifacts are deployed to infrastructure resources that were deployed by your infrastructure pipeline.

The enterprise foundation blueprint sets up your foundation pipeline and infrastructure pipeline, but doesn't deploy an application pipeline. For an example application pipeline, see the enterprise application blueprint.

Automating your pipeline with Cloud Build

The blueprint uses Cloud Build to automate CI/CD processes. The following table describes the controls are built into the foundation pipeline and infrastructure pipeline that are deployed by the terraform-example-foundation repository. If you are developing your own pipelines using other CI/CD automation tools, we recommend that you apply similar controls.

Branching strategy

We recommend a persistent branch strategy for submitting code to your Git system and deploying resources through the foundation pipeline. The following diagram describes the persistent branch strategy.

The diagram describes three persistent branches in Git (development, non-production, and production) that reflect the corresponding Google Cloud environments. There are also multiple ephemeral feature branches that don't correspond to resources that are deployed in your Google Cloud environments.

We recommend that you enforce a pull request (PR) process into your Git system so that any code that is merged to a persistent branch has an approved PR.

To develop code with this persistent branch strategy, follow these high-level steps:

1. When you're developing new capabilities or working on a bug fix, create a new branch based off of the development branch. Use a naming convention for your branch that includes the type of change, a ticket number or other identifier, and a human-readable description, like feature/123456-org-policies.
2. When you complete the work in the feature branch, open a PR that targets the development branch.
3. When you submit the PR, the PR triggers the foundation pipeline to perform terraform plan and terraform validate to stage and verify the changes.
4. After you validate the changes to the code, merge the feature or bug fix into the development branch.
5. The merge process triggers the foundation pipeline to run terraform apply to deploy the latest changes in the development branch to the development environment.
6. Review the changes in the development environment using any manual reviews, functional tests, or end-to-end tests that are relevant to your use case. Then promote changes to the non-production environment by opening a PR that targets the non-production branch and merge your changes.
7. To deploy resources to the production environment, repeat the same process as step 6: review and validate the deployed resources, open a PR to the production branch, and merge.

What's next

• Read about operations best practices (next document in this series).

Operations best practices

This section introduces operations that you must consider as you deploy and operate additional workloads into your Google Cloud environment. This section isn't intended to be exhaustive of all operations in your cloud environment, but introduces decisions related to the architectural recommendations and resources deployed by the blueprint.

Update foundation resources

Although the blueprint provides an opinionated starting point for your foundation environment, your foundation requirements might grow over time. After your initial deployment, you might adjust configuration settings or build new shared services to be consumed by all workloads.

To modify foundation resources, we recommend that you make all changes through the foundation pipeline. Review the branching strategy for an introduction to the flow of writing code, merging it, and triggering the deployment pipelines.

Decide attributes for new workload projects

When creating new projects through the project factory module of the automation pipeline, you must configure various attributes. Your process to design and create projects for new workloads should include decisions for the following:

• Which Google Cloud APIs to enable
• Which Shared VPC to use, or whether to create a new VPC network
• Which IAM roles to create for the initial project-service-account that is created by the pipeline
• Which project labels to apply
• The folder that the project is deployed to
• Which billing account to use
• Whether to add the project to a VPC Service Controls perimeter
• Whether to configure a budget and billing alert threshold for the project

For a complete reference of the configurable attributes for each project, see the input variables for the project factory in the automation pipeline.

Manage permissions at scale

When you deploy workload projects on top of your foundation, you must consider how you will grant access to the intended developers and consumers of those projects. We recommend that you add users into a group that is managed by your existing identity provider, synchronize the groups with Cloud Identity, and then apply IAM roles to the groups. Always keep in mind the principle of least privilege.

We also recommend that you use IAM recommender to identify allow policies that grant over-privileged roles. Design a process to periodically review recommendations or automatically apply recommendations into your deployment pipelines.

Coordinate changes between the networking team and the application team

The network topologies that are deployed by the blueprint assume that you have a team responsible for managing network resources, and separate teams responsible for deploying workload infrastructure resources. As the workload teams deploy infrastructure, they must create firewall rules to allow the intended access paths between components of their workload, but they don't have permission to modify the network firewall policies themselves.

Plan how teams will work together to coordinate the changes to the centralized networking resources that are needed to deploy applications. For example, you might design a process where a workload team requests tags for their applications. The networking team then creates the tags and adds rules to the network firewall policy that allows traffic to flow between resources with the tags, and delegates the IAM roles to use the tags to the workload team.

Optimize your environment with the Active Assist portfolio

In addition to IAM recommender, Google Cloud provides the Active Assist portfolio of services to make recommendations about how to optimize your environment. For example, firewall insights or the unattended project recommender provide actionable recommendations that can help tighten your security posture.

Design a process to periodically review recommendations or automatically apply recommendations into your deployment pipelines. Decide which recommendations should be managed by a central team and which should be the responsibility of workload owners, and apply IAM roles to access the recommendations accordingly.

Grant exceptions to organization policies

The blueprint enforces a set of organization policy constraints that are recommended to most customers in most scenarios, but you might have legitimate use cases that require limited exceptions to the organization policies you enforce broadly.

For example, the blueprint enforces the iam.disableServiceAccountKeyCreation constraint. This constraint is an important security control because a leaked service account key can have a significant negative impact, and most scenarios should use more secure alternatives to service account keys to authenticate. However, there might be use cases that can only authenticate with a service account key, such as an on-premises server that requires access to Google Cloud services and cannot use workload identity federation. In this scenario, you might decide to allow an exception to the policy, so long as additional compensating controls like best practices for managing service account keys are enforced.

Therefore, you should design a process for workloads to request an exception to policies, and ensure that the decision makers who are responsible for granting exceptions have the technical knowledge to validate the use case and consult on whether additional controls must be in place to compensate. When you grant an exception to a workload, modify the organization policy constraint as narrowly as possible. You can also conditionally add constraints to an organization policy by defining a tag that grants an exception or enforcement for policy, then applying the tag to projects and folders.

Protect your resources with VPC Service Controls

The blueprint helps prepare your environment for VPC Service Controls by separating the base and restricted networks. However, by default, the Terraform code doesn't enable VPC Service Controls because this enablement can be a disruptive process.

A perimeter denies access to restricted Google Cloud services from traffic that originates outside the perimeter, which includes the console, developer workstations, and the foundation pipeline used to deploy resources. If you use VPC Service Controls, you must design exceptions to the perimeter that allow the access paths that you intend.

A VPC Service Controls perimeter is intended for exfiltration controls between your Google Cloud organization and external sources. The perimeter isn't intended to replace or duplicate allow policies for granular access control to individual projects or resources. When you design and architect a perimeter, we recommend using a common unified perimeter for lower management overhead.

If you must design multiple perimeters to granularly control service traffic within your Google Cloud organization, we recommend that you clearly define the threats that are addressed by a more complex perimeter structure and the access paths between perimeters that are needed for intended operations.

To adopt VPC Service Controls, evaluate the following:

• Which of your use cases require VPC Service Controls.

• Whether the required Google Cloud services support VPC Service Controls.

• How to configure breakglass access to modify the perimeter in case it disrupts your automation pipelines.

• How to use best practices for enabling VPC Service Controls to design and implement your perimeter.

After the perimeter is enabled, we recommend that you design a process to consistently add new projects to the correct perimeter, and a process to design exceptions when developers have a new use case that is denied by your current perimeter configuration.

Test organization-wide changes in a separate organization

We recommend that you never deploy changes to production without testing. For workload resources, this approach is facilitated by separate environments for development, non-production, and production. However, some resources at the organization don't have separate environments to facilitate testing.

For changes at the organization-level, or other changes that can affect production environments like the configuration between your identity provider and Cloud Identity, consider creating a separate organization for test purposes.

Control remote access to virtual machines

Because we recommend that you deploy immutable infrastructure through the foundation pipeline, infrastructure pipeline, and application pipeline, we also recommend that you only grant developers direct access to a virtual machine through SSH or RDP for limited or exceptional use cases.

For scenarios that require remote access, we recommend that you manage user access using OS Login where possible. This approach uses managed Google Cloud services to enforce access control, account lifecycle management, two-step verification, and audit logging. Alternatively, if you must allow access through SSH keys in metadata or RDP credentials, it is your responsibility to manage the credential lifecycle and store credentials securely outside of Google Cloud.

In any scenario, a user with SSH or RDP access to a VM can be a privilege escalation risk, so you should design your access model with this in mind. The user can run code on that VM with the privileges of the associated service account or query the metadata server to view the access token that is used to authenticate API requests. This access can then be a privilege escalation if you didn't deliberately intend for the user to operate with the privileges of the service account.

Mitigate overspending by planning budget alerts

The blueprint implements best practices introduced in the Google Cloud Architecture Framework: Cost Optimization for managing cost, including the following:

• Use a single billing account across all projects in the enterprise foundation.

• Assign each project a billingcode metadata label that is used to allocate cost between cost centers.

• Set budgets and alert thresholds.

It's your responsibility to plan budgets and configure billing alerts. The blueprint creates budget alerts for workload projects when the forecasted spending is on track to reach 120% of the budget. This approach lets a central team identify and mitigate incidents of significant overspending. Significant unexpected increases in spending without a clear cause can be an indicator of a security incident and should be investigated from the perspectives of both cost control and security.

Depending on your use case, you might set a budget that is based on the cost of an entire environment folder, or all projects related to a certain cost center, instead of setting granular budgets for each project. We also recommend that you delegate budget and alert setting to workload owners who might set more granular alerting threshold for their day-to-day monitoring.

For guidance on building FinOps capabilities, including forecasting budgets for workloads, see Getting started with FinOps on Google Cloud.

Allocate costs between internal cost centers

The console lets you view your billing reports to view and forecast cost in multiple dimensions. In addition to the prebuilt reports, we recommend that you export billing data to a BigQuery dataset in the prj-c-billing-export project. The exported billing records allow you to allocate cost on custom dimensions, such as your internal cost centers, based on project label metadata like billingcode.

The following SQL query is a sample query to understand costs for all projects that are grouped by the billingcode project label.

#standardSQL
SELECT
   (SELECT value from UNNEST(labels) where key = 'billingcode') AS costcenter,
   service.description AS description,
   SUM(cost) AS charges,
   SUM((SELECT SUM(amount) FROM UNNEST(credits))) AS credits
FROM PROJECT_ID.DATASET_ID.TABLE_NAME
GROUP BY costcenter, description
ORDER BY costcenter ASC, description ASC

To set up this export, see export Cloud Billing data to BigQuery.

If you require internal accounting or chargeback between cost centers, it's your responsibility to incorporate the data that is obtained from this query into your internal processes.

Ingest findings from detective controls into your existing SIEM

Although the foundation resources help you configure aggregated destinations for audit logs and security findings, it is your responsibility to decide how to consume and use these signals.

If you have a requirement to aggregate logs across all cloud and on-premise environments into an existing SIEM, decide how to ingest logs from the prj-c-logging project and findings from Security Command Center into your existing tools and processes. You might create a single export for all logs and findings if a single team is responsible for monitoring security across your entire environment, or you might create multiple exports filtered to the set of logs and findings needed for multiple teams with different responsibilities.

Alternatively, if log volume and cost are prohibitive, you might avoid duplication by retaining Google Cloud logs and findings only in Google Cloud. In this scenario, ensure that your existing teams have the right access and training to work with logs and findings directly in Google Cloud.

• For audit logs, design log views to grant access to a subset of logs in your centralized logs bucket to individual teams, instead of duplicating logs to multiple buckets which increases log storage cost.
• For security findings, grant folder-level and project-level roles for Security Command Center to let teams view and manage security findings just for the projects for which they are responsible, directly in the console.

Continuously develop your controls library

The blueprint starts with a baseline of controls to detect and prevent threats. We recommend that you review these controls and add additional controls based on your requirements. The following table summarizes the mechanisms to enforce governance policies and how to extend these for your additional requirements:

These controls might evolve as your requirements and maturity on Google Cloud change.

Manage encryption keys with Cloud Key Management Service

Google Cloud provides default encryption at rest for all customer content, but also provides Cloud Key Management Service (Cloud KMS) to provide you additional control over your encryption keys for data at rest. We recommend that you evaluate whether the default encryption is sufficient, or whether you have a compliance requirement that you must use Cloud KMS to manage keys yourself. For more information, see decide how to meet compliance requirements for encryption at rest.

The blueprint provides a prj-c-kms project in the common folder and a prj-{env}-kms project in each environment folder for managing encryption keys centrally. This approach lets a central team audit and manage encryption keys that are used by resources in workload projects, in order to meet regulatory and compliance requirements.

Depending on your operational model, you might prefer a single centralized project instance of Cloud KMS under the control of a single team, you might prefer to manage encryption keys separately in each environment, or you might prefer multiple distributed instances so that accountability for encryption keys can be delegated to the appropriate teams. Modify the Terraform code sample as needed to fit your operational model.

Optionally, you can enforce customer-managed encryption keys (CMEK) organization policies to enforce that certain resource types always require a CMEK key and that only CMEK keys from an allowlist of trusted projects can be used.

Store and audit application credentials with Secret Manager

We recommend that you never commit sensitive secrets (such as API keys, passwords, and private certificates) to source code repositories. Instead, commit the secret to Secret Manager and grant the Secret Manager Secret Accessor IAM role to the user or service account that needs to access the secret. We recommend that you grant the IAM role to an individual secret, not to all secrets in the project.

When possible, you should generate production secrets automatically within the CI/CD pipelines and keep them inaccessible to human users except in breakglass situations. In this scenario, ensure that you don't grant IAM roles to view these secrets to any users or groups.

The blueprint provides a single prj-c-secrets project in the common folder and a prj-{env}-secrets project in each environment folder for managing secrets centrally. This approach lets a central team audit and manage secrets used by applications in order to meet regulatory and compliance requirements.

Depending on your operational model, you might prefer a single centralized instance of Secret Manager under the control of a single team, or you might prefer to manage secrets separately in each environment, or you might prefer multiple distributed instances of Secret Manager so that each workload team can manage their own secrets. Modify the Terraform code sample as needed to fit your operational model.

Plan breakglass access to highly privileged accounts

Although we recommend that changes to foundation resources are managed through version-controlled IaC that is deployed by the foundation pipeline, you might have exceptional or emergency scenarios that require privileged access to modify your environment directly. We recommend that you plan for breakglass accounts (sometimes called firecall or emergency accounts) that have highly privileged access to your environment in case of an emergency or when the automation processes break down.

The following table describes some example purposes of breakglass accounts.

Your mechanism to permit breakglass access depends on the existing tools and procedures you have in place, but a few example mechanisms include the following:

• Use your existing tools for privileged access management to temporarily add a user to a group that is predefined with highly-privileged IAM roles or use the credentials of a highly-privileged account.
• Pre-provision accounts intended only for administrator usage. For example, developer Dana might have an identity dana@example.com for daily use and admin-dana@example.com for breakglass access.
• Use an application like just-in-time privileged access that allows a developer to self-escalate to more privileged roles.

Regardless of the mechanism you use, consider how you operationally address the following questions:

• How do you design the scope and granularity of breakglass access? For example, you might design a different breakglass mechanism for different business units to ensure that they cannot disrupt each other.
• How does your mechanism prevent abuse? Do you require approvals? For example, you might have split operations where one person holds credentials and one person holds the MFA token.
• How do you audit and alert on breakglass access? For example, you might configure a custom Event Threat Detection module to create a security finding when a predefined breakglass account is used.
• How do you remove the breakglass access and resume normal operations after the incident is over?

For common privilege escalation tasks and rolling back changes, we recommend designing automated workflows where a user can perform the operation without requiring privilege escalation for their user identity. This approach can help reduce human error and improve security.

For systems that require regular intervention, automating the fix might be the best solution. Google encourages customers to adopt a zero-touch production approach to make all production changes using automation, safe proxies, or audited breakglass. Google provides the SRE books for customers who are looking to adopt Google's SRE approach.

What's next

• Read Deploy the blueprint (next document in this series).

Deploy the blueprint

This section describes the process that you can use to deploy the blueprint, its naming conventions, and alternatives to blueprint recommendations.

Bringing it all together

To deploy your own enterprise foundation in alignment with the best practices and recommendations from this blueprint, follow the high-level tasks summarized in this section. Deployment requires a combination of prerequisite setup steps, automated deployment through the terraform-example-foundation on GitHub, and additional steps that must be configured manually after the initial foundation deployment is complete.

Additional administrative controls for customers with sensitive workloads

Google Cloud provides additional administrative controls that can help your security and compliance requirements. However, some controls involve additional cost or operational trade-offs that might not be appropriate for every customer. These controls also require customized inputs for your specific requirements that can't be fully automated in the blueprint with a default value for all customers.

This section introduces security controls that you apply centrally to your foundation. This section isn't intended to be exhaustive of all the security controls that you can apply to specific workloads. For more information on Google's security products and solutions, see Google Cloud security best practices center.

Evaluate whether the following controls are appropriate for your foundation based on your compliance requirements, risk appetite, and sensitivity of data.

Naming conventions

We recommend that you have a standardized naming convention for your Google Cloud resources. The following table describes recommended conventions for resource names in the blueprint.

Alternatives to default recommendations

The best practices that are recommended in the blueprint might not work for every customer. You can customize any of the recommendations to meet your specific requirements. The following table introduces some of the common variations that you might require based on your existing technology stack and ways of working.

What's next

• Implement the blueprint using the Terraform example foundation on GitHub.
• Learn more about best practice design principles with the Google Cloud Architecture Framework.

• Review the library of blueprints to help you accelerate the design and build of common enterprise workloads, including the following:

  • Import data from Google Cloud into a secured BigQuery data warehouse

  • Import data from an external network into a secured BigQuery data warehouse

  • Deploy a secured serverless architecture using Cloud Run functions

  • Deploy a secured serverless architecture using Cloud Run

• See related solutions to deploy on top of your foundation environment.

• For access to a demonstration environment, contact us at security-foundations-blueprint-support@google.com.