Best practices for running an IoT backend on Google Cloud

Last reviewed 2023-02-08 UTC

This document provides security best practices for managing and running an Internet of Things (IoT) backend on Google Cloud. In an IoT solution, an IoT backend connects edge devices to other resources. This document focuses on the following IoT backends: Message Queuing Telemetry Transport (MQTT) broker and the IoT platform.

This document is part of a series of documents that provide information about IoT architectures on Google Cloud and about migrating from IoT Core. The other documents in this series include the following:

This document provides best practices for provisioning and managing device credentials, authenticating and accessing control edge devices, and letting IoT edge devices access Google Cloud resources.

IoT architecture

An IoT architecture includes services that let you provision and manage device credentials, authenticate and access control edge devices, and let edge devices access Google Cloud resources. This document discusses two IoT backend architectures: one using MQTT broker and the other using an IoT platform. The main security differences between these two backends are device identity and device management. IoT platforms provide these capabilities as part of their system, whereas MQTT brokers require you to provide these capabilities.

The following diagram describes the IoT architecture.

IoT backend architecture.

The architecture shows the services that are required for the following three processes:

  1. Certificate provisioning, which is the process that you must complete to prepare an edge device for configuration.

  2. Authentication and authorization, which include the authentication scheme that the edge device and the MQTT broker or IoT platform use to authenticate with each other.

  3. Connections between edge devices and Google Cloud services, which are tasks that the edge device completes to connect to cloud resources and upload or download data.

This document focuses primarily on security best practices for provisioning and authentication.

The architecture uses a combination of the following services and features:

  • An edge device (such as a medical device) that you deploy at the edges of your environment and that is geographically close to the data that you want to process. The edge devices connect bi-directionally with your IoT backend, which means that they can send messages to and receive messages from the IoT backend.
  • An IoT backend that can be an MQTT broker or an IoT platform.
    • An MQTT broker provides a secure interface for edge devices to connect using the MQTT protocol. MQTT brokers lack capabilities for device identity and device management and rely on external systems to provide them.
    • An IoT platform is a cloud application that edge devices connect and communicate with. IoT platforms provide a secure interface for edge devices to connect using the MQTT protocol. Each IoT platform has its own security implementation that determines how it authenticates and authorizes edge devices and how it manages device identities.
  • A central certificate store that hosts the certificates for all edge devices.
  • Cloud resources that edge devices must access.

Provisioning an edge device

Before an edge device can connect with your backend workloads, you must provision a certificate for the edge device. There are two main scenarios that decide how you provision the certificate:

  • If your solution is based on commercial, generic devices, you have full control over the provisioning process after purchasing the device.

  • If you use custom-built devices, the initial provisioning process happens during the manufacturing of the devices and you must integrate the provisioning process with your vendors and manufacturers.

In either scenario, you must create device certificates with a chain of trust that links to a root certificate authority (CA). These certificates authenticate the device identity and help ensure that updates and modifications done on the device are by trusted actors. Use a CA such as Certificate Authority Service to complete the following tasks:

To provision a device certificate, you must complete the following tasks:

  1. Generate the public-private key pair using a hardware-based security solution that is supported by your device, such as a secure element (SE) or a hardware secure module (HSM). By design, the SE or HSM stores the private keys locally and the private keys are never exposed externally. If you are not using a hardware-based security solution to generate a public-private key pair, use the CA to generate keys instead. For more information, see Using an auto-generated key.

  2. Sign and generate a device certificate. After you generate the public-private key pair, download the public key, create a certificate signing request (CSR), and send the CSR to the CA for signing. The CA generates a device certificate that is linked to the root CA. For more information, see Using a CSR. When using auto-generated keys, you can request a device certificate from the CA directly.

  3. Install the signed device certificate on the edge device and send the certificate to a central certificate repository such as Secret Manager.

For more information, see How to deploy a secure and reliable public key infrastructure with Google Cloud CA Service (PDF).

For information about other provisioning best practices, see Best practices for automatically provisioning and configuring edge and bare metal systems and servers.

Three types of certificates are used to help secure an IoT solution:

  • The root CA certificate provides the root for the chain of trust of all the other certificates in your system. The backend workloads use the root certificate to validate the client certificates and the edge devices use the root certificate to validate the server certificate. You must distribute the root certificate to both the IoT backend and to the edge devices.

  • The server certificates are used to secure the endpoints that are exposed by the IoT backend. You have server certificates for the different encryption algorithms that your endpoints need to support. Server certificates are linked to the root CA. A secret manager manages and stores both the private and public portions of the server certificates. You must configure your IoT backend with the server certificates and their corresponding private keys.

  • The client certificates are used to identify edge devices. Each edge device has at least one client certificate, which means that the number of certificates that you have increases with the number of edge devices in your environment. Client certificates are linked to the root CA. You must distribute client certificates to your edge devices and to the IoT backend.

Process for generating a device certificate using an HSM or SE

The following diagram shows how a device certificate is provisioned when using an HSM or SE.

Process for generating a device certificate.

In this diagram, the following steps occur:

  1. The edge device generates the public key pair in the hardware.
  2. You download the public key and create the certificate signing request (CSR) for it.
  3. You send the CSR to the CA to request a certificate.
  4. The CA completes the following actions:
    1. Signs the certificate.
    2. Returns the device certificate to the provisioner.
  5. The provisioner completes the following actions:
    1. Sends the certificate to the edge device.
    2. Stores the certificate in the central certificate store.
  6. The edge device stores the certificate in a secure location.

Process for generating a device certificate using the CA

The following diagram shows how a device certificate is provisioned when using a CA.

Process for generating a device certificate.

In this diagram, the following steps occur:

  1. You generate a CSR and send it to the CA to request a certificate.
  2. The CA completes the following actions:
    1. Generates and public key pair and signs the public key.
    2. Returns the device certificate and the private key to the provisioner.
  3. You complete the following actions:
    1. Send the certificate and private key to the edge device.
    2. Store the certificate and private key in the central certificate store.
  4. The edge device stores the certificate in a secure location.

Best practices for device identity

This section describes the best practices for device identities.

Use an identity provider with MQTT brokers

MQTT brokers authenticate edge devices by using device credentials provided by plugins, databases, and files. To manage your device identities in a systematic and scalable manner, use an identity provider (IdP). The IdP manages the identities and credentials for all devices and acts as the primary source of truth for device identities.

To keep the device identity updated in the MQTT broker, implement a system-specific integration layer. For more information about managing device credentials, see Provisioning an edge device.

Use the digital identities of the IoT platform as the source of truth

The IoT platform has security features that manage the device identities and device credentials, and authenticate and authorize devices trying to access the platform. These security features help ensure that only authorized devices are allowed to access the IoT platform and help ensure data integrity.

Verify that the device identities managed by the IoT platform represent the primary source of truth of all the devices the IoT platform manages. Other components in an IoT solution that need device identity information should rely on the security system of the IoT platform. The IoT platform grants access rights to devices and propagates any security changes throughout the IoT solution.

Best practices for network connectivity

Securing network connectivity is important for the following reasons:

  • Secure networks help ensure that a device connects to the right backend. For example, a secure network can prevent DNS spoofing, which is an attack that tries to divert devices to connect to a rogue backend that is controlled by attackers.
  • Secure networks help ensure third parties can't read your data traffic. For example, a secure network can prevent an attacker-in-the-middle attack, where attackers read the traffic between your device and the backend.

Use Transport Layer Security (TLS) to protect network communication between your edge devices and backend workloads.

Extend TLS with mTLS to implement a mutual authentication scheme that enables both connecting parties to establish the identity of each other.

For instructions on using TLS, see Standalone MQTT broker architecture on Google Cloud and IoT platform product architecture on Google Cloud.

Best practices for certificate management for MQTT brokers

This section describes best practices for managing certificates when using MQTT brokers.

Store certificates centrally

Store and manage server certificates and device certificates in a central location. Specifically, ensure that you have the following controls in place:

  • An inventory of all your devices and their certificates and the server endpoints and their certificates.
  • Additional information about the certificates such as their validity.
  • The ability to add and remove certificates for devices so that devices can connect using new certificates.
  • Access rights to your central certificate store, to limit what the different roles in your backend can do with the certificates.

Use a secret storage and management solution such as Secret Manager or HashiCorp Vault. Secret Manager lets you version, update, and invalidate device credentials, and to manage access policies to your credentials.

For an IoT platform, implement access to the credentials using Secret Manager API access.

Protect certificates on edge devices

To store certificates and keys on the edge devices, use a local trusted execution environment or certificate store to protect the credential and block unauthorized accesses. If you need to store secret material on your devices, encrypt that material using techniques such as flash encryption, and store it on tamper-proof elements to help prevent unauthorized data extraction.

Synchronize the central certificate store with the MQTT broker certificate store

MQTT brokers must access client certificates for certificate-based authentication, so you must synchronize the certificate stores of MQTT brokers with the central certificate store. Verify that changes on the central certificate store, such as add, update, and delete certificates, are synchronized with the MQTT broker certificate store. MQTT brokers use certificate stores such as MySQL, PostgresDB, and Java Key Store. Depending on which certificate store your MQTT broker uses, ensure that the following processes exist:

  • A process that monitors for changes in the central certificate store and notifies the synchronization process.
  • A process that takes changes in the central certificate store and synchronizes the changes in the central certificate store with the certificate store used by the MQTT broker.

When you use Secret Manager as your certificate store, you can use event notifications as the monitoring process. You can implement the synchronization process as a listener of the event notifications.

Distribute certificates to edge devices securely

When using MQTT brokers, distribute the root certificate and client certificates to your edge devices. When you distribute certificates, you must secure your communication channels so that the traffic doesn't get intercepted.

The main communication channels for certificate distribution are the following:

  • A direct path from the IoT backend to the edge devices over existing communication channels.
  • An indirect path in which edge devices request and download the certificates.

During certificate distribution, you require the following components:

  • A certificate store where certificates are centrally managed.
  • A distribution coordinator which sends the certificates and tracks the distribution process for each edge device.
  • An update handler on the edge device that receives or downloads the certificates and stores them on the device.

Distribute certificates during the provisioning processes for edge devices, and when you need to rotate certificates.

During the provisioning process, ensure that the provisioner has direct access to edge devices over encrypted channels such as SSH and uses tools such as SCP. Because the devices are not in operation, you can push certificates directly to the edge devices.

When rotating certificates, use the MQTT broker as the communication channel between the distribution coordinator and the edge devices. Use other channels to download certificates onto the device. To minimize disruption of the edge devices in operation, use an indirect certificate distribution path. The process would consist of the following logical steps:

  1. The distribution coordinator acquires access credentials from the certificate store.
  2. The distribution coordinator pushes the certificate access credentials to the edge devices together with additional information, such as the download URL.
  3. The on-device update handler receives the access credentials and temporarily stores the information and acknowledges receipt back.
  4. The update handler coordinates the certificate download when the device is not active. The update handler uses the access credentials to download certificates from the credential store.
  5. After the certificates are downloaded, the update handler continues with the certificate rotation process which is described in the certificate rotation section.

When you use Secret Manager as the central certificate store, you can generate short-lived access tokens to grant and restrict access to certificates. For more information, see Distribute access tokens to devices securely.

To help prevent the certificates from being exposed during transit, encrypt the connection between your edge devices and the MQTT broker. For more information, see Best practices for network connectivity.

Rotate certificates automatically

To limit the damage an exposed certificate can cause, generate certificates with a finite valid period and rotate the certificates before they expire. For large-scale IoT deployments, implement an automatic certificate rotation procedure to consistently update your devices with new certificates before the old ones expire. Deployed devices without valid certificates means that the devices can stop functioning, which can be costly to fix and negatively affect the overall functionality of your IoT solution.

Your edge devices must connect bi-directionally with your MQTT broker to ensure that they can send messages to the MQTT broker and that they can receive messages from the MQTT broker.

During certificate rotation, you require the following components:

  • A monitoring process that recurrently scans through your certificate inventory and looks for certificates that are about to expire. The monitoring process triggers certificate rotation for expiring certificates.
  • A rotation process that initializes and oversees certificate rotation.
  • A device certificate rotation handler on the edge device that communicates with the MQTT broker and executes certificate rotation steps on the device.

To rotate certificates, the IoT solution completes the following steps:

  1. The rotation process sends an initialization message to the edge device to start certificate rotation.
  2. The device certificate rotation handler acknowledges the initialization message by sending a response back to the rotation job.
  3. The rotation process requests a new certificate from the CA. This request is similar to the certificate provisioning request, except that the keys and CSR are sent as MQTT broker messages.
  4. After receiving the new certificate from the CA, the rotation job distributes the certificate to the central certificate store and to the edge device. It also synchronizes the certificate to the certificate store of the MQTT broker.
  5. The device certificate rotation handler stores the new certificate and initializes a new connection with the MQTT broker using the new certificate.
  6. After the new connection is established, the device certificate rotation handler sends a completed message to the MQTT broker.
  7. After receiving the completed message, the rotation process invalidates the old certificate in the central certificate store.

To help protect the certificates that are being sent during the rotation process, use dedicated MQTT topics for certificate rotation. Limit access to these topics to only the rotation job and the edge device.

To help protect the certificate rotation process from runtime failures, enable persistence for the changes and progress.

For more information about rotating secrets using Secret Manager, see Rotation of secrets.

Best practices for certificate management for IoT platforms

If you're using an IoT platform, use the certificate update and distribution mechanisms provided by the platform. For backup purposes, you can regularly export the credentials from your IoT platform to a secondary secret storage, such as Secret Manager.

Best practices for authentication with an MQTT broker

During the mutual authentication process, backend workloads verify the identity of edge devices, and edge devices verify the identity of backend workloads. After the backend workloads confirm the identity of the edge device, the backend workloads authorize the device access to resources.

The following sections provide best practices for authentication methods when using MQTT brokers.

Choose the authentication method for MQTT brokers

Different IoT backends support different authentication methods. The commonly used methods are the following:

  • Username and password authentication, where the edge device presents its username and password to verify its identity.
  • Token-based authentication, where encrypted security tokens are used to verify the edge device's identity.
  • Customized authentication schemes, where you implement a custom mechanism to verify the identity of the edge device.

As part of the