Troubleshooting using the serial console

This page describes how to enable interactive access to an instance's serial console to debug boot and networking issues, troubleshoot malfunctioning instances, interact with the GRand Unified Bootloader (GRUB), and perform other troubleshooting tasks.

A virtual machine (VM) instance has four virtual serial ports. Interacting with a serial port is similar to using a terminal window, in that input and output is entirely in text mode and there is no graphical interface or mouse support. The instance's operating system, BIOS, and other system-level entities often write output to the serial ports, and can accept input such as commands or answers to prompts. Typically, these system-level entities use the first serial port (port 1) and serial port 1 is often referred to as the serial console.

If you only need to view serial port output without issuing any commands to the serial console, you can call the getSerialPortOutput method or use Cloud Logging to read information that your instance has written to its serial port; see Viewing serial port logs. However, if you run into problems accessing your instance through SSH or need to troubleshoot an instance that is not fully booted, you can enable interactive access to the serial console, which lets you connect to and interact with any of your instance's serial ports. For example, you can directly run commands and respond to prompts in the serial port.

When you enable or disable the serial port, you can use any Boolean value that is accepted by the metadata server. For more information, see Boolean values.

Before you begin

• If you haven't already, then set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:
  

  Select the tab for how you plan to use the samples on this page:

  Console

  When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

  gcloud

  1. Install the Google Cloud CLI, then initialize it by running the following command:

     gcloud init

  2. Set a default region and zone.

  REST

  To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

  Install the Google Cloud CLI, then initialize it by running the following command:

  gcloud init

  For more information, see Authenticate for using REST in the Google Cloud authentication documentation.

Enabling interactive access on the serial console

Enable interactive serial console access for individual VM instances or for an entire project.

Enabling access for a project

Enabling interactive serial console access on a project enables access for all VM instances that are part of that project.

By default, interactive serial port access is disabled. You can also explicitly disable it by setting the serial-port-enable key to FALSE. In either case, any per-instance setting overrides the project-level setting or the default setting.

gcloud compute project-info add-metadata \
    --metadata serial-port-enable=TRUE

{
 "fingerprint": "FikclA7UBC0=",
 "items": [
  {
   "key": "serial-port-enable",
   "value": "TRUE"
  }
 ]
}

Enabling access for a VM instance

Enable interactive serial console access for a specific instance. A per-instance setting, if it exists, overrides any project-level setting. You can also disable access for a specific instance, even if access is enabled on the project level, by setting serial-port-enable to FALSE, instead of TRUE. Similarly, you can enable access for one or more instances even if it is disabled for the project, explicitly or by default.

gcloud compute instances add-metadata instance-name \
    --metadata serial-port-enable=TRUE

POST https://compute.googleapis.com/compute/v1/projects/myproject/zones/us-central1-a/instances/example-instance/setMetadata
{
 "fingerprint": "zhma6O1w2l8=",
 "items": [
  {
   "key": "serial-port-enable",
   "value": "TRUE"
  }
 ]
}

Configure serial console for a bare metal instance

For bare metal instances, increase the bit rate, also known as baud rate, for the serial console to 115,200 bps (~11.5kB/sec). Using a slower speed results in garbled or missing console output.

Bootloader configuration varies between operating systems and OS versions. Refer to the OS distributor's documentation for instructions.

If modifying the bit rate on the command line for the current session, use a command similar to the following:

console=ttyS0,115200

If modifying the GRUB configuration, use a command similar to the following:

serial --speed=115200

Make sure that you update the actual bootloader configuration. This can be done with update-grub, grub2-mkconfig, or a similar command.

Connecting to a serial console

Compute Engine offers regional serial console gateways for each Google Cloud region. After enabling interactive access for a VM's serial console, you can connect to a regional serial console.

The serial console authenticates users with SSH keys. Specifically, you must add your public SSH key to the project or instance metadata and store your private key on the local machine from which you want to connect. The gcloud CLI and the Google Cloud console automatically add SSH keys to the project for you. If you are using a third-party client, you might need to add SSH keys manually.

  gcloud compute connect-to-serial-port VM_NAME 

      --port=PORT_NUMBER
  

Disconnecting from the serial console

To disconnect from the serial console:

1. Press the ENTER key.
2. Type ~. (tilde, followed by a period).

You can discover other commands by typing ~? or by examining the man page for SSH:

man ssh

Don't try to disconnect using any of the following methods:

• The CTRL+ALT+DELETE key combination or other similar combinations. This doesn't work because the serial console does not recognize PC keyboard combinations.

• The exit or logout command doesn't work because the guest is not aware of any network or modem connections. Using this command causes the console to close and then reopen again, and you remain connected to the session. If you would like to enable exit and logout commands for your session, you can enable it by setting the on-dtr-low option.

Connecting to a serial console with a login prompt

If you are trying to troubleshoot an issue with a VM that has booted completely or trying to troubleshoot an issue that occurs after VM has booted past single user mode, you might be prompted for login information when trying to access the serial console.

By default, Google-supplied Linux system images are not configured to allow password-based logins for local users. However, Google-supplied Windows images are configured to allow password-based logins for local users.

If your VM is running an image that is preconfigured with serial port logins, you need to set up a local password on the VM so that you can sign in to the serial console, if prompted. You can set up a local password after connecting to the VM or by using a start-up script.

Setting up a local password using a startup script

You can use a startup script to set up a local password that lets you connect to the serial console during or after VM creation.

The following instructions describe how to set up a local password after VM creation.

1. In the Google Cloud console, go to the VM instances page.

   Go to VM instances

2. Select the VM for which you want to add the local password.

3. Click Edit.

   Linux

   1. Navigate to the Metadata > Automation section.

   2. If the VM has an existing startup script, copy it and paste it somewhere safe.

   3. Add the following startup script:

      #!/bin/bash
      useradd USERNAME
      echo USERNAME:PASSWORD | chpasswd
      usermod -aG google-sudoers USERNAME
      

      Replace the following:

      • USERNAME: the username that you want to add
      • PASSWORD: the password for the username. Avoid simple passwords, as some operating systems may requires minimal password length and complexity.

   Windows

   1. Navigate to the Custom metadata section.
   2. If the VM has an expisting startup script, copy it and paste it somewhere safe.
   3. Click Add item.
   4. In the Key field, enter windows-startup-script-cmd.

   5. In the Value field, enter the following script:

      net user USERNAME PASSWORD /ADD /Y
      net localgroup administrators USERNAME /ADD
      

      Replace the following:

      • USERNAME: the username that you want to add
      • PASSWORD: the password for the username

4. Click Save.

5. To reboot the VM, click Reset. For more information, see Reset a VM.

6. Connect to the serial console.

7. Enter your login information when prompted.

8. Remove the startup script from the VM after the user has been created.

Setting up a local password using passwd on the VM

The following instructions describe how to set up a local password for a user on a VM so that the user can log on to the serial console of that VM using the specified password.

1. Connect to the VM. Replace instance-name with the name of your instance.

   gcloud compute ssh instance-name

2. On the VM, create a local password with the following command. This sets a password for the user that you are currently logged in as.

   sudo passwd $(whoami)

3. Follow the prompts to create a password.

4. Next, log out of the instance and connect to the serial console.

5. Enter your login information when prompted.

Setting up a login on other serial ports

Login prompts are enabled on port 1 by default on most Linux operating systems. However, port 1 can often be overwhelmed by logging data and other information being printed to the port. Instead, you can choose to enable a login prompt on another port, such as port 2 (ttyS1), by executing one of the following commands on your VM. You can see a list of available ports for an VM in Understanding serial port numbering.

The following table lists images preconfigured with a serial console login and the default ports.

To enable login prompts on additional serial ports, complete the following instructions.

 sudo sed -i~ -e 's/^#T([01])/T\1/' /etc/inittab
 sudo telinit q

Understanding serial port numbering

Each virtual machine instance has four serial ports. For consistency with the getSerialPortOutput API, each port is numbered 1 through 4. Linux and other similar systems number their serial ports 0 through 3. For example, on many operating system images, the corresponding devices are /dev/ttyS0 through /dev/ttyS3. Windows refers to serial ports as COM1 through COM4. To connect to what Windows considers COM3 and Linux considers ttyS2, you would specify port 3. Use the following table to help you figure out which port you want to connect to.

Note that many Linux images use port 1 (/dev/ttyS0) for logging messages from the kernel and system programs.

Sending a serial break

The Magic SysRq key feature lets you perform low-level tasks regardless of the system's state. For example, you can sync file systems, reboot the instance, end processes, and unmount file systems using the Magic SysRq key feature.

To send a Magic SysRq command using a simulated serial break:

1. Press the ENTER key.
2. Type ~B (tilde, followed by uppercase B).
3. Type the Magic SysRq command.

Viewing serial console audit logs

Compute Engine provides audit logs to track who has connected and disconnected from an instance's serial console. To view logs, you must have permissions for the Logs Viewer or be a project viewer or editor.

1. In the Google Cloud console, go to the Logs Explorer page.

   Go to Logs Explorer

2. Expand the drop-down menu and select GCE VM Instance.
3. In the search bar, type ssh-serialport.googleapis.com and press Enter.

4. A list of audit logs appears. The logs describe connections and disconnections from a serial console. Expand any of the entries to get more information:

   

For any of the audit logs, you can:

1. Expand the protoPayload property.
2. Look for methodName to see activity this log applies to (either a connection or disconnection request). For example, if this log tracks a disconnection from the serial console, the method name would say "google.ssh-serialport.v1.disconnect". Similarly, a connection log would say "google.ssh-serialport.v1.connect". An audit log entry is recorded at the beginning and end of each session on the serial console.

There are different audit log properties for different log types. For example, audit logs relating to connections have properties that are specific to connection logs, while audit logs for disconnections have their own set of properties. There are certain audit log properties that are also shared between both log types.

All serial console logs

The following table provides audit log properties and their values for all serial console logs:

Connection logs

The following table provides audit log properties and their values specific for connection logs:

Disconnection logs

The following table provides audit log properties and their values specific for disconnection logs:

Failed connection logs

When a connection fails, Compute Engine creates an audit log entry. A failed connection log looks very similar to a successful connection entry, but has the following properties to indicate a failed connection.

Disabling interactive serial console access

You can disable interactive serial console access by changing metadata on the specific instance or project, or by setting an Organization Policy that disables interactive serial console access to all VM instances for one or more projects that are part of the organization.

Disabling interactive serial console on a particular instance or project

Project owners and editors, as well as users who have been granted the compute.instanceAdmin.v1 role, can disable access to the serial console by changing the metadata on the particular instance or project. Similar to enabling serial console access, set the serial-port-enable metadata to FALSE:

serial-port-enable=FALSE

For example, using the Google Cloud CLI, you can apply this metadata to a specific instance like so:

gcloud compute instances add-metadata instance-name \
    --metadata=serial-port-enable=FALSE

To apply the metadata to the project:

gcloud compute project-info add-metadata \
    --metadata=serial-port-enable=FALSE

Disabling interactive serial console access through Organization Policy

If you have been granted the orgpolicy.policyAdmin role on the organization, you can set an organization policy that prevents interactive access to the serial console, regardless of whether interactive serial console access is enabled on the metadata server. After the organization policy is set, the policy effectively overrides the serial-port-enable metadata key, and no users of the organization or project can enable interactive serial console access. By default, this constraint is set to FALSE.

The constraint for disabling interactive serial console access is as follows:

compute.disableSerialPortAccess

Complete the following instructions to set this policy on the organization. After setting up a policy, you can grant exemptions on a per-project basis.

gcloud resource-manager org-policies enable-enforce \
    --organization organization-id compute.disableSerialPortAccess

 POST https://cloudresourcemanager.googleapis.com/v1/organization-name:setOrgPolicy

"constraint": "constraints/compute.disableSerialPortAccess"

 {
   "policy":
   {
     "booleanPolicy":
     {
       "enforced": TRUE
     },
     "constraint": "constraints/compute.disableSerialPortAccess"
   }
 }
 

The policy is immediately effective, so any projects under the organization immediately stop allowing interactive access to the serial console.

To temporarily disable the policy, use the disable-enforce command:

gcloud resource-manager org-policies disable-enforce \
    --organization organization-id compute.disableSerialPortAccess

Alternatively, you can make an API request where the request body sets the enforced parameter to FALSE:

{
  "policy":
  {
    "booleanPolicy":
    {
      "enforced": FALSE
    },
    "constraint": "constraints/compute.disableSerialPortAccess"
  }
}

Setting the organization policy at the project level

You can set the same organizational policy on a per-project basis. This overrides the setting at the organization level.

gcloud resource-manager org-policies disable-enforce \
    --project project-id compute.disableSerialPortAccess

POST https://cloudresourcemanager.googleapis.com/v1/projects/project-id:setOrgPolicy

"constraint": "constraints/compute.disableSerialPortAccess"

{
  "policy":
  {
    "booleanPolicy":
    {
      "enforced": FALSE
    },
    "constraint": "constraints/compute.disableSerialPortAccess"
  }
}

Tips and tricks

• If you are having trouble connecting using a standard SSH client, but gcloud compute connect-to-serial-port connects successfully, it might be helpful to run gcloud compute connect-to-serial-port with the --dry-run command-line option to see the SSH command that it would have run on your behalf, and compare the options with the command you are using.

• Setting the bit rate, also known as baud rate, you can set any bit rate you like, such as stty 9600, but the feature normally forces the effective rate to 115,200 bps (~11.5kB/sec). This is because many public images default to slow bit rates, such as 9,600 on the serial console, and would boot slowly.

• Some OS images have inconvenient defaults on the serial port. For example, on CentOS 7, the stty icrnl default for the Enter key on the console is to send a CR, also known as ^M. The bash shell might mask this until you try to set a password, at which point you might wonder why it seems stuck at the password: prompt.

• Some public images have job control keys that are disabled by default if you attach a shell to a port in certain ways. Some examples of these keys include ^Z and ^C. The setsid command might fix this. Otherwise, if you see a job control is disabled in this shell message, be careful not to run commands that you will need to interrupt.

• You might find it helpful to tell the system the size of the window you're using, so that bash and editors can manage it properly. Otherwise, you might experience odd display behavior because bash or editors attempt to manipulate the display based on incorrect assumptions about the number of rows and columns available. Use the stty rows Y cols X command and stty -a flag to see what the setting is. For example: stty rows 60 cols 120 (if your window is 120 chars by 60 lines).

• If, for example, you connect using SSH from machine A to machine B, and then to machine C, creating a nested SSH session, and you want to use tilde (~) commands to disconnect or send a serial break signal, you will need to add enough extra tilde characters to the command to get to the right SSH client. A command following a single tilde is interpreted by the SSH client on machine A; a command following two consecutive tildes (Enter~~) is interpreted by the client on machine B, and so forth. You only need to press Enter one time because that is passed all the way through to the innermost SSH destination. This is true for any use of SSH clients that provide the tilde escape feature.

  If you lose track of how many tilde characters you need, press the Enter key and then type tilde characters one at a time until the instance echoes the tilde back. This echo indicates that you have reached the end of the chain and you now know that to send a tilde command to the most nested SSH client, you need one less tilde than however many tildes you typed.

Advanced options

You can also use the following advanced options with the serial port.

Controlling max connections

You can set the max-connections property to control how many concurrent connections can be made to this serial port at a time. The default and maximum number of connections is 5. For example:

gcloud compute connect-to-serial-port instance-name \
    --port port-number \
    --extra-args max-connections=3

ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.max-connections=3@ssh-serialport.googleapis.com

Setting replay options

By default, each time you connect to the serial console, you will receive a replay of the last 10 lines of data, regardless of whether the last 10 lines have been seen by another SSH client. You can change this setting and control how many and which lines are returned by setting the following options:

• replay-lines=N: Set N to the number of lines you want replayed. For example, if N is 50, then the last 50 lines of the console output is included.
• replay-bytes=N: Replays the most recent N bytes. You can also set N to new which replays all output that has not yet been sent to any client.
• replay-from=N: Replays output starting from an absolute byte index that you provide. You can get the current byte index of serial console output by making a getSerialPortOutput request. If you set replay-from, all other replay options are ignored.

With the Google Cloud CLI, append the following to your connect-to-serial-port command, where N is the specified number of lines (or bytes or absolute byte index, depending on which replay option you are selecting):

--extra-args replay-lines=N

If you are using a third-party SSH client, provide this option in your SSH command:

ssh -i private-ssh-key-file -p 9600 myproject.us-central1-f.example-instance.jane.port=3.replay-lines=N@ssh-serialport.googleapis.com

You can also use a combination of these options as well. For example:

replay-lines=N and replay-bytes=new

Replay the specified number of lines OR replay all output not previously sent to any client, whichever is larger. The first client to connect with this flag combination will see all the output that has been sent to the serial port, and clients that connect subsquently will only see the last N lines. Examples:

gcloud compute connect-to-serial-port instance-name--port port-number --extra-args replay-lines=N,replay-bytes=new

ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.replay-lines=N.replay-bytes=new@ssh-serialport.googleapis.com

replay-lines=N and replay-bytes=M

Replay lines up to, but not more than, the number of lines or bytes described by these flags, whichever is less. This option won't replay more than N or M bytes.

gcloud compute connect-to-serial-port instance-name--port port-number --extra-args replay-lines=N,replay-bytes=M

ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.replay-lines=N.replay-bytes=M@ssh-serialport.googleapis.com

Handling dropped output

The most recent 1 MiB of output for each serial port is always available and generally, your SSH client shouldn't miss any output from the serial port. If, for some reason, your SSH client stops accepting output for a period of time but does not disconnect, and more than 1 MiB of new data is produced, your SSH client might miss some output. When your SSH client is not accepting data fast enough to keep up with the output on the serial console port, you can set the on-dropped-output property to determine how the console behaves.

Set any of the following applicable options with this property:

• insert-stderr-note: Insert a note on the SSH client's stderr indicating that output was dropped. This is the default option.
• ignore: Silently drops output and does nothing.
• disconnect: Stop the connection.

For example:

gcloud compute connect-to-serial-port instance-name \
    --port port-number \
    --extra-args on-dropped-output=ignore

ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.on-dropped-output=ignore@ssh-serialport.googleapis.com

Enabling disconnect using exit or logout commands

You can enable disconnecting on exit or logout commands by setting the on-dtr-low property to disconnect when you connect to the serial console.

On the Google Cloud CLI, append the following flag to your connect-to-serial-port command:

--extra-args on-dtr-low=disconnect

If you are using a third-party SSH client, provide this option in your SSH command:

ssh -i private-ssh-key-file -p 9600 myproject.us-central1-f.example-instance.jane.port=3.on-dtr-low=disconnect@ssh-serialport.googleapis.com

Enabling the disconnect option might cause your instance to disconnect one or more times when you are rebooting the instance because the operating system resets the serial ports while booting up.

The default setting for the on-dtr-low option is none. If you use the default setting none, you can reboot your instance without being disconnected from the serial console, but the console won't disconnect through normal means such as exit or logout commands, or normal key combinations like Ctrl+D.

What's next

• Learn more about the getSerialPortOutput API.
• Learn how to retain and view serial port output even after a VM instance is deleted.
• Read more troubleshooting tips.
• Learn more about applying metadata.
• Learn about SSH keys.