Troubleshoot Spanner Vertex AI integration

MODEL DDL statements and ML functions in Spanner invoke Vertex AI endpoints and can fail due to various reasons:

Error Code Error Message Possible cause Possible solution
CANCELLED Call to Vertex AI endpoint {ENDPOINT} was cancelled due to query cancellation. Query was cancelled by the client application. Investigate why your client cancelled the query.
DEADLINE_EXCEEDED Vertex AI endpoint {ENDPOINT} exceeded the call deadline. Query deadline is too short. Increase query deadline on the client.
- - Endpoint was too busy. See Vertex AI monitoring and deploy more nodes.
FAILED_PRECONDITION Vertex AI endpoint {ENDPOINT} returned failed precondition error. Endpoint has no models deployed. Deploy models to the endpoint.
INTERNAL Unknown error when accessing Vertex AI endpoint {ENDPOINT}. Unexpected internal error. Use failover endpoints or open a support ticket.
INVALID_ARGUMENT Invalid request to Vertex AI endpoint {ENDPOINT}. Make sure that Vertex AI endpoint and Spanner model schema match. Spanner model schema and Vertex AI endpoint schema do not match Update Spanner model's schema.
NOT_FOUND Vertex AI endpoint {ENDPOINT} not found. Endpoint was deleted. Update Spanner model's schema.
PERMISSION_DENIED Access to Vertex AI endpoint {ENDPOINT} was denied. Spanner service agent does not have permissions to access the endpoint Grant service agent role permissions
- - VPC SC error See Vertex AI error message and follow VPC SC troubleshooting
RESOURCE_EXHAUSTED Vertex AI endpoint {ENDPOINT} quota has been exceeded. Too many requests to Vertex AI. Increase online prediction quota
UNAVAILABLE Could not create service agent for project {PROJECT}. Service Usage API issue. Create service agent manually
- Vertex AI endpoint {ENDPOINT} is unavailable. Too many requests to Vertex AI. Deploy more nodes.
- - Vertex AI has a regional issue Use failover endpoints