Method: images.annotate

{
  "image": {
    object(Image)
  },
  "features": [
    {
      object(Feature)
    }
  ],
  "imageContext": {
    object(ImageContext)
  }
}

object(Image)

The image to be processed.

object(Feature)

Requested features.

object(ImageContext)

Additional context that may accompany the image.

{
  "content": string,
  "source": {
    object(ImageSource)
  }
}

string (bytes format)

Image content, represented as a stream of bytes. Note: As with all bytes fields, protobuffers use a pure binary representation, whereas JSON representations use base64.

A base64-encoded string.

object(ImageSource)

Google Cloud Storage image location, or publicly-accessible image URL. If both content and source are provided for an image, content takes precedence and is used to perform the image annotation request.

{
  "gcsImageUri": string,
  "imageUri": string
}

string

Use imageUri instead.

The Google Cloud Storage URI of the form gs://bucket_name/object_name. Object versioning is not supported. See Google Cloud Storage Request URIs for more info.

string

The URI of the source image. Can be either:

1. A Google Cloud Storage URI of the form gs://bucket_name/object_name. Object versioning is not supported. See Google Cloud Storage Request URIs for more info.

2. A publicly-accessible image HTTP/HTTPS URL. When fetching images from HTTP/HTTPS URLs, Google cannot guarantee that the request will be completed. Your request may fail if the specified host denies the request (e.g. due to request throttling or DOS prevention), or if Google throttles requests to the site for abuse prevention. You should not depend on externally-hosted images for production applications.

When both gcsImageUri and imageUri are specified, imageUri takes precedence.

{
  "faceAnnotations": [
    {
      object(FaceAnnotation)
    }
  ],
  "landmarkAnnotations": [
    {
      object(EntityAnnotation)
    }
  ],
  "logoAnnotations": [
    {
      object(EntityAnnotation)
    }
  ],
  "labelAnnotations": [
    {
      object(EntityAnnotation)
    }
  ],
  "localizedObjectAnnotations": [
    {
      object(LocalizedObjectAnnotation)
    }
  ],
  "textAnnotations": [
    {
      object(EntityAnnotation)
    }
  ],
  "fullTextAnnotation": {
    object(TextAnnotation)
  },
  "safeSearchAnnotation": {
    object(SafeSearchAnnotation)
  },
  "imagePropertiesAnnotation": {
    object(ImageProperties)
  },
  "cropHintsAnnotation": {
    object(CropHintsAnnotation)
  },
  "webDetection": {
    object(WebDetection)
  },
  "productSearchResults": {
    object(ProductSearchResults)
  },
  "error": {
    object(Status)
  },
  "context": {
    object(ImageAnnotationContext)
  }
}

object(FaceAnnotation)

If present, face detection has completed successfully.

object(EntityAnnotation)

If present, landmark detection has completed successfully.

object(EntityAnnotation)

If present, logo detection has completed successfully.

object(EntityAnnotation)

If present, label detection has completed successfully.

object(LocalizedObjectAnnotation)

If present, localized object detection has completed successfully. This will be sorted descending by confidence score.

object(EntityAnnotation)

If present, text (OCR) detection has completed successfully.

object(TextAnnotation)

If present, text (OCR) detection or document (OCR) text detection has completed successfully. This annotation provides the structural hierarchy for the OCR detected text.

object(SafeSearchAnnotation)

If present, safe-search annotation has completed successfully.

object(ImageProperties)

If present, image properties were extracted successfully.

object(CropHintsAnnotation)

If present, crop hints have completed successfully.

object(WebDetection)

If present, web detection has completed successfully.

object(ProductSearchResults)

If present, product search has completed successfully.

object(Status)

If set, represents the error message for the operation. Note that filled-in image annotations are guaranteed to be correct, even when error is set.

object(ImageAnnotationContext)

If present, contextual information is needed to understand where this image comes from.

{
  "boundingPoly": {
    object(BoundingPoly)
  },
  "fdBoundingPoly": {
    object(BoundingPoly)
  },
  "landmarks": [
    {
      object(Landmark)
    }
  ],
  "rollAngle": number,
  "panAngle": number,
  "tiltAngle": number,
  "detectionConfidence": number,
  "landmarkingConfidence": number,
  "joyLikelihood": enum(Likelihood),
  "sorrowLikelihood": enum(Likelihood),
  "angerLikelihood": enum(Likelihood),
  "surpriseLikelihood": enum(Likelihood),
  "underExposedLikelihood": enum(Likelihood),
  "blurredLikelihood": enum(Likelihood),
  "headwearLikelihood": enum(Likelihood)
}

object(BoundingPoly)

The bounding polygon around the face. The coordinates of the bounding box are in the original image's scale, as returned in ImageParams. The bounding box is computed to "frame" the face in accordance with human expectations. It is based on the landmarker results. Note that one or more x and/or y coordinates may not be generated in the BoundingPoly (the polygon will be unbounded) if only a partial face appears in the image to be annotated.

object(BoundingPoly)

The fdBoundingPoly bounding polygon is tighter than the boundingPoly, and encloses only the skin part of the face. Typically, it is used to eliminate the face from any image analysis that detects the "amount of skin" visible in an image. It is not based on the landmarker results, only on the initial face detection, hence the

fd

(face detection) prefix.

object(Landmark)

Detected face landmarks.

number

Roll angle, which indicates the amount of clockwise/anti-clockwise rotation of the face relative to the image vertical about the axis perpendicular to the face. Range [-180,180].

number

Yaw angle, which indicates the leftward/rightward angle that the face is pointing relative to the vertical plane perpendicular to the image. Range [-180,180].

number

Pitch angle, which indicates the upwards/downwards angle that the face is pointing relative to the image's horizontal plane. Range [-180,180].

number

Detection confidence. Range [0, 1].

number

Face landmarking confidence. Range [0, 1].

enum(Likelihood)

Joy likelihood.

enum(Likelihood)

Sorrow likelihood.

enum(Likelihood)

Anger likelihood.

enum(Likelihood)

Surprise likelihood.

enum(Likelihood)

Under-exposed likelihood.

enum(Likelihood)

Blurred likelihood.

enum(Likelihood)

Headwear likelihood.

{
  "type": enum(Type),
  "position": {
    object(Position)
  }
}

enum(Type)

Face landmark type.

object(Position)

Face landmark position.

{
  "x": number,
  "y": number,
  "z": number
}

number

X coordinate.

number

Y coordinate.

number

Z coordinate (or depth).

{
  "mid": string,
  "locale": string,
  "description": string,
  "score": number,
  "confidence": number,
  "topicality": number,
  "boundingPoly": {
    object(BoundingPoly)
  },
  "locations": [
    {
      object(LocationInfo)
    }
  ],
  "properties": [
    {
      object(Property)
    }
  ]
}

string

Opaque entity ID. Some IDs may be available in Google Knowledge Graph Search API.

string

The language code for the locale in which the entity textual description is expressed.

string

Entity textual description, expressed in its locale language.

number

Overall score of the result. Range [0, 1].

number

Deprecated. Use score instead. The accuracy of the entity detection in an image. For example, for an image in which the "Eiffel Tower" entity is detected, this field represents the confidence that there is a tower in the query image. Range [0, 1].

number

The relevancy of the ICA (Image Content Annotation) label to the image. For example, the relevancy of "tower" is likely higher to an image containing the detected "Eiffel Tower" than to an image containing a detected distant towering building, even though the confidence that there is a tower in each image may be the same. Range [0, 1].

object(BoundingPoly)

Image region to which this entity belongs. Not produced for LABEL_DETECTION features.

object(LocationInfo)

The location information for the detected entity. Multiple LocationInfo elements can be present because one location may indicate the location of the scene in the image, and another location may indicate the location of the place where the image was taken. Location information is usually present for landmarks.

object(Property)

Some entities may have optional user-supplied Property (name/value) fields, such a score or string that qualifies the entity.

{
  "latLng": {
    object(LatLng)
  }
}

object(LatLng)

lat/long location coordinates.

{
  "name": string,
  "value": string,
  "uint64Value": string
}

string

Name of the property.

string

Value of the property.

string

Value of numeric properties.

{
  "mid": string,
  "languageCode": string,
  "name": string,
  "score": number,
  "boundingPoly": {
    object(BoundingPoly)
  }
}

string

Object ID that should align with EntityAnnotation mid.

string

The BCP-47 language code, such as "en-US" or "sr-Latn". For more information, see http://www.unicode.org/reports/tr35/#Unicode_locale_identifier.

string

Object name, expressed in its languageCode language.

number

Score of the result. Range [0, 1].

object(BoundingPoly)

Image region to which this object belongs. This must be populated.

{
  "pages": [
    {
      object(Page)
    }
  ],
  "text": string
}

object(Page)

List of pages detected by OCR.

string

UTF-8 text detected on the pages.

{
  "property": {
    object(TextProperty)
  },
  "width": number,
  "height": number,
  "blocks": [
    {
      object(Block)
    }
  ],
  "confidence": number
}

object(TextProperty)

Additional information detected on the page.

number

Page width. For PDFs the unit is points. For images (including TIFFs) the unit is pixels.

number

Page height. For PDFs the unit is points. For images (including TIFFs) the unit is pixels.

object(Block)

List of blocks of text, images etc on this page.

number

Confidence of the OCR results on the page. Range [0, 1].

{
  "detectedLanguages": [
    {
      object(DetectedLanguage)
    }
  ],
  "detectedBreak": {
    object(DetectedBreak)
  }
}

object(DetectedLanguage)

A list of detected languages together with confidence.

object(DetectedBreak)

Detected start or end of a text segment.

{
  "languageCode": string,
  "confidence": number
}

string

The BCP-47 language code, such as "en-US" or "sr-Latn". For more information, see http://www.unicode.org/reports/tr35/#Unicode_locale_identifier.

number

Confidence of detected language. Range [0, 1].

{
  "type": enum(BreakType),
  "isPrefix": boolean
}

enum(BreakType)

Detected break type.

boolean

True if break prepends the element.

{
  "property": {
    object(TextProperty)
  },
  "boundingBox": {
    object(BoundingPoly)
  },
  "paragraphs": [
    {
      object(Paragraph)
    }
  ],
  "blockType": enum(BlockType),
  "confidence": number
}

object(TextProperty)

Additional information detected for the block.

object(BoundingPoly)

The bounding box for the block. The vertices are in the order of top-left, top-right, bottom-right, bottom-left. When a rotation of the bounding box is detected the rotation is represented as around the top-left corner as defined when the text is read in the 'natural' orientation. For example:

• when the text is horizontal it might look like:

    0----1
    |    |
    3----2

• when it's rotated 180 degrees around the top-left corner it becomes:

    2----3
    |    |
    1----0

and the vertex order will still be (0, 1, 2, 3).

object(Paragraph)

List of paragraphs in this block (if this blocks is of type text).

enum(BlockType)

Detected block type (text, image etc) for this block.

number

Confidence of the OCR results on the block. Range [0, 1].

{
  "property": {
    object(TextProperty)
  },
  "boundingBox": {
    object(BoundingPoly)
  },
  "words": [
    {
      object(Word)
    }
  ],
  "confidence": number
}

object(TextProperty)

Additional information detected for the paragraph.

object(BoundingPoly)

The bounding box for the paragraph. The vertices are in the order of top-left, top-right, bottom-right, bottom-left. When a rotation of the bounding box is detected the rotation is represented as around the top-left corner as defined when the text is read in the 'natural' orientation. For example: * when the text is horizontal it might look like: 0----1 | | 3----2 * when it's rotated 180 degrees around the top-left corner it becomes: 2----3 | | 1----0 and the vertex order will still be (0, 1, 2, 3).

object(Word)

List of words in this paragraph.

number

Confidence of the OCR results for the paragraph. Range [0, 1].

{
  "property": {
    object(TextProperty)
  },
  "boundingBox": {
    object(BoundingPoly)
  },
  "symbols": [
    {
      object(Symbol)
    }
  ],
  "confidence": number
}

object(TextProperty)

Additional information detected for the word.

object(BoundingPoly)

The bounding box for the word. The vertices are in the order of top-left, top-right, bottom-right, bottom-left. When a rotation of the bounding box is detected the rotation is represented as around the top-left corner as defined when the text is read in the 'natural' orientation. For example: * when the text is horizontal it might look like: 0----1 | | 3----2 * when it's rotated 180 degrees around the top-left corner it becomes: 2----3 | | 1----0 and the vertex order will still be (0, 1, 2, 3).

object(Symbol)

List of symbols in the word. The order of the symbols follows the natural reading order.

number

Confidence of the OCR results for the word. Range [0, 1].

{
  "property": {
    object(TextProperty)
  },
  "boundingBox": {
    object(BoundingPoly)
  },
  "text": string,
  "confidence": number
}

object(TextProperty)

Additional information detected for the symbol.

object(BoundingPoly)

The bounding box for the symbol. The vertices are in the order of top-left, top-right, bottom-right, bottom-left. When a rotation of the bounding box is detected the rotation is represented as around the top-left corner as defined when the text is read in the 'natural' orientation. For example: * when the text is horizontal it might look like: 0----1 | | 3----2 * when it's rotated 180 degrees around the top-left corner it becomes: 2----3 | | 1----0 and the vertice order will still be (0, 1, 2, 3).

string

The actual UTF-8 representation of the symbol.

number

Confidence of the OCR results for the symbol. Range [0, 1].

{
  "adult": enum(Likelihood),
  "spoof": enum(Likelihood),
  "medical": enum(Likelihood),
  "violence": enum(Likelihood),
  "racy": enum(Likelihood)
}

enum(Likelihood)

Represents the adult content likelihood for the image. Adult content may contain elements such as nudity, pornographic images or cartoons, or sexual activities.

enum(Likelihood)

Spoof likelihood. The likelihood that an modification was made to the image's canonical version to make it appear funny or offensive.

enum(Likelihood)

Likelihood that this is a medical image.

enum(Likelihood)

Likelihood that this image contains violent content.

enum(Likelihood)

Likelihood that the request image contains racy content. Racy content may include (but is not limited to) skimpy or sheer clothing, strategically covered nudity, lewd or provocative poses, or close-ups of sensitive body areas.

{
  "dominantColors": {
    object(DominantColorsAnnotation)
  }
}

object(DominantColorsAnnotation)

If present, dominant colors completed successfully.

{
  "colors": [
    {
      object(ColorInfo)
    }
  ]
}

object(ColorInfo)

RGB color values with their score and pixel fraction.

{
  "color": {
    object(Color)
  },
  "score": number,
  "pixelFraction": number
}

object(Color)

RGB components of the color.

number

Image-specific score for this color. Value in range [0, 1].

number

The fraction of pixels the color occupies in the image. Value in range [0, 1].

{
  "cropHints": [
    {
      object(CropHint)
    }
  ]
}

object(CropHint)

Crop hint results.

{
  "boundingPoly": {
    object(BoundingPoly)
  },
  "confidence": number,
  "importanceFraction": number
}

object(BoundingPoly)

The bounding polygon for the crop region. The coordinates of the bounding box are in the original image's scale, as returned in ImageParams.

number

Confidence of this being a salient region. Range [0, 1].

number

Fraction of importance of this salient region with respect to the original image.

{
  "webEntities": [
    {
      object(WebEntity)
    }
  ],
  "fullMatchingImages": [
    {
      object(WebImage)
    }
  ],
  "partialMatchingImages": [
    {
      object(WebImage)
    }
  ],
  "pagesWithMatchingImages": [
    {
      object(WebPage)
    }
  ],
  "visuallySimilarImages": [
    {
      object(WebImage)
    }
  ],
  "bestGuessLabels": [
    {
      object(WebLabel)
    }
  ]
}

object(WebEntity)

Deduced entities from similar images on the Internet.

object(WebImage)

Fully matching images from the Internet. Can include resized copies of the query image.

object(WebImage)

Partial matching images from the Internet. Those images are similar enough to share some key-point features. For example an original image will likely have partial matching for its crops.

object(WebPage)

Web pages containing the matching images from the Internet.

object(WebImage)

The visually similar image results.

object(WebLabel)

The service's best guess as to the topic of the request image. Inferred from similar images on the open web.

{
  "entityId": string,
  "score": number,
  "description": string
}

string

Opaque entity ID.

number

Overall relevancy score for the entity. Not normalized and not comparable across different image queries.

string

Canonical description of the entity, in English.

{
  "url": string,
  "score": number
}

string

The result image URL.

number

(Deprecated) Overall relevancy score for the image.

{
  "url": string,
  "score": number,
  "pageTitle": string,
  "fullMatchingImages": [
    {
      object(WebImage)
    }
  ],
  "partialMatchingImages": [
    {
      object(WebImage)
    }
  ]
}

string

The result web page URL.

number

(Deprecated) Overall relevancy score for the web page.

string

Title for the web page, may contain HTML markups.

object(WebImage)

Fully matching images on the page. Can include resized copies of the query image.

object(WebImage)

Partial matching images on the page. Those images are similar enough to share some key-point features. For example an original image will likely have partial matching for its crops.

{
  "label": string,
  "languageCode": string
}

string

Label for extra metadata.

string

The BCP-47 language code for label, such as "en-US" or "sr-Latn". For more information, see http://www.unicode.org/reports/tr35/#Unicode_locale_identifier.

{
  "indexTime": string,
  "results": [
    {
      object(Result)
    }
  ],
  "productGroupedResults": [
    {
      object(GroupedResult)
    }
  ]
}

string (Timestamp format)

Timestamp of the index which provided these results. Changes made after this time are not reflected in the current results.

A timestamp in RFC3339 UTC "Zulu" format, accurate to nanoseconds. Example: "2014-10-02T15:01:23.045123456Z".

object(Result)

List of results, one for each product match.

object(GroupedResult)

List of results grouped by products detected in the query image. Each entry corresponds to one bounding polygon in the query image, and contains the matching products specific to that region. There may be duplicate product matches in the union of all the per-product results.

{
  "product": {
    object(Product)
  },
  "score": number,
  "image": string
}

object(Product)

The Product.

number

A confidence level on the match, ranging from 0 (no confidence) to 1 (full confidence).

string

The resource name of the image from the product that is the closest match to the query.

{
  "boundingPoly": {
    object(BoundingPoly)
  },
  "results": [
    {
      object(Result)
    }
  ]
}

object(BoundingPoly)

The bounding polygon around the product detected in the query image.

object(Result)

List of results, one for each product match.

{
  "uri": string,
  "pageNumber": number
}

string

The URI of the file used to produce the image.

number

If the file was a PDF or TIFF, this field gives the page number within the file used to produce the image.