Method: projects.locations.ragCorpora.ragFiles.import

Import files from Google Cloud Storage or Google Drive into a RagCorpus.

Endpoint

post https://aiplatform.googleapis.com/v1beta1/{parent}/ragFiles:import

Path parameters

parent string

Required. The name of the RagCorpus resource into which to import files. Format: projects/{project}/locations/{location}/ragCorpora/{ragCorpus}

Request body

The request body contains data with the following structure:

Fields
importRagFilesConfig object (ImportRagFilesConfig)

Required. The config for the RagFiles to be synced and imported into the RagCorpus. VertexRagDataService.ImportRagFiles.

Example request

Sync

Python


from vertexai.preview import rag
import vertexai

# TODO(developer): Update and un-comment below lines
# PROJECT_ID = "your-project-id"
# corpus_name = "projects/{PROJECT_ID}/locations/us-central1/ragCorpora/{rag_corpus_id}"
# paths = ["https://drive.google.com/file/123", "gs://my_bucket/my_files_dir"]  # Supports Google Cloud Storage and Google Drive Links

# Initialize Vertex AI API once per session
vertexai.init(project=PROJECT_ID, location="us-central1")

response = rag.import_files(
    corpus_name=corpus_name,
    paths=paths,
    chunk_size=512,  # Optional
    chunk_overlap=100,  # Optional
    max_embedding_requests_per_min=900,  # Optional
)
print(f"Imported {response.imported_rag_files_count} files.")
# Example response:
# Imported 2 files.

import_files_example.py

Async

Python


from vertexai.preview import rag
import vertexai

# TODO(developer): Update and un-comment below lines
# PROJECT_ID = "your-project-id"
# corpus_name = "projects/{PROJECT_ID}/locations/us-central1/ragCorpora/{rag_corpus_id}"

# Supports Google Cloud Storage and Google Drive Links
# paths = ["https://drive.google.com/file/d/123", "gs://my_bucket/my_files_dir"]

# Initialize Vertex AI API once per session
vertexai.init(project=PROJECT_ID, location="us-central1")

response = await rag.import_files_async(
    corpus_name=corpus_name,
    paths=paths,
    chunk_size=512,  # Optional
    chunk_overlap=100,  # Optional
    max_embedding_requests_per_min=900,  # Optional
)

result = await response.result()
print(f"Imported {result.imported_rag_files_count} files.")
# Example response:
# Imported 2 files.

import_files_async_example.py

Response body

If successful, the response body contains an instance of Operation.

ImportRagFilesConfig

Config for importing RagFiles.

Fields
ragFileChunkingConfig object (RagFileChunkingConfig)

Specifies the size and overlap of chunks after importing RagFiles.

ragFileParsingConfig object (RagFileParsingConfig)

Specifies the parsing config for RagFiles.

maxEmbeddingRequestsPerMin integer

Optional. The max number of queries per minute that this job is allowed to make to the embedding model specified on the corpus. This value is specific to this job and not shared across other import jobs. Consult the Quotas page on the project to set an appropriate value here. If unspecified, a default value of 1,000 QPM would be used.

Union field import_source. The source of the import. import_source can be only one of the following:
gcsSource object (GcsSource)

Google Cloud Storage location. Supports importing individual files as well as entire Google Cloud Storage directories. Sample formats: - gs://bucketName/my_directory/objectName/my_file.txt - gs://bucketName/my_directory

googleDriveSource object (GoogleDriveSource)

Google Drive location. Supports importing individual files as well as Google Drive folders.

slackSource object (SlackSource)

Slack channels with their corresponding access tokens.

jiraSource object (JiraSource)

Jira queries with their corresponding authentication.

sharePointSources object (SharePointSources)

SharePoint sources.

Union field partial_failure_sink. Optional. If provided, all partial failures are written to the sink. Deprecated. Prefer to use the import_result_sink. partial_failure_sink can be only one of the following:
partialFailureGcsSink
(deprecated) object (GcsDestination)

The Cloud Storage path to write partial failures to. Deprecated. Prefer to use importResultGcsSink.

partialFailureBigquerySink
(deprecated) object (BigQueryDestination)

The BigQuery destination to write partial failures to. It should be a bigquery table resource name (e.g. "bq://projectId.bqDatasetId.bqTableId"). The dataset must exist. If the table does not exist, it will be created with the expected schema. If the table exists, the schema will be validated and data will be added to this existing table. Deprecated. Prefer to use import_result_bq_sink.

JSON representation 
{
  "ragFileChunkingConfig": {
    object (RagFileChunkingConfig)
  },
  "ragFileParsingConfig": {
    object (RagFileParsingConfig)
  },
  "maxEmbeddingRequestsPerMin": integer,

  // Union field import_source can be only one of the following:
  "gcsSource": {
    object (GcsSource)
  },
  "googleDriveSource": {
    object (GoogleDriveSource)
  },
  "slackSource": {
    object (SlackSource)
  },
  "jiraSource": {
    object (JiraSource)
  },
  "sharePointSources": {
    object (SharePointSources)
  }
  // End of list of possible types for union field import_source.

  // Union field partial_failure_sink can be only one of the following:
  "partialFailureGcsSink": {
    object (GcsDestination)
  },
  "partialFailureBigquerySink": {
    object (BigQueryDestination)
  }
  // End of list of possible types for union field partial_failure_sink.
}

BigQueryDestination

The BigQuery location for the output content.

Fields
outputUri string

Required. BigQuery URI to a project or table, up to 2000 characters long.

When only the project is specified, the Dataset and Table is created. When the full table reference is specified, the Dataset must exist and table must not exist.

Accepted forms:

  • BigQuery path. For example: bq://projectId or bq://projectId.bqDatasetId or bq://projectId.bqDatasetId.bqTableId.
JSON representation 
{
  "outputUri": string
}

RagFileParsingConfig

Specifies the parsing config for RagFiles.

Fields
useAdvancedPdfParsing boolean

Whether to use advanced PDF parsing.

JSON representation 
{
  "useAdvancedPdfParsing": boolean
}