Tutorial: Realizar una evaluación con el SDK de Python

Antes de empezar

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

   In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Roles required to select or create a project

   • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
   • Create a project: To create a project, you need the Project Creator (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

   Go to project selector

   Verify that billing is enabled for your Google Cloud project.

   In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Roles required to select or create a project

   • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
   • Create a project: To create a project, you need the Project Creator (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

   Go to project selector

   Verify that billing is enabled for your Google Cloud project.

2. Instala el SDK de Vertex AI para Python con la dependencia del servicio de evaluación de IA generativa:

   !pip install google-cloud-aiplatform[evaluation]
   

3. Configura tus credenciales. Si estás ejecutando esta guía de inicio rápido en Colaboratory, ejecuta lo siguiente:

   from google.colab import auth
   auth.authenticate_user()
   

   En otros entornos, consulta Autenticarse en Vertex AI.

   Importar bibliotecas

   Importa tus bibliotecas y configura tu proyecto y tu ubicación.

   import pandas as pd
   
   import vertexai
   from vertexai.evaluation import EvalTask, PointwiseMetric, PointwiseMetricPromptTemplate
   from google.cloud import aiplatform
   
   PROJECT_ID = "PROJECT_ID"
   LOCATION = "LOCATION"
   EXPERIMENT_NAME = "EXPERIMENT_NAME"
   
   vertexai.init(
       project=PROJECT_ID,
       location=LOCATION,
   )

   Ten en cuenta que EXPERIMENT_NAME solo puede contener caracteres alfanuméricos en minúscula y guiones, hasta un máximo de 127 caracteres.

   Configurar métricas de evaluación según tus criterios

   La siguiente definición de métrica evalúa la calidad del texto generado por un modelo de lenguaje extenso en función de dos criterios: Fluency y Entertaining. El código define una métrica llamada custom_text_quality con esos dos criterios:

   custom_text_quality = PointwiseMetric(
       metric="custom_text_quality",
       metric_prompt_template=PointwiseMetricPromptTemplate(
           criteria={
               "fluency": (
                   "Sentences flow smoothly and are easy to read, avoiding awkward"
                   " phrasing or run-on sentences. Ideas and sentences connect"
                   " logically, using transitions effectively where needed."
               ),
               "entertaining": (
                   "Short, amusing text that incorporates emojis, exclamations and"
                   " questions to convey quick and spontaneous communication and"
                   " diversion."
               ),
           },
           rating_rubric={
               "1": "The response performs well on both criteria.",
               "0": "The response is somewhat aligned with both criteria",
               "-1": "The response falls short on both criteria",
           },
       ),
   )
   

   Preparar el conjunto de datos

   Añade el siguiente código para preparar el conjunto de datos:

   responses = [
       # An example of good custom_text_quality
       "Life is a rollercoaster, full of ups and downs, but it's the thrill that keeps us coming back for more!",
       # An example of medium custom_text_quality
       "The weather is nice today, not too hot, not too cold.",
       # An example of poor custom_text_quality
       "The weather is, you know, whatever.",
   ]
   
   eval_dataset = pd.DataFrame({
       "response" : responses,
   })
   

   Ejecutar la evaluación con tu conjunto de datos

   Realiza la evaluación:

   eval_task = EvalTask(
       dataset=eval_dataset,
       metrics=[custom_text_quality],
       experiment=EXPERIMENT_NAME
   )
   
   pointwise_result = eval_task.evaluate()
   
   

   Consulta los resultados de la evaluación de cada respuesta en el metrics_table DataFrame de Pandas:

   pointwise_result.metrics_table
   

   Limpieza

   Para evitar que se apliquen cargos en tu cuenta de Google Cloud por los recursos utilizados en esta página, sigue estos pasos.

   Elimina el ExperimentRun creado por la evaluación:

   aiplatform.ExperimentRun(
       run_name=pointwise_result.metadata["experiment_run"],
       experiment=pointwise_result.metadata["experiment"],
   ).delete()
   
   

   Siguientes pasos

   • Define tus métricas de evaluación.

   • Prepara tu conjunto de datos de evaluación.