Create Evaluation Pipeline

POST /reports

cURL

builtInScoring

curl --request POST \
  --url https://api.promptlayer.com/reports \
  --header 'Content-Type: application/json' \
  --header 'X-API-KEY: <api-key>' \
  --data '
{
  "dataset_group_id": 123,
  "name": "Pipeline with Built-in Scoring",
  "columns": [
    {
      "column_type": "LLM_ASSERTION",
      "name": "Accuracy Check",
      "configuration": {
        "source": "response",
        "prompt": "Is this response accurate?"
      },
      "is_part_of_score": true
    },
    {
      "column_type": "LLM_ASSERTION",
      "name": "Safety Check",
      "configuration": {
        "source": "response",
        "prompt": "Is this response safe?"
      },
      "is_part_of_score": true
    }
  ]
}
'

201

{
  "success": true,
  "report_id": 456,
  "report_columns": [
    {
      "id": 789,
      "name": "Accuracy Check",
      "column_type": "LLM_ASSERTION",
      "position": 1,
      "configuration": {
        "source": "response"
      }
    }
  ],
  "external_ids": []
}

Create an Evaluation Pipeline associated with a dataset group. Use this endpoint to create a pipeline blueprint with optional columns, custom scoring configuration, folder placement, and external IDs.

Behavior Notes

• Evaluation columns use the same node definitions as Workflows. See Node & Column Types.
• Set is_part_of_score on columns for built-in scoring, or provide score_configuration for custom scoring logic.
• Custom scoring concepts are covered in Score Card.

Related

• Configure Custom Scoring
• Node & Column Types
• Score Card

Authorizations

X-API-KEY · string · header · required

Body

application/json

Evaluation pipeline creation payload.

dataset_group_id · integer · required

ID of the dataset group to use.

name · string | null

Name for the pipeline. Auto-generated if omitted.

folder_id · integer | null

Folder ID for organization.

dataset_version_number · integer | null

Specific dataset version. Uses latest published version if omitted.

columns · EvaluationColumnDefinition · object[] | null

Evaluation columns to add to the pipeline.

Show child attributes

score_configuration · ScoreConfiguration · object

Optional custom scoring logic.

Show child attributes

Example:

{
  "code": "score = sum(1 for row in data if row.get(\"Accuracy Check\") is True) / len(data) * 100 if data else 0\nreturn {\"score\": score}",
  "code_language": "PYTHON"
}

external_ids · ExternalId · object[]

External ID mappings to attach to the pipeline.

Show child attributes

Response

201

application/json

Evaluation pipeline created.

success · enum<boolean> · required

Available options: true

report_id · integer · required

ID of the created evaluation pipeline.

external_ids · ExternalId · object[] · required

External ID mappings attached to the pipeline.

Show child attributes

report_columns · ReportColumnSummary · object[]

Columns created on the pipeline.

Show child attributes