Skip to main content

Runs API

REST API endpoints for managing evaluation runs. Runs track the execution of evals with specific runtime configurations.

Endpoints​

MethodEndpointDescription
GET/api/projects/:projectId/runsList all runs
GET/api/projects/:projectId/runs/:idGet run by ID
POST/api/projects/:projectId/runsCreate runs for an eval
POST/api/projects/:projectId/runs/playgroundCreate a playground run (without eval)
POST/api/projects/:projectId/runs/chatCreate a chat run (live chat session)
POST/api/projects/:projectId/runs/:id/chatSend a message in a chat run
PUT/api/projects/:projectId/runs/:idUpdate a run
POST/api/projects/:projectId/runs/:id/retryRetry a run with system errors
DELETE/api/projects/:projectId/runs/:idDelete a run

List Runs​

GET /api/projects/:projectId/runs
GET /api/projects/:projectId/runs?evalId=<eval-id>

Query Parameters​

ParameterTypeDescription
evalIdstringFilter runs by eval ID (returns sorted by createdAt desc)
scenarioIdstringFilter runs by scenario ID (returns sorted by createdAt desc)
personaIdstringFilter runs by persona ID (returns sorted by createdAt desc)
connectorIdstringFilter runs by connector ID (returns sorted by createdAt desc, supports additional status filter)
statusstringFilter runs by status (used with connectorId or standalone)

Response​

[
{
"id": "run-uuid",
"evalId": "eval-uuid",
"personaId": "persona-uuid",
"scenarioId": "scenario-uuid",
"executionId": 1,
"status": "completed",
"startedAt": "2026-01-29T10:00:00.000Z",
"completedAt": "2026-01-29T10:00:15.000Z",
"messages": [
{ "role": "system", "content": "You are a test agent..." },
{ "role": "user", "content": "Hello" },
{ "role": "assistant", "content": "Hi there!" }
],
"latencyMs": 1500,
"result": {
"success": true,
"score": 0.95,
"reason": "Agent responded appropriately"
},
"createdAt": "2026-01-29T10:00:00.000Z",
"updatedAt": "2026-01-29T10:00:15.000Z"
}
]

Note: Connector and LLM provider information is configured at the Eval level. The messages array includes the system prompt (generated from persona/scenario) and all conversation messages. To get connector information, fetch the parent eval with ?expand=true.

Get Run​

GET /api/projects/:projectId/runs/:id

Response​

{
"id": "run-uuid",
"evalId": "eval-uuid",
"status": "completed",
"startedAt": "2026-01-29T10:00:00.000Z",
"completedAt": "2026-01-29T10:00:15.000Z",
"latencyMs": 1500,
"messages": [
{ "role": "user", "content": "Hello" },
{ "role": "assistant", "content": "Hi there!" }
],
"result": {
"success": true,
"score": 0.95,
"reason": "Agent responded appropriately"
},
"createdAt": "2026-01-29T10:00:00.000Z",
"updatedAt": "2026-01-29T10:00:15.000Z"
}

Error Response​

{
"error": "Run not found"
}

Status Code: 404

Create Run​

Creates one or more runs for an eval. If the eval's scenario has multiple personas associated with it (personaIds), one run is created for each persona.

POST /api/projects/:projectId/runs
Content-Type: application/json

Request Body​

{
"evalId": "eval-uuid"
}

Fields​

FieldTypeRequiredDescription
evalIdstringYesID of the eval to run

Note: Runs use the connector and LLM provider configured on the parent Eval. These cannot be overridden at the run level. The personaId and scenarioId are stored directly on the run at creation time. An executionId is automatically assigned to group runs created in the same batch.

Response​

Status Code: 201 Created

Returns an array of created runs. If the scenario has 3 personas, 3 runs are returned. All runs in the batch share the same executionId.

[
{
"id": "run-uuid-1",
"evalId": "eval-uuid",
"personaId": "persona-uuid-1",
"scenarioId": "scenario-uuid",
"executionId": 1,
"status": "queued",
"messages": [],
"createdAt": "2026-01-29T10:00:00.000Z",
"updatedAt": "2026-01-29T10:00:00.000Z"
},
{
"id": "run-uuid-2",
"evalId": "eval-uuid",
"personaId": "persona-uuid-2",
"scenarioId": "scenario-uuid",
"executionId": 1,
"status": "queued",
"messages": [],
"createdAt": "2026-01-29T10:00:00.000Z",
"updatedAt": "2026-01-29T10:00:00.000Z"
}
]

If the scenario has no personas, a single run is created with personaId set to null.

Error Responses​

StatusDescription
400Eval ID is required
404Eval, Scenario, or Persona not found

Create Playground Run​

Creates a run directly from a scenario without requiring an eval. Useful for testing scenarios in a playground environment before setting up formal evaluations.

POST /api/projects/:projectId/runs/playground
Content-Type: application/json

Request Body​

{
"scenarioId": "scenario-uuid",
"connectorId": "connector-uuid",
"personaId": "persona-uuid"
}

Fields​

FieldTypeRequiredDescription
scenarioIdstringYesID of the scenario to run
connectorIdstringYesID of the connector to use for invoking the agent
personaIdstringNoID of the persona to simulate (optional)

Note: Playground runs store connectorId directly on the run since there's no parent eval. LLM provider for evaluation is resolved from the project's llmSettings.

Response​

Status Code: 201 Created

{
"id": "run-uuid",
"scenarioId": "scenario-uuid",
"connectorId": "connector-uuid",
"personaId": "persona-uuid",
"status": "queued",
"messages": [],
"createdAt": "2026-02-03T10:00:00.000Z",
"updatedAt": "2026-02-03T10:00:00.000Z"
}

Error Responses​

StatusDescription
400Scenario ID is required / Connector ID is required
404Scenario, Connector, or Persona not found

Create Chat Run​

Creates a live chat run for a connector. Used by the Agents page for interactive chat sessions. Chat runs have status: "chat" and are not processed by RunProcessor.

POST /api/projects/:projectId/runs/chat
Content-Type: application/json

Request Body​

{
"connectorId": "connector-uuid"
}

Fields​

FieldTypeRequiredDescription
connectorIdstringYesID of the connector for the chat session

Response​

Status Code: 201 Created

{
"id": "run-uuid",
"connectorId": "connector-uuid",
"status": "chat",
"messages": [],
"startedAt": "2026-02-26T10:00:00.000Z",
"createdAt": "2026-02-26T10:00:00.000Z",
"updatedAt": "2026-02-26T10:00:00.000Z"
}

Error Responses​

StatusDescription
400Connector ID is required
404Connector not found

Send Chat Message​

Send a user message in a chat run. The server invokes the connector, appends the user message and assistant response to the run, and persists the updated state. This is the primary endpoint for the Agents page live chat.

POST /api/projects/:projectId/runs/:id/chat
Content-Type: application/json

Request Body​

{
"content": "Hello, I need help with my booking"
}

Fields​

FieldTypeRequiredDescription
contentstringYesThe user message text

Response​

Status Code: 200 OK

{
"run": {
"id": "run-uuid",
"connectorId": "connector-uuid",
"status": "chat",
"messages": [
{ "role": "user", "content": "Hello, I need help with my booking" },
{ "role": "assistant", "content": "I'd be happy to help! What's your booking number?" }
],
"threadId": "thread-uuid",
"startedAt": "2026-02-26T10:00:00.000Z",
"createdAt": "2026-02-26T10:00:00.000Z",
"updatedAt": "2026-02-26T10:01:00.000Z"
},
"messages": [
{ "role": "assistant", "content": "I'd be happy to help! What's your booking number?" }
],
"latencyMs": 523
}

Response Fields​

FieldTypeDescription
runobjectThe full updated run with all messages
messagesarrayOnly the new response messages from this turn
latencyMsnumberResponse time in milliseconds
errorstringError message if the connector invoke failed

Error Responses​

StatusDescription
400Message content is required / Run is not a chat run
404Run not found

Update Run​

PUT /api/projects/:projectId/runs/:id
Content-Type: application/json

Request Body​

{
"status": "completed",
"startedAt": "2026-01-29T10:00:00.000Z",
"completedAt": "2026-01-29T10:00:15.000Z",
"latencyMs": 1500,
"messages": [
{ "role": "user", "content": "Hello" },
{ "role": "assistant", "content": "Hi there!" }
],
"result": {
"success": true,
"score": 0.95,
"reason": "Agent responded appropriately"
}
}

Fields​

FieldTypeDescription
statusstring"queued", "pending", "running", "completed", "error", or "chat"
startedAtstringISO timestamp when run started
completedAtstringISO timestamp when run completed
latencyMsnumberTotal execution time in milliseconds
threadIdstringThread ID for LangGraph
messagesarrayConversation messages
outputobjectStructured output
resultobjectEvaluation result with success, score, reason
errorstringError message if run has system error

All fields are optional. Only provided fields will be updated.

Response​

{
"id": "run-uuid",
"evalId": "eval-uuid",
"status": "completed",
"...": "..."
}

Error Response​

StatusDescription
404Run not found

Delete Run​

DELETE /api/projects/:projectId/runs/:id

Response​

Status Code: 204 No Content

Error Response​

{
"error": "Run not found"
}

Status Code: 404

Retry Run​

Retry a run with system errors by resetting it to "queued" status. Only runs with status: "error" can be retried. Evaluation failures (status: "completed" with result.success: false) cannot be retried.

POST /api/projects/:projectId/runs/:id/retry

No request body required. The run is reset to "queued" status with cleared messages and a new thread ID.

Response​

{
"id": "run-uuid",
"evalId": "eval-uuid",
"status": "queued",
"messages": [],
"createdAt": "2026-01-29T10:00:00.000Z",
"updatedAt": "2026-01-29T10:05:00.000Z"
}

Error Responses​

StatusDescription
400Cannot retry run with status "X". Only runs with system errors can be retried.
404Run not found

Run Status​

Runs have the following status values:

StatusDescription
queuedRun created and waiting to be executed
pendingRun is being prepared for execution
runningRun is currently executing
completedRun finished (check result.success for pass/fail). Evaluation failures use this status
errorRun encountered a system error (retryable). Check error field for details
chatLive chat session from the Agents page. Not processed by RunProcessor