API configuration

Set up experiments from the Velvet application or API. We recommend creating your first experiment from the application to get familiar with the functionality. Use the API to support more advanced use cases.

Watch a video overview

How it works

1. Define a dataset using a SQL query or CSV upload
2. Configure the experiment in JSON
3. Run the experiment with the API
4. View results in your database or in the Velvet UI

(1) Create your dataset

Create a dataset containing the input data you want to test. Use SQL queries to fetch data from your database, or upload a CSV file.

Here's an example SQL query to select 5 random system prompts, user prompts, and responses from your logs database.

SELECT 
  request->'body'->'messages'->0->>'content' AS system_prompt, 
  request->'body'->'messages'->1->>'content' AS user_prompt, 
  response->'body'->'choices'->0->'message'->>'content' AS response,
  (response->'body'->'choices'->0->'message'->>'content')::json->>'search_type' AS search_type_value
FROM 
  llm_logs
WHERE 
  response->'body'->'choices'->0->'message'->>'content' LIKE '%search_type%'
ORDER BY 
  RANDOM()
LIMIT 5;

Here's the same dataset, in CSV format (Google Sheet version here).

Once you've created and saved your dataset using either SQL or CSV, you can reference it in your experiment configuration.

(2) Configure your experiment with JSON

Provide a JSON configuration of the experiment. Note, you can also set up an experiment via the UI, but the API is best for more advanced or custom use cases.

Define prompts to test

Use double curly braces as placeholders for variables: {{variable_name}}

"prompts": [
  [
    {
      "role": "system",
      "content": "{{system_prompt}}"
    },
    {
      "role": "user",
      "content": "{{user_prompt}}"
    }
  ]
]

Define models to test

Specify the LLM providers you want to test and their model configurations.

Note, OpenAI supports the following model formats:

• openai:<model name> - uses a specific model name (mapped automatically to chat or completion endpoint)
• openai:embeddings:<model name> - uses any model name against the /v1/embeddings endpoint

"providers": [
  {
    "id": "openai:gpt-3.5-turbo-0125",
    "config": {
      "max_tokens": 1024,
      "temperature": 0.5
    }
  },
  {
    "id": "openai:gpt-4o-mini",
    "config": {
      "temperature": 0.5,
      "max_tokens": 1024,
      "top_p": 1,
      "frequency_penalty": 0,
      "presence_penalty": 0
    }
  }
]

Define inputs to evaluate

The var question below contains example inputs from a logs dataset. Each variant of the experiment will run on top of each input you include.

Set metrics, each will run automatically and output a score. Tests should contain core use cases and potential failures for your prompts.

 "tests":[
    {
			"assert":[
				  {
             "type":"latency",
             "threshold": 1500
          },
          {
             "type":"model-graded-closedqa",
             "value":"Provides a clear, concise answer"
          }
       ],
       "vars":{
          "question": [
						"What are large language models (LLMs)?",
						"Can you tell me a joke?",
						"How can I write my own compiler?",
						"What's the meaning of life?",
						"Can you write a query that joins two tables?",
						"Can you recommend a good book?",
						"What's your favorite food?",
						"What are your hobbies and how did you get into them?",
						"What's your go-to board game?",
						"What does your future hold?"
					]
       }
    }
 ]

Velvet also supports importing tests from a Google Sheet url. See the Google Sheet link below for an example.

"tests": "https://docs.google.com/spreadsheets/d/1KUZqbEZvOcglaNC4JGte8ICkKpp-1W0k07L7gi5j2sQ/edit?usp=sharing"

See Deterministic Evaluations and LLM Based Evaluations for a list of supported evaluation metrics. Velvet supports a variety of evaluation assertions out of the box.

Complete JSON example

Here's an example experiment configuration on JSON outputs that evaluates the quality of a prompt between gpt-3-turbo and gpt-4o-mini with logs and evaluation metrics imported from a Google Sheet csv.

{
	 "name": "Google Sheet Experiment",
	 "description": "Testing the performance of two OpenAI models on JSON outputs.",
   "prompts":[
		 [
				{
					"role": "system",
					"content": "{{system_prompt}"
				},
				{
					"role": "user",
					"content": "{{user_prompt}}"
				}
			]
   ],
   "providers":[
      {
         "id":"openai:gpt-3.5-turbo-0125",
				 "config":{
					 "response_format": { "type": "json_object" }
         }
      },
      {
         "id":"openai:gpt-4o-mini",
				"config":{
					 "response_format": { "type": "json_object" }
         }
      }
   ],
   "tests": "https://docs.google.com/spreadsheets/d/1KUZqbEZvOcglaNC4JGte8ICkKpp-1W0k07L7gi5j2sQ/edit?usp=sharing"
}

An experiment will run each prompt through a series of variable inputs and check if they meet the evaluation assertions.

(3) Run experiment

To run an experiment, make an API call to the below endpoint.

POST https://www.usevelvet.com/api/experiments

Required headers:

• Content-Type: application/json
• Authorization: Bearer [Your-Velvet-API-Token]
• velvet-user: [Your-User-ID]

Body:

• [Your JSON experiment configuration] (see example configuration above)

Complete request example

See the example curl request with the required headers to run a new experiment with the example config from above.

Fill in your and to try. Find these under the Experiment tab of your Velvet workspace ("API setup").

curl --request POST \
  --url https://www.usevelvet.com/api/experiments \
  --header 'Content-Type: application/json' \
  --header 'authorization: sk_velvet_prod_<api-key>' \
  --header 'velvet-user: user_<user-id>' \
  --data '{
	 "name": "Google Sheet Experiment",
	 "description": "Testing the performance of two OpenAI models on JSON outputs.",
   "prompts":[
		 [
				{
					"role": "system",
					"content": "{{system_prompt}}"
				},
				{
					"role": "user",
					"content": "{{user_prompt}}"
				}
			]
   ],
   "providers":[
      {
         "id":"openai:gpt-3.5-turbo-0125",
				 "config":{
					 "response_format": { "type": "json_object" }
         }
      },
      {
         "id":"openai:gpt-4o-mini",
				"config":{
					 "response_format": { "type": "json_object" }
         }
      }
   ],
   "tests": "https://docs.google.com/spreadsheets/d/1KUZqbEZvOcglaNC4JGte8ICkKpp-1W0k07L7gi5j2sQ/edit?usp=sharing"
}'

(4) View results of your experiment

Review results in your database

You can view the raw data of the experiment directly in your own database. This allows you to build your own analytics dashboards with SQL, and analyze the experiments in more detail.

Review results in the UI

You can also view the results in the ‘Experiments’ tab of your Velvet workspace. This makes it easier for non-technical stakeholders to analyze outcomes.

See a list of each experiment you’ve run, with a summary of configuration and pass rate.

Tap into an experiment to inspect the complete variants and results on each input.

Head over to our configuration reference docs for additional evaluation examples the API supports.

Email [email protected] with any questions.