UI configuration

Set up an experiment in the Velvet app

Set up evaluations from the Velvet application or API. We recommend creating your first experiment from the application to get familiar with the functionality. Use the API to support more advanced use cases.


How it works

  1. Create and run a new experiment
  2. Review experiment results in the Velvet UI

(1) Create a new experiment

Navigate to the experiments tab inside your workspace to get started.

  1. Click the button for 'new experiment'
  2. Select the dataset (log(s)) you want to run tests against. Choose an existing log from your logs database, or enter a manual prompt.
  3. Select which model(s) you want to test.
  4. Define the metric(s) you want to test.


Supported metrics in the app

Note, metrics in the app are limited in beta. Configure any metric you want using the API.

MetricDescription
LatencyDefine a threshold for how long an LLM response should take (in milliseconds)
CostDefine a threshold for the cost of an LLM response
LLM RubricAsk an LLM to grade the quality of an LLM response on any qualitative criteria

(2) Review experiment results

Navigate to the experiments tab after your experiment is configured.

Find the experiment you want to review and tap 'view detail'.

  • Experiments can have 1+ variants of the test. Based on how many logs you used as inputs (test variables).
  • Tests can have 1+ outputs. Based on how many models you selected in the experiment configuration.
  • Outputs will have 1+ results. Based on the metrics you selected in the experiment configuration.

View output summary


Inspect details of an output


Watch a video overview


Email [email protected] with any questions.