UI configuration
Set up an experiment in the Velvet app
Set up evaluations from the Velvet application or API. We recommend creating your first experiment from the application to get familiar with the functionality. Use the API to support more advanced use cases.
How it works
- Create and run a new experiment
- Review experiment results in the Velvet UI
(1) Create a new experiment
Navigate to the experiments tab inside your workspace to get started.
- Click the button for 'new experiment'
- Select the dataset (log(s)) you want to run tests against. Choose an existing log from your logs database, or enter a manual prompt.
- Select which model(s) you want to test.
- Define the metric(s) you want to test.
Supported metrics in the app
Note, metrics in the app are limited in beta. Configure any metric you want using the API.
Metric | Description |
---|---|
Latency | Define a threshold for how long an LLM response should take (in milliseconds) |
Cost | Define a threshold for the cost of an LLM response |
LLM Rubric | Ask an LLM to grade the quality of an LLM response on any qualitative criteria |
(2) Review experiment results
Navigate to the experiments tab after your experiment is configured.
Find the experiment you want to review and tap 'view detail'.
- Experiments can have 1+ variants of the test. Based on how many logs you used as inputs (test variables).
- Tests can have 1+ outputs. Based on how many models you selected in the experiment configuration.
- Outputs will have 1+ results. Based on the metrics you selected in the experiment configuration.
View output summary
Inspect details of an output
Watch a video overview
Email [email protected] with any questions.
Updated 6 days ago