Use the Batch and Files APIs

With OpenAI’s batch API, you can process large volumes of data asynchronously. As a result, you’ll get lower costs, higher rate limits, and a guaranteed 24-hour response time. See the OpenAI Batch API docs.

Velvet's proxy gives you extra data to observe and analyze the batch process.

Example use cases:

• Text generation: Write review summaries for every product page
• Classification: Add categories to a large dataset of cancelation requests
• Embeddings: Add a vector number to each article to indicate relatedness

Observe and analyze

Velvet's proxy supports logging of the Batch and Files APIs. When you send or recieve the .jsonl file, we warehouse each individual dataset in your file.

Query the batch start, completion, and status. Velvet unlocks extra data so you can analyze what happens to each file inside the batch.

Batch API metadata example

Batching unlocks additional metadata stored with each warehoused log. Refer to these example when querying your batch requests.

batch_example.jsonl (with two completion requests)

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo-0125", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello velvet!"}],"max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo-0125", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}

Completion request 1 metadata

{
  "cost": {
    "input_cost": 0.000005,
    "total_cost": 0.000012,
    "output_cost": 0.000007,
    "input_cost_cents": 0.0005,
    "total_cost_cents": 0.0012,
    "output_cost_cents": 0.0007
  },
  "batch": {
    "id": "batch_req_vfgFkPOcs5XISaULjJRawiJC",
    "custom_id": "request-1",
    "input_file": {
      "id": "file-TbeJuO1LUW0Kk3jDB3nPkuWp",
      "bytes": 517,
      "log_id": "log_mtxycg38r2k1e160",
      "object": "file",
      "status": "processed",
      "purpose": "batch",
      "filename": "./batch_example.jsonl",
      "created_at": 1724346043,
      "status_details": null
    },
    "output_file": {
      "id": "file-ZLOUc7ny6ovSDojNP78nmp6d",
      "log_id": "log_56qvrmmxnf04e9ay"
    }
  },
  "error": null,
  "model": "gpt-3.5-turbo-0125",
  "usage": {
    "model": "gpt-3.5-turbo-0125",
    "total_tokens": 29,
    "calculated_by": "openai",
    "prompt_tokens": 20,
    "completion_tokens": 9
  },
  "provider": "openai",
  "expected_cost": {
    "input_cost": 0.000005,
    "total_cost": 0.000012,
    "output_cost": 0.000007,
    "input_cost_cents": 0.0005,
    "total_cost_cents": 0.0012,
    "output_cost_cents": 0.0007
  },
  "expected_usage": {
    "model": "gpt-3.5-turbo-0125",
    "total_tokens": 29,
    "calculated_by": "openai",
    "prompt_tokens": 20,
    "completion_tokens": 9
  }
}

Completion request 2 metadata

{
  "cost": {
    "input_cost": 0.000006,
    "total_cost": 0.000018,
    "output_cost": 0.000013,
    "input_cost_cents": 0.0006,
    "total_cost_cents": 0.0018,
    "output_cost_cents": 0.0013
  },
  "batch": {
    "id": "batch_req_poqf5DCmLNjuRH8UeS4Ukn8A",
    "custom_id": "request-2",
    "input_file": {
      "id": "file-TbeJuO1LUW0Kk3jDB3nPkuWp",
      "bytes": 517,
      "log_id": "log_mtxycg38r2k1e160",
      "object": "file",
      "status": "processed",
      "purpose": "batch",
      "filename": "./batch_example.jsonl",
      "created_at": 1724346043,
      "status_details": null
    },
    "output_file": {
      "id": "file-ZLOUc7ny6ovSDojNP78nmp6d",
      "log_id": "log_56qvrmmxnf04e9ay"
    }
  },
  "error": null,
  "model": "gpt-3.5-turbo-0125",
  "usage": {
    "model": "gpt-3.5-turbo-0125",
    "total_tokens": 39,
    "calculated_by": "openai",
    "prompt_tokens": 22,
    "completion_tokens": 17
  },
  "provider": "openai",
  "expected_cost": {
    "input_cost": 0.000006,
    "total_cost": 0.000018,
    "output_cost": 0.000013,
    "input_cost_cents": 0.0006,
    "total_cost_cents": 0.0018,
    "output_cost_cents": 0.0013
  },
  "expected_usage": {
    "model": "gpt-3.5-turbo-0125",
    "total_tokens": 39,
    "calculated_by": "openai",
    "prompt_tokens": 22,
    "completion_tokens": 17
  }
}