Artificial Analysis Launches Optima for Custom AI Model Benchmarking

Standardized benchmarks like MMLU or HumanEval provide a useful baseline for general model capability, but they often fail to reflect the specific nuances of enterprise workflows. Artificial Analysis has addressed this gap with the launch of Optima, a tool designed to let teams build custom benchmarks around their own tasks. The platform enables users to compare models not just on raw accuracy, but on performance, cost, and time efficiency for specific use cases.

Moving Beyond Generic Metrics

The core value proposition of Optima is specificity. Instead of relying on generic datasets, users can describe their work, attach examples, and collaborate with a build agent to draft tasks and rubrics. For instance, a legal tech team could create a benchmark specifically for reviewing supplier contracts, flagging risks, and extracting terms.

The platform supports two primary evaluation methods:

• Rubric-based grading: A judge model checks each response against specific criteria, ensuring deterministic scoring (right or wrong, criterion by criterion).
• Head-to-head ranking: Using a panel of judges from major evaluations to rank models directly.

Cost and Time Efficiency

A key differentiator for Optima is its focus on operational efficiency. The tool claims to help users cut cost and time by over 10x by providing transparent comparisons of cost and time per task. This is particularly relevant for developers integrating LLMs into production systems, where the total cost of ownership often depends more on latency and token pricing than on marginal differences in accuracy.

Bring Your Own Agent

Optima also supports interoperability. Users can bring their own agents to compete in the same benchmark run over HTTP, allowing for a direct comparison between off-the-shelf models and custom fine-tuned or RAG-augmented pipelines.

Pricing Model

The pricing structure is usage-based:

• Benchmark creation and running: Priced on token usage. Artificial Analysis holds a credit based on a cost estimate, but users are only charged for the tokens actually used.
• Grading: Priced per unit judged. The platform holds a quoted total for the entire pass, ensuring the cost never exceeds the quote, and potentially costing less if fewer units are judged than planned.

For teams struggling to justify model selection based on generic leaderboards, Optima offers a path to data-driven decisions grounded in actual business tasks.