Researchers Introduce RTL-Verify: A New Benchmark for Evaluating AI Agents on Hardware Design Tasks

A team of researchers has introduced RTL-Verify, a benchmark designed to evaluate the performance of AI agents on hardware design and verification tasks at the Register-Transfer Level (RTL). As AI coding assistants become more capable, there is growing interest in extending their use beyond software to hardware description languages like Verilog and VHDL. RTL-Verify addresses the lack of standardized evaluation in this domain, providing a diverse set of tasks that mirror real-world RTL design workflows.

The benchmark includes tasks such as writing RTL code from specifications, debugging existing designs, and verifying properties using formal methods. The dataset covers a range of complexities, from simple combinational logic to complex finite state machines and pipeline controllers. By testing agents on these tasks, the researchers aim to identify strengths and weaknesses of current models in handling hardware-specific challenges, such as concurrent execution, timing constraints, and low-level bit manipulations.

Early results using state-of-the-art LLMs show that while AI agents can handle straightforward RTL tasks, they struggle with nuanced verification scenarios and large design spaces. The authors note that this gap highlights the need for specialized training data and techniques for hardware design. RTL-Verify is publicly available, inviting the research community to benchmark new models and approaches, potentially accelerating the adoption of AI in hardware development.