Draft:DABstep Benchmarks

You can also browse Wikipedia:Featured articles and Wikipedia:Good articles to find examples of Wikipedia's best writing on topics similar to your proposed arti

Draft:DABstep Benchmarks


DABstep (Data Agent Benchmark for Multi-step Reasoning) is a benchmark for evaluating artificial intelligence agents on complex, multi-step data analysis tasks. It was developed jointly by Adyen and Hugging Face and first published in 2025.[1]

Background

Existing benchmarks for data analysis AI evaluated models on isolated code generation tasks or synthetic question-answering problems.[1] DABstep was designed to address limitations in these earlier evaluations by grounding tasks in actual operational workloads from a financial analytics environment.[1]

Design and structure

DABstep comprises over 450 real-world challenges derived from a financial analytics platform, requiring models to combine code-based data processing with contextual reasoning over heterogeneous documentation.[1] Tasks are derived directly from operational workloads at Adyen and reflect the complex, iterative problem-solving scenarios faced by professional data analysts.[1]

The benchmark integrates both structured data, including CSV and JSON files representing transaction telemetry and business metadata, and unstructured documentation such as Markdown files defining domain-specific formulas and rules.[1] Tasks require technical proficiency in data manipulation, including filtering, aggregation, and joins, as well as the ability to extract and apply domain-specific rules from documentation.[1]

Tasks are divided into two difficulty splits. The easy split covers tasks resolvable by reasoning over a single file or data source. The hard split requires multi-step reasoning across multiple files and source documents simultaneously and accounts for the majority of the benchmark's tasks.[1]

DABstep uses factoid-style evaluation in which each task output maps to a binary outcome — correct or incorrect — enabling objective scoring at scale without requiring human interpretation.[1] Unlike benchmarks such as SWE-bench or MLE-bench, DABstep is designed for low-barrier usage; generating answers requires only access to a code execution environment, and participants can submit answers directly to a leaderboard for automatic evaluation.[1]

Results

Results from the benchmark's original evaluation revealed a substantial performance gap between AI agent capability and real-world data analysis requirements.[1] Even the best agent at the time of publication achieved only 14.55% accuracy on the hard tasks, while performance on the easy split was considerably higher, with the top model reaching 76.39% accuracy.[1]

Subsequent systems improved substantially on the original baseline results. DS-STAR, a role-decomposed data science agent developed at Google, raised accuracy on DABstep to 45.2% and secured the top rank on the public leaderboard as of September 2025.[2]

Publication and availability

The benchmark paper was submitted to the NeurIPS 2025 Datasets and Benchmarks Track.[3] The benchmark is released with a public leaderboard on Hugging Face Spaces and an open dataset at huggingface.co/datasets/adyen/dabstep.[1]

References

  1. ^ a b c d e f g h i j k l m Egg, Alex; Iglesias Goyanes, Martin; Kingma, Friso; Mora, Andreu; von Werra, Leandro; Wolf, Thomas (2025). "DABstep: Data Agent Benchmark for Multi-step Reasoning". arXiv:2506.23719 [cs.LG].
  2. ^ "DS-STAR: A state-of-the-art versatile data science agent". Google Research. 2025.
  3. ^ "DABstep: Data Agent Benchmark for Multi-step Reasoning". OpenReview. April 2025.
  • Egg, Alex; Iglesias, Martin; Kingma, Friso; Mora, Andreu; Von Werra, Leandro; Wolf, Thomas (2025). "DABstep: Data Agent Benchmark for Multi-step Reasoning". arXiv:2506.23719. Submitted to NeurIPS 2025 Datasets and Benchmarks Track.
  • Adyen Tech (February 2025). "Data Agent Benchmark for Multi-step Reasoning (DABstep)". Medium / Adyen Tech Blog.
  • Hugging Face (February 2025). "DABstep: Data Agent Benchmark for Multi-step Reasoning". Hugging Face Blog.
  • Google Research (2025). "DS-STAR: A state-of-the-art versatile data science agent". Google Research Blog.
  • DABstep Leaderboard. Hugging Face Spaces. huggingface.co/spaces/adyen/DABstep.
  • DABstep Dataset Card. Hugging Face Datasets. huggingface.co/datasets/adyen/dabstep.


Content Disclaimer

Informasi ini disarikan dari Wikipedia dan disajikan kembali untuk tujuan edukasi. Konten tersedia di bawah lisensi CC BY-SA 3.0. Kami tidak bertanggung jawab atas ketidakakuratan data yang bersumber dari kontribusi publik tersebut.

  1. The information displayed on this website is sourced in part or in whole from Wikipedia and has been adapted for the purpose of restating it. We strive to provide accurate and relevant information, however:
  2. There is no guarantee of absolute accuracy. Wikipedia is an open, collaborative project that can be edited by anyone, so information is subject to change.
  3. It is not intended to constitute professional advice. The content displayed is for informational and educational purposes only. For important decisions (e.g., medical, legal, or financial), please consult a professional.
  4. Content copyright. Wikipedia is licensed under the Creative Commons Attribution-ShareAlike License (CC BY-SA). This means that content may be reused with appropriate attribution and shared under a similar license.
  5. Responsible use. Any risk arising from the use of information from this website is entirely the responsibility of the user.