Apache DataFusion

Apache DataFusion is an open-source, extensible analytical query engine written in Rust, built on Apache Arrow's columnar memory format.[1][2] I

Apache DataFusion
Apache DataFusion
DeveloperApache Software Foundation
Written inRust
TypeQuery engine
LicenseApache License
Websitedatafusion.apache.org
Repository

Apache DataFusion is an open-source, extensible analytical query engine written in Rust, built on Apache Arrow's columnar memory format.[1][2] It provides SQL and DataFrame interfaces for analytical query execution and is designed to be used as a library by developers building databases, query engines, and analytical tools, rather than as a standalone database server.[1][2] The project originated in 2017, was donated to the Apache Arrow project in 2019, and became a top-level project of the Apache Software Foundation in 2024.[3][4] As of March 2026, DataFusion exceeded one million monthly downloads on crates.io.[5]

History

DataFusion originally authored by Andy Grove starting in 2017. It was donated to the Apache Arrow Project in February 2019.[3] In 2024, a paper describing DataFusion was accepted to the industry track of the ACM SIGMOD conference.[6][1] In April 2024, the project graduated from Apache Arrow and became a top-level Apache project.[4]

Features

DataFusion is a fast, extensible query engine for building data systems. It provides a SQL interface and a DataFrame API for constructing queries programmatically, a query planner and rule-based optimizer, and a multithreaded vectorized execution engine that processes data in columnar batches rather than row by row.[1][2]

The engine reads common analytical file formats natively, including Apache Parquet, CSV, JSON, Avro, and Arrow IPC, and uses Apache Arrow's columnar memory format throughout execution, avoiding serialization overhead between stages.[1]

DataFusion is designed for in-process embedding: it runs within the host application's process rather than as a separate server, using threads for parallel query execution. Its extension points allow downstream systems to add user-defined functions, custom data sources, custom query languages, and new optimizer rules, enabling developers to build specialized database systems on top of DataFusion's planning and execution components without reimplementing them.[1][2]

DataFusion is frequently compared with other columnar analytical systems including DuckDB, Polars, and Velox, but these systems differ significantly in scope and intended use.[7]

Adoption and reception

DataFusion has been adopted across a range of analytics and database products. Cloudflare used DataFusion in its Log Explorer product to execute SQL queries over log data stored in Cloudflare R2.[8] Palantir Lightweight Pipelines are powered by DataFusion.[9][10] InfluxDB 3.0 uses DataFusion as part of the FDAP stack: Apache Flight, DataFusion, Arrow, and Parquet.[11] Other users described in public sources include EDB Postgres AI,[12] Cube,[13] Spice AI,[14] Pydantic Logfire,[15] and Kamu.[16]

In 2024, CRN included Apache DataFusion in its list of "The 10 Coolest Open-Source Software Tools Of 2024".[17]

Language support

DataFusion itself is written in Rust. The project also has official Python bindings and community-maintained bindings and tooling for other languages and runtimes.[18][19]

Ecosystem projects

Several projects in the broader Apache ecosystem and the community-maintained datafusion-contrib organization extend DataFusion's capabilities.[19]

References

  1. ^ a b c d e f Lamb, Andrew; Shen, Yijie; Heres, Daniel; Chakraborty, Jayjeet; Kabak, Mehmet Ozan; Hsieh, Liang-Chi; Sun, Chao (2024). "Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine". Companion of the 2024 International Conference on Management of Data. pp. 5–17. doi:10.1145/3626246.3653368. ISBN 979-8-4007-0422-2.
  2. ^ a b c d "Introduction". Apache DataFusion. Apache Software Foundation. Retrieved 2026-03-22.
  3. ^ a b "DataFusion: A Rust-native Query Engine for Apache Arrow". Apache DataFusion Blog. Apache Software Foundation. 2019-02-04. Retrieved 2026-03-22.
  4. ^ a b "Apache Software Foundation Announces New Top-Level Project Apache DataFusion". The ASF Blog. Apache Software Foundation. 2024-06-11. Retrieved 2026-03-22.
  5. ^ "datafusion". crates.io. Retrieved 2026-03-26.
  6. ^ "SIGMOD 2024 Industrial Track: Accepted Papers". SIGMOD 2024. Retrieved 2026-03-22.
  7. ^ Pedreira, Pedro; Erling, Orri; Mühleisen, Hannes; Muñoz, Ruben; Khaled, Wael; Dürsch, Peter (2023). "The Composable Data Management System Manifesto". Proceedings of the VLDB Endowment. 16 (10): 2679–2685. doi:10.14778/3603581.3603604.
  8. ^ "Cloudflare Log Explorer is now GA, providing native observability and forensics". The Cloudflare Blog. Cloudflare. 2025-06-18. Retrieved 2026-03-22.
  9. ^ "Announcements: July 2025". Palantir Foundry Documentation. Palantir Technologies. 2025-07-29. Retrieved 2026-03-22.
  10. ^ "Announcements: February 2024". Palantir Foundry Documentation. Palantir Technologies. February 2024. Retrieved 2026-03-22.
  11. ^ "Flight, DataFusion, Arrow, and Parquet: Using the FDAP Architecture to build InfluxDB 3.0". InfluxData. 2023-10-25. Retrieved 2026-03-22.
  12. ^ "Enterprise DB begins rolling AI features into PostgreSQL". SiliconANGLE. 2024-05-23. Retrieved 2026-03-22.
  13. ^ "Query pushdown in Cube's semantic layer". Cube. 2024-06-03. Retrieved 2026-03-22.
  14. ^ "How we use Apache DataFusion at Spice AI". Spice AI. 2026-01-17. Retrieved 2026-03-22.
  15. ^ "We're changing database". GitHub. 2024-08-29. Retrieved 2026-03-22.
  16. ^ "100X faster ingestion, and FlightSQL support for connecting BI tools". Kamu Data. 2023-09-26. Retrieved 2026-03-22.
  17. ^ "The 10 Coolest Open-Source Software Tools Of 2024". CRN. 2024-11-21. Retrieved 2026-03-22.
  18. ^ a b "datafusion-contrib". GitHub. Retrieved 2026-03-22.

Content Disclaimer

Informasi ini disarikan dari Wikipedia dan disajikan kembali untuk tujuan edukasi. Konten tersedia di bawah lisensi CC BY-SA 3.0. Kami tidak bertanggung jawab atas ketidakakuratan data yang bersumber dari kontribusi publik tersebut.

  1. The information displayed on this website is sourced in part or in whole from Wikipedia and has been adapted for the purpose of restating it. We strive to provide accurate and relevant information, however:
  2. There is no guarantee of absolute accuracy. Wikipedia is an open, collaborative project that can be edited by anyone, so information is subject to change.
  3. It is not intended to constitute professional advice. The content displayed is for informational and educational purposes only. For important decisions (e.g., medical, legal, or financial), please consult a professional.
  4. Content copyright. Wikipedia is licensed under the Creative Commons Attribution-ShareAlike License (CC BY-SA). This means that content may be reused with appropriate attribution and shared under a similar license.
  5. Responsible use. Any risk arising from the use of information from this website is entirely the responsibility of the user.