Asynchronous I/O in DuckDB: Work, Thread, Work

6 comments

• pjot 9 minutes ago
  I wonder how this would work in trying to parallelize the worker threads (multiple duckdb instances) coordinating them via Quack.

  Ducks all the way down!
• bburnett44 37 minutes ago
  Using 512gb of ram for a 22gb remote file does feel a bit weird for a benchmark but maybe they couldn’t get a large number of cores without lots of memory?
  [-]
  • otterley 20 minutes ago
    Most cloud providers start with a 2:1 ratio of memory in GiB to CPU cores and go up from there. Databases also are the most common use of large-memory systems because they benefit so much from large buffer caches.
• myshapeprotocol 2 hours ago
  Deep dive into asynchronous I/O architectures like this is pure engineering gold for high-performance data processing. Excellent breakdown.
• abofh 41 minutes ago
  Do they have SSL updates yet? Signing is great, but using https means not fighting firewalls to start a job
• datadrivenangel 52 minutes ago
  DuckDB is trending towards becoming a query engine, specifically the fastest analytical query engine. This is very good.
• mansi1010 2 hours ago
  This is such a long waited feature!