Database Write Vs. Disk I/O for C/C++

Database Write Vs. Disk I/O for C/C++

So I am in the process of generating a workflow for a new project. In the first phase there will be a C/C++ code that generates large amounts of data (~1TB based on binary file format for test cases). This data will then be imported into a Database for comparison with another dataset which will be resident there.

The question I have is would it be faster to open a port to the database software and write directly to there or disk write into many small files (~1 billion) and import into the database later? The C Code will be running on a time limited cluster so it needs to finish quickly.

4

3 Answers

Although the correct answer should be "it depends, you need to measure", this is a case where an unqualified answer can be given with acceptable certainity:

Talking to the database server directly will almost certainly be faster.

The reason is not only that writing to disk first and then reading the data in again prior to sending it over the network involves additional latency and bandwidth constraints from the disk drive (and this is unavoidable, as every file takes up at least one page in the page cache, a billion files -- even with only 1 byte contents each -- takes at least 4TiB worth of buffers, so there will be no caching). Reading 4TiB of data takes 3-5 hours, assuming sequential reads on a fast disk. While writing to disk may actually happen mostly sequentially thanks to lazy writeback, reading the data in again prior to sending it is almost guaranteed not to be contiguous (and there is no such thing as readahead happening, not possible for distinct files). Some operations may still be overlapped if you're lucky, hiding their latency -- but all in all this won't be "for free".

While it is true that you may be limited by the database's ability to accept your requests or by network bandwidth, this is something you are always being limited by, even if you do not add an extra copy, and at least the former is something you can optimize rather easily. You can insert data into the database without any indices, which will be very fast. The database server can then create any incides you may want later. That may of course take some time, but it is likely an order of magnitude faster than updating indices with every update (and besides, who cares, if what's important is that your limited-time job finishes fast).

More importantly, however, merely opening and closing, and re-opening a billion files takes a very noticeable amount of time (this involves traversing directory hierarchies, name to inode translation, and access checks among other things), and the amount of time spend on accessing the individual files will be sheer insane. Even on solid state disks where no physical "seeking" takes place, random access times are not "zero" (they're much smaller, but 0.1ms times one billion is still almost 3 hours extra spend on seeking!).

You didn't specify what database you're running so in theory the answer could be anything. However, in practice modern hard disk drives are slow: they write at about 100 MB/s. Typically, this is the limiting factor if you're inserting massive amounts of data to the database. The RAM in the computer won't help as 1TB won't fit into the RAM. However, you may need to use database-specific tricks such as adding all the data in a single transaction and perhaps pre-processed queries that are compiled once and called multiple times. If you're using solid-state disks (SSDs), the answer may be different but that depends on the speed of the SSD.

Do note also that the network interface may limit the addition performance. Gigabit link means in practice a little less than 125 MB/s. So, if your database is on another computer it may be the case that the network interface performance is the limiting factor. However, if you are generating the data on the same computer or using a ten gigabit link, then it's unlikely that the network link performance would be a limiting factor.

The only definite answer, however, is to test it in YOUR environment. If performance is important to you, do learn the art of benchmarking.

Does the table have indexes? If so, then inserting the data first and building the index only after all data is inserted may improve performance.

You HAVE to test this to know.

Different database products go at different speeds, we don't know database's hardware, and if there's a lot of indices on REALLY big data, it could make everything slow.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Sophia Al-Mansoor
Author

Sophia Al-Mansoor

Sophia analyzes international trade, startup ecosystems, retail transformation, and supply chain logistics for modern digital publications.