Skip to content
Hallway
0:00 / 0:00

#fsync#fileio#posix

Eat My Data: How Everybody gets File IO Wrong

Stewart Smith

45 subscribers • 1 video on Hallway

YouTube
  • Recommended by 2 people: 2 in engineering communities.
  • A classic. People kept recommending it from 2019 to 2024.
  • Presented by Stewart Smith

Intermediate. Assumes working knowledge.

3.3K views9 years ago2 recommendations

Systems programmingDatabases and dataOperating systemsDatabase internals

Stewart Smith explains the complexities of POSIX file IO, why common patterns like write-rename are unsafe, and how to safely persist data.

My linux.conf.au 2007 talk, originally up at linux.org.au/conf/2007/talk/278.html - The talk that led to flamingspork.com/projects/libeatmydata/ Abstract: Too many applications don't deal with important files in a safe way. Along with that, they don't check error codes properly or do things like blindly rely on fsync() actually sending data to disk (which becomes a problem if you intend your application to be portable). Write to temp file, close, rename is not a safe operation. We'll discuss why - along with the various behaviours of different file systems and platforms and all the different ways you can expect your important data to get mangled. More importantly, we'll talk about how to write code using the POSIX file IO api and get it right - keeping your application portable, fast and most importantly - respecting the data that users trust in it. We'll discuss: - when data hits disk - what causes data to hit disk - how you can ensure that data is on disk - how applications get this wrong - how a file system is not a database (and vice versa)

Why it's here

2 people recommend this

What people say. Engineers recommend this talk for understanding the pitfalls of file persistence, fsync, and how to avoid data corruption during power cuts.

u/lordlodin r/C_Programming • Recommends

On “Potential Errors For File I/O”

There is a lot that could go wrong, it is one of those crazy complicated areas that gets worse the deeper you go. I saw this talk live several years ago. In 40 minutes he covers safely saving a file in Linux and MacOS, Windows wouldn't fit in the time. Note that there are several corrections from Linus and Tridge in the audience because even after preparing the talk there were complications he had missed. link to this video
1View on Reddit

u/-electric-skillet-in r/embedded • Recommends

On “Using Sqlite on an embedded software as key value store”

Problem is that it's not easy to reliably do this kind of persistence while covering all the corner cases. What if power is cut in the middle of updating the file and you end up with partial garbage? Or you forget to sync the directory after a rename? And a dozen more possibilities. Using sqlite or an equivalent gives you a very robust solution that you basically don't have to think about. Related: "Eat My Data: How Everybody gets File IO Wrong" link to this video
1View on Reddit