Skip to content
Hallway
0:00 / 0:00

#raceconditions#filesystem#c++

CppCon 2015: Niall Douglas “Racing The File System"

CppCon

189K subscribers • 101 videos on Hallway • Conference channel

YouTube
  • Recommended by 2 people: 1 in engineering communities and 1 on Stack Overflow and its sister sites.
  • A classic. People kept recommending it from 2016 to 2024.
  • Presented by Niall Douglas
  • Linked in the descriptioncppcon/cppcon2015

Advanced. Deep dive for people who know the field.

9.5K views11 years ago2 recommendations

Systems programmingSecurityConcurrency and parallelismOperating systemsPerformance and optimization

Niall Douglas explains the dangers of file system race conditions and how to write code that safely handles concurrent path changes.

Cppcon.org — Presentation Slides, PDFs, Source Code and other presenter materials are available at: github.com/cppcon/cppcon2015 — Almost every programmer knows about and fears race conditions on memory where one strand of execution may concurrently update data in use by another strand of execution, leading to an inconsistent and usually dangerous inconsistent read of program state. Almost every programmer therefore is aware of mutexes, memory ordering, semaphores and the other techniques used to serialise access to memory. Interestingly, most programmers are but vaguely aware of potential race conditions on the filing system, and as a result write code which assumes that the filing system does not suddenly change out from underneath you when you are working on it. This assumption of a static filing system introduces many potential security bugs never mind ways of crashing your program, and of course creating data loss and corruption. This workshop will cover some of the ways in which filing system races can confound, and what portable idioms and patterns you should employ to prevent misoperation, even across networked Samba shares. Finally, an introduction of the proposed Boost library AFIO will be made which can help application developers writing filing system race free code portably. — Niall Douglas is a lead author of proposed Boost.AFIO, Boost.APIBind and Boost.Outcome. He is currently the lead Google Summer of Code administrator for Boost. He is an Affiliate Researcher with the Waterloo Research Institute for Complexity and Innovation at the University of Waterloo, Canada, and holds postgraduate qualifications in Business Information Systems and Educational and Social Research as well as a second undergraduate degree double majoring in Economics and Management. He has been using Boost since 2002 and was the ISO SC22 (Programming Languages) mirror convener for the Republic of Ireland 2011-2012. He formerly worked for BlackBerry 2012-2013 in their Platform Development group, and was formerly the Chief Software Architect of the Fuel and Hydraulic Test Benches of the EuroFighter defense aircraft. He is a published author in the field of Economics and Power Relations, is the Social Media Coordinator for the World Economics Association and his particular interest lies in productivity, the causes of productivity and the organizational scaling constraints which inhibit productivity. He is presently between contracts as an expert C++ consultant. — Videos Filmed & Edited by Bash Films: BashFilms.com Work at Hudson River Trading (HRT): tinyurl.com/safxfctf

Why it's here

2 people recommend this

What people say. Recommended for its insights into file system concurrency issues and how to handle them safely.

Moments people linked to

u/TribladeSlicein r/C_Programming • Recommends

Examples of why we use the *at family of functions? I'm making an attempt at writing a portable file system library in C, and after watching this talk, it opened my eyes to several issues that I am going to need to think about when dealing with the file system. One of them is issues with concurrent path changes (shown in the talk at 6:28). The solution he suggests is the use of the \`\*at\` family of functions like openat, mkdirat, etc, where we use a handle to a directory as a base to access files relative to it. I myself can't think of any scenario where I would need to use the \`\*at\` family, certainly due to me just not entirely understanding the problem. I'm sure there is a re…
5View on Reddit

EmDroidon Stack Overflow • Recommends

On “removing directories and all sub-directories in it”

…moving each sub-file and sub-folder to some temporary location (on the same partition!), and then deleting it. That way the parent folder will be empty and deletable, even though the contents has not been "physically" deleted yet (because everything is already moved outside - the move is synchronous). You can see this talk for more info on that: link to this video In general, handling the files……
5View the answer

u/skeetoin r/C_Programming • Mentions

On “Black Magic Wizard Bullshit”

This "pattern" is both unnecessary and a race condition: // do not do this DeleteFile(path); CreateFile(path, ..., CREATE_ALWAYS, ...); `CREATE_ALWAYS` truncates, so deleting the file first accomplishes nothing even in the best case. Worse, `DeleteFile` merely [schedules the file for deletion]. When it returns the file may still exist *in its original path*, untouched until sometime after the last handle is closed, which might result from, say, a virus scanner). Deleting files properly on Windows is tricky business, but fortunately you don't need it in the typical case, including here where you're updating a file in place. You're checking for errors, and along with the exclusi…
3View on Reddit

u/skeetoin r/C_Programming • Mentions

On “Examples of why we use the *at family of functions?”

That's a really great talk ([CppCon 2015: Niall Douglas “Racing The File System"]). I hadn't seen that one before. Thanks for sharing it! As for your question, perhaps look at how existing programs use it. Run your favorite programs under `strace` and look for the system calls. For example: $ strace -e openat find ~ >/dev/null Then dig into the source. (Though in come cases libc is doing it automatically in the background.)
2View on Reddit

u/yamaxanduin r/C_Programming • Mentions

On “Help w/ Programmatic Recursive File/Folder Moving on Linux”

rename should work with directories? but the \*at filesystem functions are easier to use (as long as you don't have to pass files to other programs...) because they don't require string concatenation (and identifying directories with file descriptors prevent some race conditions). renameat2 can also guarantee that you don't overwrite existing files in the trashes directory. e.g. running `rm a/file` and `rm b/file` (the alias program) could otherwise result in a toctou bug in between the file-exists-check and move-to-trash operations. #define _GNU_SOURCE #include <stdio.h> #include <fcntl.h> int move_to_trash(int dfd, cont char *path) { int trash_fd = open("/…
1View on Reddit