0:00 / 0:00
"NTFS really isn't that bad" - Robert Collins (LCA 2020)
linux.conf.au
32K subscribers • 14 videos on Hallway
- Recommended by 6 people: 5 on Hacker News and 1 on Lobsters.
- Presented by Robert Collins, software engineer
- An opinion. It argues one side of a question engineers disagree on.
Intermediate. Assumes working knowledge.
160K views6 years ago6 recommendations
Systems programmingTools and practiceOperating systemsPerformance and optimizationRust
Robert Collins explains how antivirus hooks on file close operations made rustup slow on Windows, and how optimizing file access patterns resolved it.
Robert Collins lca2020.linux.org.au/schedule/presentation/38/ Why was rustup slow (3m30s to install (not including download time)) in early 2019 on Windows, and why isn't it slow (14s to install) now? Early in 2019 I was developing some things in Rustlang on Windows, got frustrated at the performance of rustup (the installer for rust) and looked for bug. I found one which blamed a combination of NTFS, small files and anti-virus software for the problem. This wasn't an entirely satisfactory answer to my mind. Fast forward a bit, and with a combination of changes to rustup and to Windows itself and rustup is now pleasantly usable.... which also improved performance for rustup on NFS mounted home directories. I'd like to share with you the story of this experience as well as the technical constraints that drove both the poor performance and the solution we put in place, which has also helped rust-doc performance. linux.conf.au is a conference about the Linux operating system, and all aspects of the thriving ecosystem of Free and Open Source Software that has grown up around it. Run since 1999, in a different Australian or New Zealand city each year, by a team of local volunteers, LCA invites more than 500 people to learn from the people who shape the future of Open Source. For more information on the conference see linux.conf.au/ Produced by NDV: youtube.com/channel/UCQ7dFBzZGlBvtU2hCecsBBg?sub_confirmatio… #linux.conf.au #linux #foss #opensource Thu Jan 16 2020 at Room 5Why it's here
6 people recommend thisWhat people say. Engineers recommend this talk to explain Windows filesystem performance characteristics, particularly the overhead of antivirus hooks on file operations.
jmmvon Hacker News • Recommends
On “When to use Bazel?”
Add your npm directories to the virus file exclusion lists of Windows Defender. Note that some of the directories may not be obvious (those under AppData for example). The speed improvements are massive for tools like these that touch lots of small files. Also take a look at this talk on NTFS, file system speed, and how the presenter optimized the Rust installer: link to this video . There is an awful lot that can be done to tools that deal with lots of files to perform well under Windows too.
ablobon Hacker News • Recommends
On “Show HN: Dut – a fast Linux disk usage calculator”
I want to take this opportunity to recommend the talk "NTFS isn't that bad" ( link to this video ). NTFS prefers a different access pattern than most usual file systems. I remember that a part of the talk was about speed-ups on Linux as well. So even if it doesn't sway your opinion it should enhance your perspective on how file systems work.
unchar1on Hacker News • Recommends
On “SQLite: 35% Faster Than the Filesystem”
That may be due to a combination of Malware detection + most unix programs not really written to take advantage of the features NTFS has to offer This is a great talk on the topic link to this video
amiga386on Hacker News • Recommends
On “SQLite: 35% Faster Than the Filesystem”
NTFS can be especially awful if you're used to Linux behaviour, port code over to Windows and expect the same performance. Here's a talk on porting rustup to Windows: link to this video To begin with, it takes rustup 3m30s to install on Windows. After rejigging the code again and again away from its naive implementation which works fine on Linux, to perform sympathetically towards NTFS, it takes the same rustup 14s to install. That's quite a performance gain! But it needed a lot of changes to rustup, and to Windows itself.
miki123211on Hacker News • Recommends
On “Wild – A fast linker for Linux”
And as a result of this, if you want to optimize your program to be fast on Windows, you will make very different optimization decisions than on other platforms. For example, Windows runs virus scans on close(), which makes it very slow. This means that sometimes it makes sense to have one or more background threads exclusively dedicated to closing files. There's a good talk on this at link to this video
for_xyzon Hacker News • Mentions
On “Windows Subsystem for Linux: The lost potential”
…. Linux first software will always outperform windows implementation and Windows first software will outperform Linux implementations unless you provide separate code paths to properly handle underlying OS architecture and assumptions. On Windows closing the file handle is extremely costly operation due AV checks and file content indexing 1 1 link to this video…
d2waon Hacker News • Mentions
On “TeamViewer installs suspicious font only useful for web fingerprinting”
> Even if windows indexing aint malicious, it certainly behaves as if it is. The problem is that closing files takes a lot of time on Windows. link to this video
CoolColdon Hacker News • Mentions
On “Microsoft's backwards compatibility is insane”
to my understanding it's not really IO (reading/writing bytes), but overhead on OpenFile/CloseFile being under AV/Defender inspection. I'm taking that understanding basing on link to this video with " "NTFS really isn't that bad" - Robert Collins (LCA 2020) " > Why was rustup slow (3m30s to install (not including download time)) in early 2019 on Windows, and why isn't it slow (14s to install) now? >Early in 2019 I was developing some things in Rustlang on Windows, got frustrated at the performance of rustup (the installer for rust) and looked for bug. I found one which blamed a combination of NTFS, small files and anti-virus software for the problem. This wasn't an entirely satisfactory answer to my mi…
CoolColdon Hacker News • Mentions
On “Bun v0.8.0”
Here likely meant that system level hooks of antiviral software are invoked on CloseFile call when you do writes, not on OpenFile. You may want to take a look on link to this video
bayindirhon Hacker News • Mentions
On “Where Are the Sanity Checks?”
However, the code traverses that amount of data. It doesn't matter if you pass over the same inode n times. From OS point of view, there's no difference. Of course you can write a more complicated calculation algorithm, but it'll require tons of syscalls to determine the type of the file handle you're interacting with and will be probably very slow. This is what rustup did inadvertently back in the day, and it caused NTFS to melt down to a halt because of stat() overhead 0. 0: link to this video
nullindividualon Hacker News • Mentions
On “Windows NT vs. Unix: A design comparison”
But it simply isn't slower than Linux. Robert Collins explains that performance is just as good as Linux and the performance loss on Windows is due to file system filters (Defender)0. This is what DevDrive intends (and does) fix. 0 link to this video
amiga386on Hacker News • Mentions
On “MinC Is Not Cygwin”
…n/close a Windows filehandle each time because it can't make the calling code pass a filehandle instead. So it may be more comfortable and familiar, but also Windows APIs are unsympathetic to code designed for Unix originally. See this talk on how the developers of rustup massively improved their performance on Windows by restructuring their code: link to this video…