IDA Pro Decompiler Basics Microcode and x86 Calling Conventions

An exploration of how IDA Pro optimizes its microcode intermediate representation and how x86 calling convention bugs can affect decompilation.

** I MADE A MISTAKE ** Check out Rolf's excellent explanation of how I got the microcode optimization part wrong: reverseengineering.stackexchange.com/questions/26803/hex-ray… ----- OALABS DISCORD discord.gg/6h5Bh5AMDU OALABS PATREON patreon.com/oalabs OALABS TIP JAR ko-fi.com/oalabs OALABS GITHUB github.com/OALabs UNPACME - AUTOMATED MALWARE UNPACKING unpac.me/#/ ----- Join us for a look under the hood at how IDA Pro optimizes their microcode (IL) to provide a clean decompiled view. We also reverse engineer a bug in calling convention identification for x86 can lead to cascading issues in the intermediate representation as it is optimized. Automated Malware Unpacking unpac.me/ Introduction to IDA's microcode youtube.com/watch?v=T-YkhNElvng __fastcall docs.microsoft.com/en-us/cpp/cpp/fastcall?view=msvc-160 __thiscall docs.microsoft.com/en-us/cpp/cpp/thiscall?view=msvc-160 How to force a call type hex-rays.com/products/decompiler/manual/cmd_force_call_type.… Lucid IDA Plugin View Microcode github.com/gaasedelen/lucid Feedback, questions, and suggestions are always welcome : ) Sergei twitter.com/herrcore Sean twitter.com/seanmw As always check out our tools, tutorials, and more content over at openanalysis.net #ReverseEngineering #IDAPro #IntermediateRepresentation