MicroStrategy (now Strategy)

The most expensive Databricks interview mistake: 'My cluster OOM'd, so I'd add memory.' Why $200k engineers never say this. This separates engineers who diagnose from those who throw hardware at problems. The interviewer is testing whether you know which JVM died and why. Here's the offer-winning answer: ⚠️ 𝗧𝗵𝗲 𝗥𝗲𝗷𝗲𝗰𝘁𝗶𝗼𝗻 𝗔𝗻𝘀𝘄𝗲𝗿 'I'd bump executor memory or scale up the cluster until it stops failing.' This signals you treat OOMs as a single bucket. There are two completely different failure modes, and the fix for one makes the other worse. ✅ 𝗧𝗵𝗲 𝗦𝗲𝗻𝗶𝗼𝗿 (𝗢𝗳𝗳𝗲𝗿-𝗪𝗶𝗻𝗻𝗶𝗻𝗴) 𝗔𝗻𝘀𝘄𝗲𝗿 'First question I'd ask: did the driver die or did an executor die? A 200GB cluster crashing on a 5GB dataset is almost always a driver OOM, which means something in the code pulled the data back to one JVM. I'd open the Spark UI driver tab to confirm before changing a single config.' ⚙️ 𝗗𝗿𝗶𝘃𝗲𝗿 𝘃𝘀 𝗘𝘅𝗲𝗰𝘂𝘁𝗼𝗿 𝗙𝗮𝗶𝗹𝘂𝗿𝗲 𝗠𝗼𝗱𝗲𝘀 𝗝𝘂𝗻𝗶𝗼𝗿: 'OOM is OOM, more memory fixes it.' 𝗦𝗲𝗻𝗶𝗼𝗿: Executor OOMs are distributed (skew, oversized partitions, spill). Driver OOMs are single-JVM (collected results, broadcast variables, planning overhead). Adding executor memory does nothing for a driver OOM. The cluster log will show ' java.lang.OutOfMemoryError ' on the driver, not the workers. 🔍 𝗧𝗵𝗲 𝗦𝗶𝗹𝗲𝗻𝘁 𝗗𝗿𝗶𝘃𝗲𝗿-𝗦𝗶𝗱𝗲 𝗖𝘂𝗹𝗽𝗿𝗶𝘁𝘀 𝗝𝘂𝗻𝗶𝗼𝗿: 'My code is distributed, Spark handles it.' 𝗦𝗲𝗻𝗶𝗼𝗿: Three patterns quietly bypass the cluster and run on the driver alone: .collect() and .toPandas() pull every row back to one JVM, and broadcast variables materialize the full dataset on the driver before shipping it out. A 5GB result on a 4GB driver heap dies instantly, no matter how big the workers are. 💡 𝗖𝗼𝗻𝗳𝗶𝗿𝗺𝗶𝗻𝗴 𝗶𝘁 𝗶𝗻 𝗦𝗽𝗮𝗿𝗸 𝗨𝗜 𝗝𝘂𝗻𝗶𝗼𝗿: 'I'd just check the error message.' 𝗦𝗲𝗻𝗶𝗼𝗿: Open the Executors tab. The driver row sits at the top with its own memory and GC stats. If driver GC time is climbing while executors look idle, the work is on the wrong JVM. The SQL tab shows if the last action was a CollectLimit, pointing to the exact line. 📝 𝗧𝗟;𝗗𝗥: A 5GB OOM on a 200GB cluster is almost always a driver OOM .collect(), .toPandas(), and broadcast variables silently move work to the driver Adding executor memory fixes nothing if the driver is the one dying Confirm in the Spark UI driver tab before touching any config What's the most expensive 'just add memory' fix you've watched fail in production? 👇 --- P.S. If you are preparing for interviews - I built cheat sheets for junior ($9), mid ($9), and senior ($19) Databricks interviews - plus a $24 bundle if you're between levels or coaching someone up. Each question has the answer that gets rejected and the answer that gets offers. See a link in the comments 👇