For a long time, the coding interview measured one thing reasonably well: can you, alone, in a quiet room, turn a problem into working code under time pressure. It was never a great proxy for the job; software development has always been a team sport of building, integration, and maintaining. But it was something. That proxy is now dead. The frontier model in the candidate's editor would ace the take-home.
I have no interest in making people work in a sterile sandbox, or spending hours on unpaid take-home homework. Instead, we run a live, forty-five-minute coding exercise. We hand them a small, real-world task: fetch data from a public API, transform it, and display it in a clean format. Do the actual job we want you to do. And we explicitly tell them to use AI assistance (Copilot, Cursor, Claude, or whatever feels best).
But there is a catch: that forty-five minutes has to cover both the building and the judging. How long the candidate spends coaxing the model to generate the solution eats directly into their review time. The brutal reality of the round is that most candidates never even reach the review; they spend so long navigating the spec and fighting with the model's first drafts that the clock runs out before they can verify a single line of output. The standouts are the ones who steer the model efficiently, get to a stable draft in fifteen minutes, and leave the remaining half-hour to audit the model's silent assumptions and write tests.
How they build is how they think
Watching someone review the code after the LLM has built something is how you know whether they actually know code, and whether they can spot a failure from the model. It is a pure window into their decision-making. Do they notice the test that asserts nothing? Do they trust the confident, model-generated comment or check the actual API payload? The candidates I want are the ones who can look at what the model produced and find where its logic falls over.
But the most important signal in the entire forty-five minutes is at the very start: how they translate the challenge description into a specification. A mediocre candidate copies the raw, messy requirements and pastes them into the prompt window. The standout candidate stops. They reframe, clarify, and translate the product description into a rigorous engineering spec before they ever touch the model. They constrain the AI first, because they know that the quality of the spec determines the fidelity of the generation.
Conversely, if they spend too much time on the spec, it is a waste of time. We are not writing a treaty; we are coding on a forty-five-minute clock. The candidate needs a sharp sense of "good enough" — the threshold where planning ends and execution starts — and then fall back on the old IRC advice: TIAS (try it and see). Build the spec, get the first draft on disk, and run it. You cannot verify a model's assumptions from the chat history; you have to look at the logs of the live environment.
During interview seasons, I sit in these rounds with my local meeting recorder running in developer mode (I built Grey Eminence specifically to keep these transcripts on my own disk). Reading the transcripts later is a masterclass in calibration. The candidates who "talk the talk" with perfect jargon often fail so friggin hard when they meet the real code, because they are just running their own pattern-matching retrieval loops. But the candidates who can actually ship are different; they are the ones who look at what the model typed and instantly spot the load-bearing flaw.
I script the entire panel as a ninety-minute, high-signal loop so every interviewer runs it the same way:
01. The oblique warmup
Resumes are fiction, so I never read them before the round. Instead, we spend fifteen minutes talking about recent work, things they built that were genuinely fun, or what they want to learn next. I ask questions based on their answers and I come at them obliquely.
For example, rather than asking how much experience they have with Python, I ask: "What do you hate about Python?" If they do not have anything they dislike about their primary tool, they have not been using it long enough. A senior who has spent years in a language carries scar tissue; they should be able to rattle off personal pain points in seconds.
02. The live challenge & review
This is the API-fetch-and-transform core. How fast the candidate gets the AI to generate the first draft determines how much time is left for the review. If they spend forty minutes fighting the spec, they have five minutes to judge the code. Managing the model is part of the test.
A mediocre candidate dumps the spec into the chat box, hits enter, and prays the first output runs. The standouts operate differently: they treat the model like an assistant, not a magic box. They take the challenge, feed it to the model, and ask it to summarize and probe the spec for understanding first. Only when they are comfortable that they and the model agree on a plan do they execute.
And they do not look away while it is typing. They watch the scrolly bits. If their gut gets triggered by something in the stream, they halt generation instantly to ask what it is doing, or remember the line for when the model stops typing. Once they have a draft, they have the model generate tests to verify the build, run a refactoring task to simplify the code, and only then do they roll up their sleeves to inspect the code.
This is where we measure real taste: do they let the model ship a monolithic, eighty-line function where all operations live together, or do they force it to separate fetching from transformation? Do they audit how it handles API failures or missing payloads? And do they catch the absurd AI hallucinations? (I have watched models bafflingly throw an infinite for(;;) loop into TypeScript code processing a finite array of data, and watched careless candidates wave it through.) The standout catches the infinite loop, rejects the monolith, and refuses to ship until the boundaries are verified.
03. System design
The system design is an entirely new thing that we expect them to draw out using some drawing tool. The prompt is practical: design a system that takes user surveys and tries to match them up to benefits.
It is incredibly interesting to see if they use the models to help them with this part. They are completely free to use AI as a sparring partner (asking it to brainstorm components, validate DB schemas, or outline the data flow). But it is exceedingly rare that they do. It makes me sad, honestly; it shows how deeply candidates still compartmentalize the model as a mere syntax typewriter, forgetting that its highest and best use is as a cognitive partner for architectural trade-offs.
04. The handoff
The final five minutes are for the candidates to ask us anything they want. It is a soft-looking segment, but it carries a massive signal. A candidate who says, "I don't really have any questions" is prolly not going to make the cut. Inquisitive engineers are the only ones who survive in a world where the machine answers the easy questions; if you cannot think of a single thing to ask a future peer about their platform, your curiosity is already retired.
Generation is cheap now. The expensive thing (the thing they earn a salary for) is judgement about which generated thing to keep.
Seeding the spec with traps
A great interview challenge doesn't need to be massive. We keep the core task simple (fetch, transform, display) but we seed the requirements description with a few quiet, highly plausible logic traps. We want to see if the candidate's translated specification catches these before they prompt, and whether their code review catches where the model fails them:
The candidate's job isn't to nitpick the formatting or complain about the naming style. I want the ones who reframe the spec, find the load-bearing logic bug during review, and refuse to ship until it is verified. Nitpicking is cheap; triage is the job.
What I stopped weighting
I no longer care about speed of typing. I have never felt any value in LeetCode-like challenges; I have always valued real-world examples (fetch data from this API, process and display it). Do the job we actually want you to do. None of those abstract puzzle algorithms predict who is good on a team where most first drafts arrive from a model.
What predicts it is simple: knowing when a thing is done, knowing when it is wrong, and being honest about which of the two you are looking at.
None of this is a clean science. I am still calibrating, still arguing with my own panel about what a "strong no" looks like. But the shape is clear enough to commit to: hire for taste and judgement, interview by handing people work to evaluate instead of work to produce, and stop pretending the keyboard is the bottleneck.
It is exhausting, of course. Interviewing is heavy, non-linear work, and when you are staring at your third panel of the week while your own backlog is pile-up high, it is easy to treat it like overhead. That is a dangerous mistake. Hiring someone onto your team is the single most important thing you can do as an engineer. Much like jury duty, it can often feel like a waste of time, a chore that takes you away from your "real" work. But you cannot let fatigue win out. Safeguarding who joins your team is a necessary part of engineering culture and team cohesion, the same way participating in a jury is a key underpinning for a functioning society. You are guarding the joy of delivering surprises that make your users love the work you do. You are guarding the work/life balance of yourself and your teammates.
