Should you go back to AI-free coding interviews?
Usually not. If your AI-allowed coding interview gave results that were all over the place, and many candidates pasted the whole task into the assistant, the round did not fail. It showed you a real difference between engineers. What was missing is a way to turn that difference into a score. Before you go back, write down what a weak session looks like, change the task so that pasting it in does not work, and tell candidates what you score. Go back to an AI-free round only when you have a question that needs one.
A team that tried it and stopped
In a Hacker News thread from September 2026, “Ask HN: How do you interview devs in a post-AI world?”, one interviewer described two one-hour formats they had tried.
In the first, the candidate gets a codebase and a Docker image and writes a new service in any language. No agentic workflows are allowed. They wrote that this format “has been the most successful for us to find candidates”.
In the second, the candidate gets an existing full-stack TypeScript codebase a week or two before the interview. In the hour, they get one ticket to build, and they may use any agentic workflow they like. In the interviewer’s words: “I find a ton of variability in this interview and a lot of people will just take the entire ticket and one shot into Claude without prepping.” It led to “the biggest disparity in results of candidates so we stopped doing it.”
That was a reasonable decision with what they could see. Many teams that try an AI-allowed round will see the same thing, so it is worth looking at closely.
A wide spread is not noise
Read the description again. Every candidate had the codebase for a week. Some came prepared. Some pasted the ticket in and waited for the answer. Those are two different ways of working — and they are the same two you will see later on a normal workday, with a real ticket and nobody watching.
So the spread is what you were looking for. A format where everyone lands close together is calm to run, but it gives you less to decide with.
The trouble is what the round did with the spread. With no written scale, wide results look like a broken test, not a finding. And if one prompt can finish the ticket, the candidate who pasted it in and the candidate who planned may hand in code that looks the same. The difference is real, but the round has no way to show it.
Write down what the weak session looks like
The behaviour that worried this team is the easiest one to score. Write it as the lowest band on the scorecard, in plain words:
- pasted the whole ticket into the assistant before opening a single file
- kept what came back without running it
- cannot say what the change does, or which files it touched
Now that pattern is a low score with a reason behind it, not a reason to drop the format. And the other end becomes visible too: the candidate who opened the code first, split the ticket into parts, asked the assistant for one part at a time and ran each one before moving on. That is the engineer you are trying to find.
Change the task so pasting it in does not work
If a single prompt can finish the ticket, the round is mostly testing the model. Two other replies in the same thread show ways to change that.
One interviewer starts with a very small take-home — a CSV file with three columns and ten lines. In the live call, the candidate extends it without AI writing the code for them: one rule changes, and five lines are added to the file. Their result so far: “Every candidate that has submitted an ai submission thus far has failed because they have literally no idea where to go.” A change in the middle of the session shows quickly who understands the code they handed in.
Another plants a bad requirement in the task: “Something nonsensical that the candidate should give a bit of pushback on, and a coding assistant will probably happily implement for you.”
Both ideas fit an AI-allowed round. Add one moment where the assistant’s first answer is likely to be wrong — a rule that changes halfway through, a requirement that makes no sense — and the two kinds of candidate stop handing in the same code. The one who pasted builds the wrong thing. The one who read the task asks a question.
Tell candidates what you score
“Use any agentic workflow you want” can sound like “speed is what counts”. A candidate who reads it that way will paste the ticket in, because that is the fastest thing to do.
Say it plainly in the invitation. The working code matters, and so does how you got there: what you read first, what you asked the assistant, what you checked, and what you threw away. Say what the week with the codebase is for — read it, run it, find the tests. Then a candidate who pastes the whole ticket in has made that choice knowing the rules, and a low score is fair to them.
Score a few old sessions before you decide
If you still have recordings or notes from the AI-allowed round, take three or four and score them with the new lowest band. Two people score each session separately.
If the candidates you liked most now land far from the ones who pasted the ticket in, the round was working, and only the scale was missing. If they still land close together, the task is the problem, and changing the task is where to start.
When going back is the right call
An AI-free round is right when you have a specific question that an assistant would hide — for example, whether someone can read an unfamiliar error and reason about it alone — and you can supervise the round so that the rule is real. We wrote about when that holds.
The team in the thread may be right to keep its AI-free format; it works for them. But “the results varied too much” is not that kind of question. It is a sign that the round saw something and had nowhere to write it down.
Sources: Hacker News, “Ask HN: How do you interview devs in a post-AI world?” (thread started 19 September 2026), three replies posted 19–22 September 2026, quoted word for word.
Look at your last AI-allowed round. Can you say in one plain sentence what the lowest score looked like? If not, that is the part to fix first.
codesolara scores sessions this way: criteria with written bands, and every score points at the moment behind it. For writing the bands, see how to write an interview rubric for an AI-assisted round.