Should you use an AI interviewer to screen engineers?
Only if you know what it scores. An AI interviewer is good at one thing: running the same first conversation with hundreds of candidates, at any hour, and writing each one up the same way. If your problem is interviewer hours, it helps. But many of these tools only talk with the candidate. They can judge how well someone explains their work. They cannot see the work. For an engineering role, ask what the score is built on before you buy one, and put the AI where the engineer will meet it on the job: next to them, while they build.
What an AI interviewer does well
Start with what it gets right, because it gets real things right.
- It does not get tired. The eightieth screen of the week gets the same questions as the first.
- It runs at any hour. A candidate in another time zone does not wait four days for a slot.
- It writes the same kind of summary every time, so two screens can be read side by side.
- It gives your engineers their hour back. An hour not spent on a first screen is an hour spent on the product.
For a role where the job is mostly talking, a good conversation may be most of what you need to know. Engineering is not that role.
What a talk-only screen cannot see
Look at what a voice screen leaves behind: the questions, the answers, a transcript and a score. Everything in that record is something the candidate said.
Now look at what an hour of real work leaves behind. The files they opened. The commands they ran. The test that failed, and what they changed before it passed. The suggestion from the assistant that they read and then deleted. Almost none of that is said out loud, and most of it would never come up in a conversation, because nobody thinks to mention it.
The two records answer different questions. The first asks: can this person describe engineering work well? The second asks: what do they do when they sit down to it? Some people are good at both. Some are much better at one. A screen that only listens cannot tell you which one you have in front of you.
There is a second gap. A good answer to “how would you find a slow database query?” is easy to prepare, and easier every year. A slow query in front of them, with the logs open and the clock running, is not.
Where the AI is in the job now
Many engineers now work with an AI assistant open all day. It suggests code, explains errors and drafts tests. The engineer’s part is the deciding: what to ask for, what to check, what to keep and what to throw away.
That deciding is a large part of what you are hiring for, and it only shows when the candidate has an assistant and a real task. An AI that asks the questions brings AI into the interview. It does not bring it into the work.
Some products now do both: an AI runs the conversation, and the candidate also gets an assistant for a coding part. That is a better design. The questions below still apply, because what matters is not where the AI is. It is what the score rests on.
Five questions to ask before you buy one
- What does the score rest on: talk or work? Ask to see one finished report. Look for a file, a command or a test run. If everything in it is a quote from the conversation, you have your answer.
- Can it show the moment behind each score? “Problem solving: 8 out of 10” tells you little. A score that links to the minute in the session, or the lines of code, that it came from can be checked by someone who was not there.
- Who checks the evidence before the score reaches you? And what happens when the evidence does not hold up? We wrote about why that check matters.
- Can a person change the score, and is the change kept? The AI should not make the hiring decision. A reviewer should be able to change any score with a written reason, and both the old score and the reason should stay on the record.
- What does the candidate know before they start? They should know an AI is running the conversation, whether they may use AI themselves, and what is being scored.
Where an AI interviewer still fits
- Questions about facts, not skill: work permit, notice period, salary range, time zone. A conversation handles these well, and nobody needs an engineer for them.
- Roles where talking is the job.
- One step before a work sample, not instead of one. If it decides who gets the real task, keep its bar low and its reasons visible. A screen that rejects on talk alone will turn away some good engineers who are simply not good at describing their work, and you will never know which ones.
The short version: buy an AI interviewer for the hours it saves. Do not buy it to find out how someone builds software. For that, give them real work and the AI they would use on the job, and look at what they do with it.
This post quotes no outside data. It is our argument, and we build the other approach, so read it with that in mind.
codesolara gives every candidate a real VS Code workspace with an AI assistant next to them, and every score points at the moment behind it. For what to look at in that session, see how to assess an engineer who uses AI.