How to write an interview rubric for an AI-assisted round
To write a rubric for an AI-assisted interview, write it at the same time as the task, not after. For each criterion, describe high, mid and low as things two people watching the session would both agree happened. Write the low band first and the mid band last. Then have people on your own team do the exercise under interview conditions and score them, before any candidate sees it.
A rubric is the written scale behind each score: for every criterion, what a high, a mid and a low performance look like. We published a filled-in example two weeks ago. This post is about how to write your own, because the bands that fit your role are the ones nobody can write for you.
Start from the task, not from a list of criteria
Most rubrics begin as a list of nice words: problem solving, code quality, communication. Then someone writes a task and hopes the words apply to it.
Turn that around. A criterion is only worth scoring if the task makes it happen in front of you. If you want to see whether a candidate checks the assistant’s work, the task needs a place where the assistant is likely to be wrong: a test that its first fix will edit instead of the code, or a requirement it will build without asking about. If nothing in the task forces the moment, the criterion gets scored from a general impression.
This matters more once AI is allowed, because the finished code no longer separates people. DoorDash, describing its new 60-minute AI-assisted round in March 2026, put it plainly: “This format cannot be graded with a simple rubric that checks for the right output.” What they score instead is a short list of behaviours: “how you get oriented, how you use AI and verify outputs, how you debug, how you manage scope, and how you communicate tradeoffs.”
When the task is open and every candidate builds something different, the same rule holds from the other side. Sierra wrote in April 2026 that it uses “a set of evaluation criteria that are agnostic to what the candidate builds”. The criteria describe how people work, so they still apply when the products differ.
Turn each word into something you can see
Take one criterion and rewrite it until an observer could tick it off.
The usual first version is “strong debugging skills”. Two interviewers will agree with that sentence and give the same candidate a 1 and a 3, because “strong” is a feeling.
Now tie it to the task. Say the exercise has a pagination bug: the second page repeats the last item of the first. The assistant’s first suggestion is to change the test’s expected value. (Illustrative task, not one we ship.)
- Low: keeps the assistant’s change to the test, or edits code until the test passes, and cannot say what the bug was.
- High: runs the failing test before changing anything, finds the line where the page offset goes wrong, and refuses a fix that edits the test instead of the code.
- Mid: finds the real cause, but only after trying several fixes at random, or fixes the first page case and never checks the last page.
Each band now names something that either happened or did not. Two people can still disagree about the score, but they will be disagreeing about the same moment.
A quick test for every band you write: delete these words and see what is left. Good, strong, solid, clean, appropriate, effectively, understands, comfortable with. If nothing is left, the band was an opinion. “Understands the code” becomes the thing they did that showed it, such as explaining why a line was there before deleting it.
Write low first, then high, then mid
Low first, because it is the score you will have to defend. A candidate who asks why they got a 1 deserves a sentence like “submitted with a failing test”, not “below expectations”. If you cannot describe low in one plain sentence, the criterion is not ready.
High second. This is usually the easiest band, which is why it is the one most rubrics stop at.
Mid last, and not as “some of high”. A mid band that says “partly does the high things” gives the scorer no help at all. Look at what real middle candidates actually do. It is usually a specific mixed pattern: the right fix found slowly, the assistant used well for small pieces but trusted blindly on the hard one, a good explanation given only when asked. Write that pattern down.
Name the record each band points at
An AI-assisted session leaves a record: every message to the assistant, every change to the code, every test run. Each band should say which part of that record would prove it.
For the debugging example: the test run before the first change, the diff around the offset, and the message where the candidate refused the test edit. When the scorer writes a score, they cite that moment, with a time or a file and line. If you cannot name any part of the record that would show a band, the band is describing something you cannot see. Rewrite it or drop the criterion.
Decide what a hint costs, before you need to
In a live round, the interviewer sometimes helps. Decide in advance what that does to the score, and write it on the rubric, or every interviewer will decide in the moment and differently.
The assistant is part of the task, so using it is not a hint. Help from the interviewer is different. Two workable rules:
- A hint on a criterion caps that criterion at mid.
- The hint is written in the record with its time, and the score is about what the candidate did after it.
Either is fine. Having no rule is what makes two interviewers’ scores impossible to compare.
Test it on your own team before a candidate sees it
Perplexity describes a careful version of this on its candidate guide for hands-on coding, a round where AI is not allowed: “we calibrate each exercise by having Perplexity’s own team members attempt the exercise under interview conditions.” They use those sessions to “define expectations for what strong performance looks like”, and the panel includes people who no longer “code by hand in their day jobs”.
The method works the same for an AI-assisted round. Ask three or four colleagues to do the exercise with the assistant, under the real time limit. Score their sessions with your draft rubric. You will find bands that nobody could reach, a mid band that matched nobody, and a criterion the task never actually tested. Fix those now, not after ten candidates.
Then keep checking. A task that tests AI use has a short life, because a newer model may solve it outright. Coinbase, in July 2026, called this “the half-life problem”: a question “may be trivially solvable by next quarter’s model”, so it reviews its loop every quarter and retires questions that stop “producing signal”. When the task changes, the rubric changes with it. They were written together, and they expire together.
The short version
Write the task and the rubric at the same time. Make every band something an observer can see. Write low first, high second, and mid from what middle candidates really do. Name the part of the session record that proves each band. Decide what an interviewer’s hint costs. Score your own team on it before any candidate, and rewrite it when the task changes.
Sources: DoorDash, “Why DoorDash is rebuilding its engineering interviews around AI” (19 March 2026); Sierra, “The AI-native interview” (22 April 2026); Perplexity’s hands-on coding candidate guide (undated, read October 2026); Coinbase, “Interviewing engineers in the AI era: lessons from a year of rebuilding” (13 July 2026).
Take one criterion from your current scorecard and remove the words good, strong and solid from its bands. What is left is what your interviewers are really scoring.
codesolara rubrics work this way: criteria with authored bands, and every score cites the moment behind it. For a ready-made card to start from, see the interview scorecard template, and for what to grade in the first place, how to assess an engineer who uses AI.