LLMs in interviews
Written on August 26th, 2024 by Jonathan Hilgart
Large Language Models (LLMs) in interviews
Recently, there has been a lot of discussion about using LLMs for various asynchronous skill verification tasks.
• College essays? Everyone gets an A+ because they use ChatGPT to generate them.
• Interview questions? Gone are the days of take-home interviews as the barrier to producing high-quality answers. Now, bam, ChatGPT can generate anything for you.
• Coding questions? Why have candidates regurgitate algorithms when they can prompt an LLM to provide the answers?
Critics will point out that the current generation of LLMs can’t truly “reason,” and you still need a human to curate a coherent answer. Yes, but that won’t be true for the next generation!
Instead of these traditional methods of skill verification, we should focus on two, not new, methods to evaluate candidates:
1. Oral exams
2. Apprenticeships
Oral exams harken back to Socrates, and the key idea is simple: have a conversation that probes deeper and deeper until you determine the limits of the person’s knowledge. Yes, the burden of time and energy is now on the examiner, but in an era where knowledge is at your fingertips, this is the only way to ensure someone “owns” knowledge instead of borrowing it without understanding it.
Apprenticeships are very common in “blue-collar” work but haven’t caught on in traditional “white-collar” industries. These industries have traditionally relied on external signals (school credentials, leetcode, take-home assignments) to verify that a candidate is capable of completing the job described. However, given that anyone can access any and all knowledge, we should instead focus on the experience of working with the individual. Set them up for a 2-4 day (paid!) hackathon where they work alongside you in the trenches. There’s no better way to get to know someone than being alongside them in a sprint.
• Do you enjoy working with them?
• Are they proactive and take on work without needing you to assign it to them?
• Do they bring a fresh perspective to the table and another way to view the world?
These are the new facets to consider when hiring.
Agree? Disagree? Let me know what you think!
