Introduction
Large language models are good at finding answers in the "middle" of data in their training sets, but what of outlying "niche" information? This exercise pits human expertise in esoteric topics against the accumulated knowledge that generative AI systems have been trained on.
Your team will need three roles: an expert in an obscure specialty, a typist to trade questions with the AI, and a scorekeeper to keep track of how many correct answers earned by each side.
You should use best practices for prompting, eg by asking the AI to play the role of an expert in your subject and to ask questions at an advanced level.
Instructions
- As a team, pick a unique specialty of one or more team members that makes them qualified to answer obscure questions about the topic.
Examples might include motorcycle maintenance, teams that beat the Celtics in the 1980s, or albums by an obscure Folktronica band.
- Come up with 5 esoteric questions you know the answer to but the AI may not.
- Using the tool provided in class, ask the AI your 5 questions and count how many it gets right.
- Using best practices for prompting, ask the AI to challenge you with 5 expert questions of its own.
- Report orally to the class how many questions the AI got correct versus the humans got right.
The instructor will tally the results to see who won. (There is no Slack submission for this exercise.)