Project Ada
A mixed-reality workspace full of language-model agents. Ada listens, reads your paper notebook through the camera, takes screenshots, and plays word games with her colleagues.
Reconstructed from the hackathon and devlog videos. Bill's own account is still to come.
The Curiosity
In 2024 language models could suddenly see and hear. A headset can see and hear too. What happens when you put an assistant in the room with you, with panels you can grab, rather than in a chat box?
The Build
Ada lives in passthrough mixed reality on a Quest as a set of floating panels: a transcription history fed by the microphone, a chat, and tools she can call. Ask her to open the photo of your notebook and she reads your handwriting into a document. Ask for a screenshot and she takes one and transcribes it. Built in a week for an AI hackathon in May 2024, then extended for Meta's mixed-reality hackathon days later.
The same room later hosted a bake-off: Ada on GPT-4, Bella on Llama 3 70B via Groq, and GPT-4o, side by side, playing a word game and being judged on speed and sense.
VideoGPT-4o vs Llama3 70B vs GPT4 in MR GameThe Result
Two hackathon entries, a comparison video, and a question that fed directly into the web experiments a year later: Secret Sauce and Mystery Oracle both put a model behind a character and ask how it behaves.
A hot-dog vendor with a secret ingredient. A hotel clerk who knows where the famous author eats tonight. Talk them into telling you.
A cooperative lateral-thinking mystery. The Oracle knows exactly what happened and will only ever answer yes, no, or irrelevant.
A word game where your teammate is a language model. You give the clues; it has to guess the code word, and the taboo words are off limits.