# SIG-DRG 2026-08-20 — Session Notes Participants: - Anuraj R | UTC+0300 - rafa (UTC+1) - shreeram - Spencer Norwick U-0700 - Maier - 🙊 Jenna Dixon | UTC-5 - Giovanni Merlino | UTC+2 --- ## 📖 The Reading The primary reading was Anthropic's report/blog post on using LLMs to control robots (referred to as "the Anthropic one" / "the first link"), alongside a second linked piece described as "the drone link." Exact titles were not stated in the transcript. The discussion centered on Anthropic's experiments testing LLMs at various levels of robot control (limb control, task planning, navigation) across different robotic platforms and models. ## 🧭 Overview The group went around sharing reactions to Anthropic's robotics/LLM experiments, then broadened into discussion of the practical stack for LLM-controlled robots — teleoperation, latency, and where compute should live. Jenna introduced an organizational note about a planned PI (Protocol Institute) quarterly journal and possible written/interactive outputs from the group. The session ended with Anuraj demonstrating a live browser-based teleoperation setup controlling his robot in Finland. ## 💡 Key Points & Themes **Reactions to the Anthropic reading** - **rafa:** Robotic control by LLMs feels increasingly like a "solved problem" in principle — the open questions are *when* and *which narrow use cases* get solved first. Noted a recent general AI model for robotics that learns by watching examples. Raised the "stack of problems" that remain even if driving/control is solved: latency, real-time reconstruction, and deciding what compute lives on the robot vs. off it. Anchored this in his own struggle to get reliable home Wi-Fi/mesh networking. - **shreeram:** Wants to see the actual implementation code, not just the writeup (his robotics background is minimal). Found it notable that in the "policy task," supervision made older models *worse*. Confused by unexplained ~20-second gaps in the pendulum/classic-control tasks. His main critique: the paper never establishes what the *best non-LLM method* can achieve as a baseline, so it's unclear whether LLMs are actually the right tool or just a hammer being applied everywhere. - **Spencer:** Most interested in the experiment-level details — how small setup changes yield outsized improvements. Highlighted two examples: adding a "compass" giving the locomotive robot orientation in degrees, and adding a crosshairs/reticle to the camera view. Argued we should look for ways to help models "in the way they want to be helped," rather than forcing them into human-style control paradigms. - **Anuraj:** Had a "meh" reaction. Argued the path of using a language model alone won't yield much for real-world navigation — you need a *world model* for embodied action (humans don't verbally reason about walking or holding a cup; it's a different mode of thinking). Thinks LLMs are better suited to higher-level planning, while low-level control needs models trained from scratch with embodiment. Agreed with shreeram's "hammer" framing — Anthropic seems to try many reward setups to see what sticks. Suspects robotics isn't a major Anthropic priority, hence the simple experiments. - **Maier:** Read the goal as a mix of *safety* (will people use our AI to build killer robots?) and *capability assessment* (how good are LLMs today?). Noted they tested LLMs at ~four levels of abstraction but tried to solve all layers with one model — whereas he'd use a *different LLM per layer*, both because that mirrors biology and because these problems are too hard to solve simultaneously in one decision. Called Anuraj's "let LLMs do what they're good at, glue from elsewhere" view exactly the kind of prescription the paper lacked. Found "points of light": LLMs writing their own control code, and the question of what *protocol* a robot should use to get help from an LLM (given LLMs are energy/CPU heavy — maybe a robot runs a small/pretrained model locally). **Practical stack & experiments (Anuraj)** - Built a teleoperation effort for his robot last weekend; fed camera screenshots into the controlling agent, which (with Claude memory/context enabled) could locate a "Tumblr" robot it had prior context on — demonstrating higher-level visual reasoning. - Proposed connecting robots directly to real-time responses/live-audio APIs (e.g., ChatGPT's real-time transcription) to see how robots behave with fast voice control. - Physical safety limits can be enforced on the robot itself, so it won't execute dangerous commands even if the LLM issues them. **Documentation as interactive experience (rafa)** - Suggested going beyond static diagrams: ask Claude to render a codebase as an *interactive walkthrough* or even a 3D/game-like exploration to build intuition (e.g., understanding a teleoperation failure by "seeing the shape of the code"). May experiment with this over the weekend. Consolidated accumulated workshop info (including a shopping/equipment list) onto a single website. **Two architectures for voice control (Maier)** - (1) Human talks → single LLM translates speech into WebSocket commands → robot complies (supports compound commands like "go forward five times"). - (2) Two LLMs: a *private* LLM that translates human intent into commands, and a *black-box* server-hosted LLM acting as a translation/protection layer for the robot (analogous to a robot operating system). Anuraj connected this to the psychohistory/SIG-SAI "world machines" project, where a public-facing LLM front end passes through a "witness" that returns only vetted content from a corpus — here the corpus would be replaced by a real robot. ## 🔀 Questions & Disagreements - **Is LLM-only control the right approach?** Anuraj (and to a degree shreeram) are skeptical for low-level, real-world navigation, favoring embodied/world models; Spencer and rafa are more optimistic about incremental improvements and helping the model in its own terms. Not resolved. - **Missing baseline:** shreeram's open question — what's the best a non-LLM method achieves, and is LLM control even the best path forward, or a hammer looking for nails? - **Unexplained experimental details:** the 20-second gaps in classic-control tasks, and why supervision degraded older models. - **One model vs. many:** Maier questioned Anthropic's use of a single LLM across all abstraction layers. - **How to "publish" an interactive robot demo** (Maier): a written report doesn't capture the experience; letting people interact does. Jenna and Maier debated the tension — exposing voice control on the internet raises security ("how do you keep the bad guys out?") and it must be gated somehow. - **Token-gating friction:** Maier argued a non-token-gated demo solution would be valuable to avoid shrinking the audience with wallet requirements — but he had no solution to offer. - **Latency:** Spencer asked how responsive/laggy teleoperation feels; measured ping was ~75–80 ms (US East Coast / Helsinki to Finland), ~170 ms from further east, with West Coast expected higher. ## 🔗 References Mentioned - Anthropic's robotics/LLM experiment report (main reading) and a second "drone link." - A recent general AI model for robotics that learns by watching examples (mentioned by rafa; not named). - The Ukraine war (rafa, as context for the "dystopian" feel of military robotics). - Claude / Cloud (used for code generation, diagrams, agent control with memory). - ChatGPT / Codex real-time live audio & responses API. - WebSockets; a gateway/proxy; FastAPI server on the robot; JSON command structure; "Xeno" (phonetic — likely a binary serialization format) for binary streams; standard ROS-style commands. - SIG-SAI's "world machines" project, with a public-facing LLM front end and a "witness" gating layer (attributed to "Anish"). - The PI (Protocol Institute) planned quarterly journal / nonfiction outputs (Jenna, working with "Timber"). - Robot hardware kit: Raspberry Pi, batteries; two GitHub repos; a hosting provider offering ~1 GB free bandwidth (phonetically "Animgra" — unclear). - rafa's marketplace project (to potentially integrate with the robot demo). ## ✅ Action Items & Next Time - **rafa:** May experiment over the weekend with LLM-generated interactive/game-like code walkthroughs; continue consolidating workshop info (equipment/shopping list) onto the group website. - **Anuraj:** Continue teleoperation experiments; plans to replace JSON commands with binary/ROS-style commands; explore payment gating (crypto wallet or Stripe) since camera streaming consumes bandwidth; possibly integrate with rafa's marketplace. Will post the GitHub repo links and demo URL in the group chat. - **Follow-up:** Explore collaboration with the psychohistory/SIG-SAI "world machines" group — possibly reusing their code — to build a public-facing robot control front end (Anuraj). - **Workshop prep:** Participants who want hands-on experience should buy the robot kit now; assembly takes roughly one weekend, so they can focus on the software stack during the workshop. - **Jenna's asks for the journal:** SIG hosts to write a forward-looking piece (~500 words) on the group's focus for the next three months (targeting Q4 2026 activity for issue #2, vol. 1), to be posted as a call for participation. Also floated writing up the demo build (stack, tooling, what's manual vs. LLM-generated) as a "lab research note." - Consideration of publishing an interactive/live web demo (with sliders or direct robot control) as a group output — pending security/gating solution. ## ⭐ Memorable Quotes - **shreeram:** "they have this hammer, and then they're just trying to fit it everywhere." (paraphrasing the critique later echoed by Anuraj) - **Maier:** "instead of doing your work, you just phone it in ... mediated by an LLM, and somewhere around the world, a robot starts walking around and doing whatever it does. Now how do you publish that?" - **Anuraj:** "since last December, I haven't written any manual code. At least at home ... this is also made with Cloud, the graphic itself."