LLM drawing robot

In a three-week module we built a speech-to-drawing machine. You say what you want, and it draws a custom pen illustration for you to take home. It runs on GPT-3.5, Stable Diffusion and a plotter we built ourselves. The point was to explore new ways of interacting with natural language and AI models.

Team: David Polke, Elia Salerno, Andreas Kohler, Stepan Vedunov
Date: 2023, three-week university module
Role: Human-robot interaction, voice UI, ZHdK

My role

User research on human-robot interaction.
Facilitated ideation rounds.
Designed and built the voice-input front end.
Industrial design of the robot.

In action

Interacting with the robot. Plays from YouTube.

Process

Brainstorm of possible input mechanisms — Brainstorming input mechanisms.

Technical pipeline from spoken input to the final drawing — The tech pipeline, from human input to the drawing.

Sketches of possible industrial designs — Sketching versions of the industrial design.

CAD assembly of the drawing robot — The CAD assembly.

Talking to it

The voice interface borrows from voice messages in chat apps, to make talking to it feel natural. It was central to the idea: replicate a sense of AI sentience through conversation.

Animation of the speech-input interface — The speech-input interaction.

Result

Final 3D rendering of the robot — Final rendering of the robot.

The robot at the project exhibition — At the exhibition.

Outcome

The prototype is a speculative proof of concept for pseudo-sentience in AI. We wanted to learn about the field, and to shape how people interact with robotic systems.