Data Whisperers, Pinocchios, and Sentient Design

It took all of three minutes for Google’s NotebookLM to create this lively 12-minute podcast-style conversation about Sentient Design:

Bonkers, right? The script and the voices—along with the very human pauses, ums, and mm-hmms—all of it is machine-generated. It even nails the emotional tenor of the podcast format. I gave NotebookLM several chapters of the Sentient Design book manuscript, pushed a button, and this just popped out.[1]

The speed, quality, and believability of this podcast are remarkable, but they’re not even the most interesting thing about it. Here’s the bit that gets me excited: by transforming data from one format into another, this gives the content new life, enabling a new use case and context. It’s a whole new experience for content that was previously frozen into a different shape. Instead of 100 pages of PDFs that require your eyes and a few hours of time and attention, you’ve got a casual, relatable conversation that you can listen to on the go to get the gist in just a few minutes. It’s a new format, new mindset, new context, new user persona… and a new level of accessibility.

Listen to the data whisperer

In Sentient Design, Veronika and I call this the data whisperer experience pattern. The data whisperer shifts content or data from one format to another. One super-pragmatic example is extracting structured data from a mess of unstructured content: turn blobs of text into JSON or XML so that they can be shared among systems. Machine intelligence is great at doing things like this—translation among formats. But this can go so much farther than file types.

If you listen to the podcast, you’ll hear the robot hosts talk about the Sentient Triangle, a way to describe different postures for machine-intelligent experiences. Data whisperers stake out the interoperable point of the triangle.

Interoperability is typically associated with portability among systems. But data whisperers become even more powerful when they enable portability among experiences. Instead of focusing only on the artifacts that machine intelligence can generate with these transformations, consider the new interaction paradigms they could enable.

As a designer, ask yourself: what becomes possible when I liberate this content from its current form, and how can machine intelligence help me do that? And even better: what if you think of it beyond rote translation of modes or formats (text to speech, English to Chinese, PDF to JSON) but also manner, interaction, or even meaning? That’s what the podcast example hints at; it’s doing something far more than converting 100 PDF pages into audio. It’s reinterpreting the content for a whole new context.

Tim Paul of GOV.UK put together this side project showing another example of using AI to rescue content from PDFs. His experiment translates static government forms into interactive multi-step web forms rendered in the gov.uk design system.

Here at Big Medium, we did something similar when we built a Figma plugin for one of our clients. The plugin takes a sketch (or text prompt or screenshot) and creates a first-pass Figma layout using the company’s design system components. Here it is in action, acting as a sous chef to organize the components so that the designer “chef” can take over and create the final refined result:

The Pinocchio interaction pattern

This plugin is a tight example of the data whisperer pattern: it thaws meaning and agency from a frozen artifact. But it’s also doing something more: it transforms content from low fidelity to high fidelity.

In Sentient Design, we call this the Pinocchio interaction pattern—turning the puppet into a real boy. Use the Pinocchio pattern to flesh an outline into text, zap a sketch into an artwork, or transform a wireframe into working code. Pinocchio is a data whisperer that elevates an idea into something functional.

Here’s another Pinocchio example. tldraw is a framework for creating digital whiteboard apps. The tldraw team created a “Make Real” feature that transforms hand-drawn sketches or wireframes into interactive elements built with functional code. Just draw a sketch, select it, and click the Make Real button to insert a working web view into the canvas. It’s an inline tool to enable quick and easy prototyping, turning a Pinocchio sketch into “real boy” markup.

When you bake the Pinocchio pattern into the fabric of an application, its interface becomes a radically adaptive surface—a free-form canvas that adapts to your behavior and context. You can see Apple working this angle in the iPad applications it previewed earlier this year. In Notes for iPad, Apple’s demo shows how you can circle a sketch, and the app creates a high-fidelity drawing based on the sketch and surrounding content context.

Or the Math Notes feature of iPad’s new calculator app lets you scrawl equations on the screen, and it transforms them into working math—spreadsheet functionality in sketch format. Scribbled numbers become variables. A column of figures becomes a sum. An equation becomes an interactive graph.

Experience over artifact

With generative AI, the generation tends to get all the attention. That’s a missed opportunity. With the data whisperer and Pinocchios patterns, you can create not only new content artifacts but new experience paradigms. You can liberate content and interaction from frozen formats, unlock new use cases, and help people move from rough idea to refined concept. Lean into it, friends, there’s much to explore here.

Is your organization trying to understand the role of AI in your mission and practice? We can help! Big Medium does design, development, and product strategy for AI-mediated experiences; we facilitate Sentient Design sprints; we teach AI workshops; and we offer executive briefings. Get in touch to learn more.