---
title: "Transcript: OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack"
category: "transcripts"
videoId: "akk6KRlcwW4"
sourceLabels: ["YouTube transcript", "Cached transcript markdown"]
wordCount: "2391"
---

# Transcript: OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack

## Source Video
- [YouTube](https://www.youtube.com/watch?v=akk6KRlcwW4)

## Local Cache
- `raw/sources/youtube-transcripts/akk6KRlcwW4.txt`
- 2,391 words

## Transcript

Probably all of you once planned the clean up activities. And you can imagine I also planned one weekend to clean up of my basement. And I just uh went there uh and went to clean up all of the stuff there. You know, throw up uh old washing machine. And I just look at the corner there and realized there's something weird there lying on the floor. So, I just grab it, look at the the strange device looks a bit familiar, but a bit old like from '80s with keyboard but with a weird displays. Of course, covered with dust. I just take it to my hand. Look at it.

And I was completely surprised that um device have a shell. And it's on. So, um I used my superpowers skills in a terminal and I asked I provided there a simple question. Who am I? And the device responds in the weirdest way I couldn't imagine. In the natural way language. So, the story I just described to you it can be a great story for a marketing. But honestly, um I just wanted to build a device which is physical and AI native like the device which comes from the future. Hello. My name is Lech Kalinowski.

I hold a PhD in physics and today I want to present you a device and uh backend and a system I built over last couple of months. And the whole story begins because I just simply wanted to build a remote controller to my open claw instance on my DJX park. And um the story is simple. Because when you are just first think about the LLMs, then you're not thinking about the audio, about the generative uh generative uh videos or pictures. First, you're thinking about the text. And then I just get the bright idea that maybe I can use some kind of better display to read and type with my LLM.

Of course, on the market, there is a electronic paper, but um I realized electronic paper can be a bit slow. Of course, it's perfect for reading, but if you just want to provide dynamically text to it, you proba- probably need uh faster display. And then I just uh used and uh in my design uh a simple connection of the OLED display, the small one, one color, with the electronic paper display. And then with this simple, let's say, dual display approach, I realized I can build quite a powerful and energy-efficient terminal to play with my claw. And um of course, behind it uh there is uh some kind of lot of complications.

Because from one side, I just wanted to build a remote control, but from other side there is such a niche on the market and this is the AI-native operational systems. And when I just started to think more clearly about it I just realized I can build such a system for my purposes. But not with a super powerful powerful computers, but with a small microcontrollers. And I just designed a system a built in let's say with a with a say uh four classes. One is the internal shell to control the terminal itself, you know, system settings, basic configurations, Wi-Fi connection.

And within my idea to build an AI-native device I build an assist control and uh one thing which really surprised me and what I will explain you later, RPG mode. And uh here it is. I started to prototype DGX Park at the um at the background here on the picture, two displays. One it's a live surface, that means the dynamic part. You can type the whole text provided is written there. And when you adjust trigger action, hit enter, then the whole render also happens on the second bistable electronical paper.

But there is a lot of complications and lot of tech and lot of steps you need to do to get an clear user experience and clear interface. So, I just found out a lot of the different approaches how to render properly on the two displays properly render the the the the interface. I just interact to control my LLMs and to get in responses and to do whatever I want with my DGX park through the through the open claw. And this is the one of the approaches that you can just have fixed static buffers and render one-bit images stored in the in the memory. That means the pages live in a pre-allocated memory.

There's a no markdown engine and no malloc on the MCU side. I just wanted to really to show you how the system is really complicated here. And this is not because I just wanted to use that much components. It's because there is a huge need to build a to build a power management system because I just blow up two these place over my prototype build. And um I just needed to be sure that the current and the voltage is stable as possible. So, here inside you have an MCU, which is ESP32 dual-core microcontroller. Here you have an OLED display, keyboard, and an encoder. And of course, the mention um power supply.

But, the A native end-to-end device is not uh let's say that simple because of the, you know, high power consumption and the and the hunger for the compute capabilities of your device. So, here is a full backend created. That means I just created a full backend and here's the presented architecture of it. That first we have a vault firmware deployed on the on the terminal, and there's a huge backend, which um handle all of the all of the agentic work with the open claw and handle the LLM.

I just want to point out here um within my today's presentation, I just want to demonstrate you um the device connected to an open-source GPT 120 billion parameters model. Um and I just serve that model with a let's say most possible optimal way using and I I I use their TensorRT serving system. And I just exposed OpenAI style to uh also like LLM proxy because I just hit a lot of walls with uh other open source models um because that's not all models match the style of the Open AI API.

Um as I said uh the first idea was to build and mhm play with the Open Claw through that terminal and this is the stuff I just do with this. Here you can look at the picture and see an example. I was I was possible to hit a command like write a Java example uh and store it on my local machine and Open Claw with the with the LLM support just did work and make it happened. Okay, and here's my field notes, the things that doesn't go as expected. Uh lessons learned. Uh first of all the uh software I2C and because I just wired up things there and it doesn't work correctly.

Um and there's a way to um to do proper control over over I2C without any additional physical pull-ups mounted. From other side GPIO 13 um there's a silent failure. Uh there was a need to move to the other port. Um to um to be sure that everything works correctly. From other side I need to build up the regulator the power supply um the whole unit of the power supply um because the the regulator uh kills the LED and and fragile parts of the display and cost me a lot of time and and getting the replacement parts from the market took so long. A couple of weeks.

And the last part, quality of your parts, that means encoder cheap and low quality give me a lot of rotational noise and there was a need to build up the the pull-ups and to and to wire it up additional capacitors there. But this is the funnest stuff. My favorite one. Because the this part really surprised me. You already know there's a huge part called RPG and behind it is a bit sad maybe story. Because uh I just never played RPGs game like on the paper. That means with a with a lot of friends.

You can go to a to apartment, you know, open a book and make an someone game master and play in a real text-based RPG. And after a lot of those years, I just never tried it. But I built um a RPG game and a console um which give me a pure experience of the text-based um RPG games, role-playing games. And actually I it's a bit funny because those device it's really like perfect fits for that kind of gaming. I build out an NPC and memory around that.

I just created the mood of the world, the contents, omnious, and I just used all of the, you know, the LNM um advantages to build uh the best uh role-playing game experience ever. Um I just created four different worlds. From one side, I just wanted to build something cyberpunk related. From other side, I just wanted something uh The Witcher part related, that means some kind of fantasy world with the dragons. But also my favorite one is the is the void in a deep space somewhere um in the cosmos. And this is really uh really good example how we can just use uh generative AI to build computer games.

From one side, I just generate the characters, their worlds, maps, the skills, uh and all of those converted with the mechanism I just described at the beginning of that presentation with the one-bit memory allocation, that means all of the pictures transform to the to the matrices, to the uh to the to the maps. And um one um important uh important thing, actually uh the device is really bulletproof. Um if the LED doesn't work, you the e-paper works. If the cardboard doesn't work, that means the keyboard, uh then you'll you have an encoder. Wi-Fi falls, you always can use your uh local uh shell. Um And um funny stuff, the area of the intellectual property there.

I just uh do some kind of research in the field and everyone just wants to build the devices around the audio interfaces, the video capture, you know, but for the quiet places, when you just want to, you know, sit down, keep calm, and play your RPG game, you don't need any of the, you know, colorful, powerful displays, you know, all of these, you know, popping up with the colors and all of the distractions, commercials, web pages, and other things.

Sometimes you just want to sit down and you know, do do the right job with the with the with the open claw or or or you know, just chat with your LLM, read and write in the quiet environment without any of the distractions. And there's a niche on the market. So, I also decided to build a the provisional patent here in that field and maybe it will bring some commercial device in the future to a to a market. Some numbers. In my project I just pushed 130 comments around 3 months of work. Two deep displays. Each of the display do other job. Four modes. For worlds, for gaming. In general, 16 classes.

That means you can generate RPG worlds inside of it. Firmware is is really tiny, really small, and it works extremely fast. And everything is powered by one single polymer cell. Lithium polymer cell. But maybe some remarks and takeaways. Much of this place to the job. Uh there's a plenty of them on the market. Keep the model of the metal because it's really heavy. That means for now we don't have such a models which runs really on the super tiny MCUs. In in means LLMs. And try narrative narrative. The context matters and in general not numbers.

So here I also record a the demo, the usage, how the device work and some kind of uh some kind of um last um last session. Um please enjoy. And I will place a couple of comments there. Device is booting. Connecting to a local Wi-Fi network. Here's the welcome screen and the main screen. Small display. Keyboard first simple help command. Here you will have all of the things you can do with the device. There's also a tutorial for gaming and tutorial for the device itself.

And when you just switch it on for the first time, then you need to or can go you don't have to, but you can go through the tutorial itself to learn how to use it. Usage of the Open Claw agent, that means right now I'm just checking my DGX Spark. And want to look at the disk space. Here's that. Let's maybe now look on the my favorite part with gaming. >> For worlds Older guns, neon abyss The hollows and the void witch. You can select your mm your character. And L and just generates the stuff. Everything is generated there. Um pictures, uh uh personalities of the of the people. And here, there's a mood.

There's a breach. You're in the deep space and can't deep dive into the other world. Yeah. >> So, that's it. That was the the video. Funny part is that the game like this without the three-dimensional graphics use the most advanced Nvidia uh processor mounted in the DGX Park that means the the the this is the bug and currently I just prepare for the demonstration. And so, the quiet game with the you know, text interface requires the the most powerful computer computer graphics card in the in the world.

And maybe another thing is that the presentation was about the handle device to control uh agent and open close with the local LLMs, but in general, it was about the Game Boy. The device to to play with LLMs. Thank you for the attention thank you very much for the for the Kostak support because I'm just working in at Kostak in the technological incubator uh and the whole work was supported by by Kostak company. Thank you very much.