News
Our humanoid avatar, Kilian, can now see. We’ve equipped him with a Vision LLM: Using a camera, he recognizes objects and reads text—and afterward, you can talk to him quite naturally about what there was to see. For example, if you hold a newspaper article or a letter up to his camera, he’ll summarize the content in seconds and answer questions about it.
The solution was developed as a showcase for an event hosted by the Hessian Ministry of Digital Affairs - a glimpse into what’s practically possible today with current AI technology.
How does it work?
Behind the avatar lies a coordinated interplay of several components: The video and audio streams are first converted into text; a vision model analyzes the image and the question and generates a response. This response is then translated back into speech—complete with matching lip movements and facial expressions. A game engine (Unreal with MetaHuman) renders the avatar in real time and sends it as a video stream to the browser.
This means users don’t need any special software—a standard browser on a desktop or smartphone is all that’s needed. Interestingly, “the AI” is only a small part of the whole process; the vast majority of the effort goes into the orchestration that brings all the components together in real time.
Why do we build something like this?
Kilian is a showcase—and that’s exactly the point. We build solutions like this to gain firsthand experience with the latest vision models and to test the stability of a complex overall system. And also to get a sense of where human-machine interfaces using avatars create real added value.
Many use cases are conceivable, ranging from a citizen service that explains a complicated government letter in simple language to a pharmaceutical company whose customers hold a partially torn package insert up to the camera and have the key points read aloud to them.
Try it out yourself
Kilian is available live on our website—give it a try, either on your desktop or directly on your phone: INOSOFT.de/kilian
Please note: Just like in a phone call, Kilian can only speak with one person at a time. If the system is currently busy, you’ll automatically be connected after a short wait.
Behind the Scenes
We discuss in detail how the interplay between the vision model, speech recognition, and real-time rendering works from a technical standpoint in the episode Kilian Reads Along—From Document to AI Conversation of our WeeklyTalk.