Built with Unity WebGL

Challenge

This project involved developing a web application featuring personalized AI assistants with real-time 3D avatars. The goal was to help users manage everyday tasks with a visually interactive assistant—directly in the browser.

During development, several key challenges emerged, especially when targeting Safari and iOS devices: Safari and iOS devices:

Memory Crashes in Safari

The Unity WebGL build relied on a caching system that stored avatar assets in memory without releasing them.
Each time a new avatar was loaded, memory usage increased. Over time, this caused Safari to crash due to excessive memory consumption.

Audio Not Playing on iOS

Safari on iOS blocks audio playback until a user interacts directly with the page.
This caused the application’s voice assistant to remain silent on load, breaking the user experience.

Avatars Stuck in Loading

Some avatars would become stuck during the loading phase, never completing initialization.
This was caused by large model sizes and dropped packets on slower internet connections, leaving the loading process incomplete.

Solution

I addressed these platform-specific issues with a combination of optimization and user-driven interaction flows:

Removed Unity’s Caching System for WebGL

Disabled Unity’s default asset caching for WebGL builds, ensuring unused assets were properly discarded.
This stabilized memory usage across avatar changes and prevented long-session crashes, especially on Safari.

User Interaction Trigger for Audio

Implemented a start button that required a user tap or click before initializing the voice assistant.
This complied with Safari’s autoplay policy and ensured audio would play reliably after the user engaged.

Avatar Optimization & Load Handling

Reduced texture resolution and removed unnecessary geometry from avatar models to make them lighter and faster to load.
Improved asset streaming logic to detect failed or incomplete downloads, allowing the app to retry or fallback gracefully, avoiding stuck avatars.

Result

The result was a stable, cross-platform WebGL experience with responsive 3D avatars and smooth audio interaction. Users on iOS and Safari could now engage with their personalized AI assistant without crashes, audio issues, or avatar loading problems. These optimizations improved both performance and reliability, helping deliver a polished and accessible AI assistant experience in the browser.

Interruptible Dialogue and Mute Functionality

Challenge

During testing and user feedback, two key issues were identified with the avatar’s audio interaction system:

There was no mute option, meaning users couldn’t silence the assistant if needed—leading to poor accessibility and user control.
Once the avatar started speaking, it couldn’t be interrupted. If a user asked a new question mid-response, the assistant would continue talking over the new input, creating confusion and a disjointed experience. These issues negatively impacted the usability and fluidity of interactions, especially in quick, back-to-back conversations.

Solution

To enhance user control and improve the conversational flow, I implemented two key features:

Mute Functionality

Added a simple mute toggle that allowed users to enable or disable the avatar’s voice output at any time.
This gave users instant control over the assistant’s audio, improving the experience in quiet or shared environments.

Interruptible Audio Playback

Updated the conversation logic to detect when a user asked a new question.
If the avatar was currently speaking, the system would immediately stop the ongoing audio, clear the current speech process, and begin playback of the new response.
This made conversations feel more natural, responsive, and interactive.

Result

These improvements gave users more flexibility and control over their interactions with the AI assistant. The ability to mute the avatar and interrupt ongoing dialogue created a smoother and more dynamic experience, bringing the assistant closer to how real conversations function in everyday life.

User-Uploaded Avatars and Environments

Challenge

To make the AI assistant experience more personal and flexible, a new feature was needed that allowed users to upload their own 3D avatar models in formats such as FBX and GLB.

The challenge was ensuring these user-uploaded models:

Worked with Unity’s humanoid rig system, which is necessary for features like lip sync, avatar animations, and movement.
Maintained a consistent skeleton structure, despite coming from different sources.
Could be processed automatically without requiring manual rigging or correction by developers.

Additionally, users needed the ability to upload custom background images (PNG or JPEG) to personalize the environment in which their avatar appears.

Solution

To support user-uploaded models and backgrounds, I implemented the following:

Custom Model Upload System

Added support for FBX and GLB model uploads through the client interface.
Developed a custom mapping format that defined the required bone structure and naming conventions, ensuring compatibility with Unity’s humanoid rig.
Created guidelines and documentation for users to follow when preparing their models, reducing upload errors and increasing success rate.
Integrated automatic model validation and conversion, so compatible avatars could be imported, rigged, and animated without manual setup.

Background Image Support

Enabled users to upload PNG or JPEG images as virtual backgrounds.
Once uploaded, backgrounds were dynamically applied to the 3D scene on the client side, allowing users to personalize the assistant's environment.

Result

These additions allowed customers to bring their own avatars and environments into the application, significantly increasing personalization and engagement. Uploaded avatars retained full functionality—including lip sync and movement—thanks to the enforced humanoid rig structure, and background customization gave users more control over the assistant's look and feel.

Real-Time Audio-to-Avatar Mouth Movement

Challenge

A key part of delivering a believable AI assistant was ensuring that the avatar’s lip movements matched the spoken audio. However, different voice lines had varying speeds, phonemes, and pacing, which caused issues with lip sync timing.

Sometimes the mouth would move too fast or too slow, making the interaction feel unnatural.
The default animation curves didn’t adapt well to the variation in speech speed or audio complexity.

Solution

To improve synchronization, I enhanced the lip sync system with more accurate timing and phoneme mapping:

Dynamic Timing Adjustment

Analyzed the generated audio waveform and adjusted the animation playback speed to match the pace of the speech in real time.
This allowed the avatar’s mouth to move faster for quick speech, or slower for longer, drawn-out words—keeping the visual in sync with the audio.

Improved Facial Animation Ranges

Refined and expanded the facial rig and blendshapes to support a wider range of mouth shapes and expressions.
Adjusted animation curves and viseme mappings for more accurate mouth positions corresponding to speech sounds.

Speech-Aware Animation Blending

Introduced smoother transitions between visemes (mouth shapes for specific phonemes) to avoid robotic or choppy movement, resulting in more natural expressions.

Result

The improved system delivered much more accurate and realistic lip sync, enhancing the sense of presence and emotional engagement with the 3D assistant. Regardless of how fast or slow the generated audio was, the avatar’s facial movements stayed responsive and expressive, creating a more lifelike and immersive user experience.

Personalized Models with Unique Animations

Challenge

To make each AI assistant feel truly personalized, the system needed to support unique 3D avatars per customer, with customizations including:

Posture and body stance
Lip sync behavior and facial movement
Hairstyles and visual features
Idle animations and personality expressions

The key challenge was finding a scalable way to adjust animations and behavior per model without rewriting logic or duplicating animation controllers for each customer.

Solution

I developed a modular animation override system that made avatar customization flexible and efficient:

Override System for Animations

Built a system that allowed default animations (e.g., idle, talking, blinking) to be overridden per avatar, using a custom configuration layer.
This made it easy to adjust posture, idle styles, or facial expressions on a per-model basis, without needing separate controllers.

Support for Model-Specific Adjustments

Enabled fine-tuning of lip sync curves, mouth shapes, and facial timings to better match each avatar’s facial rig.
Allowed different hairstyles and accessories to be swapped and matched with specific customer avatars.
Integrated custom idle animations to reflect personality or tone (e.g., professional, casual, friendly).

Data-Driven Configuration

Used a data-driven setup where each customer’s avatar could be linked to a specific animation profile, making it easy to update or expand without manual rework.

Result

The system allowed each customer to have a visually and behaviorally unique assistant, improving engagement and brand identity. By supporting model-specific overrides, we created a scalable way to deliver** personalized 3D avatars** with distinct styles, expressions, and behaviors—without bloating the project with redundant assets or logic.

Case study

AI Foundation / AI.XYZ

AI.xyz is an innovative platform focused on helping users create personalized AI assistants designed to support them with personal and professional tasks.

Unity
URP
C#
CC4

Website Website

AI Foundation / AI.XYZ

Start of development: 2021
I am in project: 2023-2025

Contribution to project

Fix existing bugs
Enchance avatars
Custom models

Case impression

The project has an amazing idea of giving customers a personalized AI helpers

It's begun with resolving existing issues

My initial tasks were to fix existing bugs with project, fix crashes, optimize memory usage and add support for custom format

Pest controlDue to fixing bugs and crashesRead more

Silence Is GoldenDue to adding mute mechanic and interruption mechanic Read more

Be creative with you own modelsDue to adding ability to load custom model and backgroundRead more

Issues are gonna what next

Next steps were to improve workflow, improve lip sync, add more models

Read my lipsDue to improving the audio system and updating animationsRead more

Personalized experienceDue to adding customer specific modelsRead more

What will come next?

We staying in close touch with customer, so to ensure future cooperation in further improvements

All projectsLet’s talk