Inverse Kinematics (IK) System Enhancement

Challenge

Storyboarder integrates both 2D drawing tools and a 3D environment for scene composition. The 3D characters in Storyboarder rely on an Inverse Kinematics (IK) system to position joints realistically based on target locations. Initially, the project used the three-ik library, which has since been abandoned. This created several issues:

The library was outdated and lacked active maintenance.
It did not support custom constraints required for more complex rigging scenarios.
The original system expected characters to be oriented Z**-forward**, while Storyboarder's characters were Y-forward, requiring additional transformations for compatibility.

Solution

Forked the three-ik library and integrated it directly into the Storyboarder project, enabling dependency updates and custom modifications
Extended the IK system by adding support for custom constraints, including a Pole Constraint, which allows better control over joint bending directions. This was crucial for more accurate limb positioning in various modules such as Scene Graph (SG), XR, and AR.
Implemented a coordinate conversion system to bridge the difference between Storyboarder's Y-forward characters and three-ik's Z-forward requirements. A duplicate skeleton is maintained, rotated into Z-forward space for IK solving. After calculations, the results are applied back to the original skeleton using a change-of-basis formula.
Developed an IKHelper singleton to manage IK instances across different Storyboarder modules:

Separate instances were created for SG and XR to accommodate their differing methods of interacting with objects (e.g., local intersection control points in SG versus controller-attached intersections in XR). -The IKHelper also leverages InstancedMesh to efficiently render and manage IK control points.

Managed 10 IK control points for each character:

**4 pole constraints **(one for each limb)
6 end effectors (four for the limbs, plus the head and hips)

Result

The enhanced IK system provided more accurate and flexible joint control across Storyboarder's 3D modules, including Scene Graph, XR, and AR. Performance remained efficient, thanks to instancing and optimized update logic. Animators and storyboard creators gained greater precision and realism when posing characters within the application.

Drop Object / Character System

Challenge

Storyboarder needed a simple and intuitive** drop system** to simulate free fall for both objects and characters. The goal was to allow users to easily stack objects or drop them to the ground, ensuring proper alignment and collision without manual adjustments. This feature was built using Three.js but had to account for Storyboarder's Y-forward coordinate system, which added complexity to position calculations—especially when dealing with objects’ bounding boxes and character rigs.

Solution

Implemented two distinct drop systems:

Object Drop: Designed for static objects.
Character Drop: Tailored to handle skeletal meshes where bounding boxes don't reflect pose changes.

Three-Stage Object Drop Process

Stage 1: Calculate Lowest Point Determined the object's lowest point by calculating its bounding box minimum Y, while using the object's current X and Z position.
Stage 2: Raycast to Find Drop Target Performed a raycast from the lowest point directly downward to find the nearest object or ground surface to drop onto.
Stage 3: Position Adjustment with Coordinate System Translation Since Storyboarder operates in Y-forward space and bounding boxes are Z-forward, implemented a coordinate conversion. The new object position was computed by translating the lowest point into the correct space and applying the appropriate offset to ensure accurate placement.

Three-Stage Character Drop Process The character drop shared the same logic as object drop, with one key difference in Stage 1:

Stage 1 (Character-Specific): Calculate Lowest Bone Position Unlike static objects, character bounding boxes are not affected by skeleton animations. Therefore, used a BoxHelper to identify the lowest bone in the character's hierarchy. This bone became the reference point for raycasting and positioning, ensuring characters were placed correctly according to their pose.

Support for Stacking and Ground Placement The system allows users to drop either objects or characters onto other objects or the ground plane, streamlining scene assembly and improving workflow efficiency.

Result

The Drop Object/Character system provided users with an intuitive tool to quickly place and stack items in 3D space. It eliminated the need for manual adjustments, ensuring accurate positioning based on physics-like behavior. This feature is now seamlessly integrated into Storyboarder's Scene Graph workflows.

Localization System

Challenge

Storyboarder required a localization system that allowed the entire application to support multiple languages, with the ability for users or contributors to edit and share localizations easily. The system needed to handle both** React-based** modules, while ensuring language changes remained synchronized across the app's Electron windows (e.g., Scene Graph, Storyboarder, Welcome Window).

Solution

Implemented i18next as the core localization framework, providing a consistent API for key-based translations. Supporting libraries included:

react-i18next for seamless React component integration.
i18next-fs-backend for loading translation files directly from the filesystem.

Each language is stored in its own .json file, following a language-country code naming convention (e.g., en-US for English - United States).

Translation keys are referenced throughout the app via the i18n.t(key) function.
The system automatically loads the appropriate language file based on the user's current language setting.
If a key or language file is missing, the system gracefully falls back to the default language (determined by the system locale, or English as the final fallback).

Language state and changes are synchronized across multiple Electron windows using ipcRenderer communication. This ensures that any language updates (manual or system-driven) are reflected throughout the application in real time.
React Modules: Utilized the useTranslation hook from react-i18next. Components access the current language through const { t } = useTranslation(), and all display strings use t(key) for localization.
Developed a dedicated window that acts as a JSON editor, allowing users to add, edit, and manage translation files. This editor simplifies the process of creating new localizations and makes it easier to share language files with others.

Result

The localization system allows Storyboarder to support multiple languages across React modules while maintaining synchronization across all windows. The dedicated editor streamlines the translation process, empowering users and contributors to customize their language experience. The fallback mechanisms ensure reliable operation, even in cases of missing translations.

GPU Picker System

Challenge

In Storyboarder, interacting with 3D characters and other complex objects required an accurate picking system. However, THREE.js's default raycasting could not reliably detect intersections with skinned meshes, since skeleton deformations are computed on the GPU, and the CPU-side bounding volumes remain static (a known issue).

The previous solution involved manually defining and updating Axis-Aligned Bounding Boxes (AABBs) for each skeleton bone—an approach that was complex, inefficient, and error-prone. Additionally, there was a need to pick through transparent materials, which the default raycaster did not support.

Solution

Developed a GPU Picker system that performs object picking via offscreen rendering, bypassing the limitations of CPU-based raycasting in THREE.js.
Color-Based Object Identification Pass

For every pickable object in the scene, assigned a unique color ID and temporarily replaced its material with a solid color shader using that ID.
On mouse click (or controller interaction), the system renders the scene to an offscreen render target (texture), where each object appears in its unique color.
The mouse click position is translated to pixel coordinates, and the color of the pixel is read from the render target to determine which object was clicked, using the color ID as the lookup key.

Depth Pass for Precise Positioning

After selecting an object, the GPU Picker performs a second rendering pass to a depth texture, adding the selected object to a dedicated depth scene.
The depth pixel at the cursor position is read and unpacked (using THREE.js' depth-to-world-position conversion) to obtain the exact scene position where the pick occurred.

Transparent Material Support Because the system renders objects with flat color materials in the picking pass, it allows picking through transparent materials, which was not possible with the default raycaster.

Result

The GPU Picker system provides accurate, per-pixel picking, even on skinned meshes with complex GPU-driven deformations. It eliminates the need for manual bounding box management, improves performance and accuracy, and works consistently across all object types in Storyboarder. It also enhances user interaction, allowing precise selection of objects in complex scenes, including transparent elements.

Cubemaps and Texture Drawing System

Challenge

Storyboarder required support for cubemap backgrounds and a system for drawing directly onto textures, including background images, cubemaps, and image objects. The key challenges included:

Automating cubemap setup from image assets with varying layout patterns.
Allowing users to draw directly onto cubemap backgrounds, despite cubemaps not being physical objects in the scene.
Handling drawing interactions both on 3D objects and backgrounds in the Scene Graph (SG) while maintaining cross-module synchronization.
Managing real-time texture updates without overwriting original assets, ensuring changes propagate correctly through the app.

Solution

Cubemap Background System

Leveraged THREE.js's native cubemap support, which requires six individual images (one for each side of the cube).
Implemented an automatic pattern recognition system to correctly interpret various cubemap layout patterns (e.g., cross layouts, strip layouts).
- This system identifies and assigns each image region to the correct cubemap face, reducing manual setup and error risk.
Cubemap backgrounds are configured and applied through the World Settings panel in Storyboarder.

Drawing on Textures

Introduced a drawing system that allows users to paint on both image objects and background textures, including cubemaps (outside of XR mode).
Image Objects:
- Performed raycasting to detect intersections with image planes and retrieve their UV coordinates.
- Converted UVs to 2D canvas coordinates, allowing drawing operations on a CanvasTexture that updates the image in real time.
- Handled cases where the mouse leaves the object by implementing continuous drawing, extrapolating the stroke based on the last known direction and position if the mouse button remains pressed.
Cubemap Drawing:
- Addressed the challenge of cubemaps not being physical objects by creating a small invisible cube around the camera.
- Raycasted against this proxy cube to simulate interaction with the cubemap texture.
- Drawing inputs are then applied to the corresponding CanvasTexture, maintaining accurate spatial mapping between the view and the texture.
All drawing operations are centralized in the LargeSceneManager, maintaining a list of drawable textures and processing updates consistently across modules.

Texture Management and Update Workflow

When a drawing session is completed, a temporary image is generated from the updated canvas and passed to the React store.
- This ensures texture updates are reflected throughout the application without immediately overwriting the original assets.
Finalized textures are only committed and saved when the scene is saved, ensuring non-destructive editing and safe handling of texture assets.

Limitations Drawing on cubemaps and textures is not supported in XR mode due to technical limitations, but drawn elements remain visible when switching back to SG.

Result

The Cubemap and Texture Drawing System provides artists and storyboard creators with an intuitive, flexible tool to customize their environments directly within the app. By automating cubemap setup and offering real-time drawing capabilities, users can enhance their scenes with custom backgrounds and annotations—without leaving Storyboarder.

Attachables System

Challenge

Storyboarder needed a flexible system for character accessories, such as hats, glasses, or props, that would move, scale, and update seamlessly with the characters they’re attached to. The previous approach grouped characters and objects together, which worked for basic scenarios but introduced significant limitations:

Accessories required constant manual adjustments whenever a character’s pose or scale changed.
Grouped objects lacked a direct binding to specific bones or body parts, reducing flexibility.
Managing scaling differences (e.g., between adult and child characters) was cumbersome.

Solution

Independent Attachables System

Developed an Attachables system where accessories act as independent objects in the application’s data store.
Attachables track which character they’re attached to, but characters remain agnostic of their attachables.
- This approach reduces unnecessary dependencies and keeps data flow clean and modular.

Binding and Update Mechanism

Each attachable references a character ID, and the system retrieves all attachables linked to a character when the character is updated (e.g., on pose or scale changes).
Attachables maintain their relative scale to the character, ensuring consistency when switching between different body types (e.g., adult ↔ baby).
Because attachables are independent, they run an internal update cycle that checks for changes in their attached character and updates their position, rotation, and scale accordingly.

Built-In Attachables

Built-in attachables can be added by placing the accessory models in the directory:
Entries are defined in attachables.json, where each attachable can specify:
- bindBone: the bone it attaches to (e.g., head, hand).
- position (offset): its relative position/rotation/scale.
fine-tune positioning, developers can add a custom attachable, adjust it in the scene, and read its final transform values in the console for easy transfer to the JSON file.

Resrictions and Considerations

Any model can be used as an attachable except skinned mesh models, due to their complexity and the potential for conflicting skinning/animation systems.

Result

The Attachables system allows users to easily equip and manage props and accessories on their characters without manual adjustments. It supports dynamic scaling, bone attachment, and pose synchronization, improving usability and reducing setup time for complex scenes in Storyboarder

Emotions System

Challenge

Storyboarder needed a way to add 2D emotion overlays to characters, allowing users to quickly convey expressions without complex facial rigging or animation. The system had to be simple, efficient, and compatible with existing character customization workflows, similar to hair presets—meaning only one emotion image can be active on a character at a time. Key challenges included:

Precisely positioning the emotion image on the character’s face.
Ensuring emotion images fit correctly within the texture’s UV space, with minimal distortion or quality loss.
Providing a non-destructive, easy-to-use solution that integrates seamlessly with the existing character system.

Solution

Overlaying Emotion Images on Character Textures

Emotion images are drawn directly onto the character’s texture using an offscreen HTML canvas, which is converted into a CanvasTexture for use in the scene.
To determine the correct drawing position, a custom raycasting system was implemented:
- A midpoint is calculated between the eyes and neck bones of the character.
- This midpoint is moved forward by 2 units, and a ray is cast backwards toward the character to find the UV coordinate on the head texture.
- The raycasting logic is a simplified version of THREE.js raycasting, optimized for this specific use case.

Image Scaling and Size Management

A standard clamp size of 512x512 pixels was established as the maximum recommended resolution for emotion images.
- If the user provides an image larger than the clamp size, it is rescaled to fit within the 512x512 boundary, maintaining its aspect ratio to avoid distortion.
- Images smaller than the clamp size retain their original dimensions, allowing for rectangular or non-square emotion graphics when desired.
This ensures emotion images fit cleanly within the UV space, preventing overlap or stretching.

Non-Destructive Application

Emotion images are applied in a non-destructive way, similar to hair presets:
- Users can add, remove, or replace emotion overlays without permanently altering the underlying character texture.
- Only one emotion image can be active at a time, streamlining management and reducing visual clutter.

Result

The Emotions System gives users a quick and flexible way to add facial expressions to characters, enhancing storytelling and scene composition in Storyboarder. The system is simple to use, with minimal setup required, and integrates smoothly with existing character presets and texture workflows.

Hair Preset System

Challenge

Storyboarder needed a system to allow characters to wear different hairstyles, with the flexibility to add, remove, and swap hair models easily. While this could technically be done using the existing Attachables system, the design called for a streamlined approach where each character could only have one hair preset active at a time, mirroring the behavior of character presets rather than standard attachables.

Solution

Hair Presets as Specialized Attachables

Hair presets are implemented using the Attachables framework with a key distinction:
- The attachableType property is explicitly set to "hair" in the attachables.json configuration file.
This designation allows the system to treat hair presets differently, enforcing the one-at-a-time rule while reusing existing attachable infrastructure.

Exclusive Hair Selection Logic

When a user selects a new hair preset:
- If a previous hair attachable exists, it is automatically removed.
- The new hair model is instantiated as a new attachable and attached to the designated bind bone (typically the head).
(Future improvement consideration): Instead of removing and recreating hair attachables, the system could be optimized to switch models and update the position of an existing attachable—this would require extending the current attachable logic.

Custom Hair Preset Support

Users can add custom hair models by placing them in the appropriate directory and defining them in attachables.json with the attachableType set to "hair".
Once added, custom hair presets behave the same way as built-in ones, appearing in the UI and following the exclusive selection rule.

Result

The Hair Preset system allows users to quickly and easily customize a character’s hairstyle while ensuring consistency in the attachment logic and user experience. By building on the existing attachables framework, it maintains a modular design and simplifies asset management.

Camera Controls System

Challenge

The previous camera controls in Storyboarder were limited and time-consuming, making scene navigation difficult, especially in complex setups. Users could only:

Zoom in and out.
Move left, right, forward, and backward at a fixed height.
Rotate (pan) the camera around its center point.

These limitations forced users to rely on manual adjustments, such as changing the camera’s Z value in the properties panel, just to move vertically or reposition behind an object. The workflow was inefficient and unintuitive for dynamic scene building.

Solution

The new Camera Controls system introduced multiple features aimed at streamlining navigation, improving precision, and enabling intuitive movement within the scene. These new controls include:

Orbiting
Dollying
Pedestal & Truck
Dolly Zoom

Features and Logic

Orbiting

Enables the camera to rotate around a focused object.
- If a focused object exists (the last selected object), the camera orbits around it while maintaining the object’s relative screen position (e.g., if the object starts on the left side, it stays there during orbit).
- If no object is selected, the camera orbits around a point 7 meters in front of its current position.
Achieved by using a change of basis formula to preserve the relative orientation between the camera and the focused object.
- This creates a smooth, consistent orbiting effect that helps users maintain context while rotating the scene.

Dollying

Moves the camera forward or backward in the direction it is facing.
Unlike zooming (which adjusts the camera’s field of view (FOV)), dollying translates the camera itself, offering a more natural and physical movement through the scene.

Pedestal & Truck

Pedestal: Moves the camera vertically (up and down).
Truck: Moves the camera horizontally (left and right).
These movements provide finer control for repositioning the camera without rotating it.

Dolly Zoom (Vertigo Effect)

Performs a dolly movement toward or away from the selected object while adjusting the FOV to maintain the same frame height.
Creates a cinematic effect where the subject stays the same size on screen, but the background scales dramatically (popularized by Hitchcock's Vertigo).
Implemented by calculating the initial frame height, then adjusting the FOV in real-time as the camera moves to ensure that height remains constant.

Result

The updated Camera Controls greatly enhance user navigation and control within Storyboarder. Users can now quickly reposition, frame scenes with precision, and explore the 3D space more intuitively. These improvements speed up scene composition and provide a more natural, cinematic feel to camera manipulation.

Hand Pose Preset System

Challenge

Storyboarder needed a way for users to quickly pose character hands without manually adjusting each finger joint. While character presets and pose presets existed, they didn’t offer fine-grained control over individual hand gestures. A hand pose preset system was required to:

Allow fast switching between common hand poses (grips, gestures, etc.).
Provide flexibility to create and remove custom presets.
Support selective application to either hand, including proper mirroring between left and right hands.

Solution

Hand Pose Preset System

Introduced hand pose presets, functioning similarly to other preset systems (e.g., hair, emotions).
Users can create, remove, and apply hand pose presets via the UI.
Presets can be applied to either the left or right hand, regardless of the hand they were originally created for.

Priority Handling in Preset Application

Hand pose presets have a secondary priority, meaning that:
- A character preset or a full-body pose preset will override any existing hand pose preset when applied.
Hand poses are designed to be independent, giving users the flexibility to pose hands separately from the rest of the body.

Mirroring Logic for Cross-Hand Application

Hand pose presets can be saved for either hand and applied to either hand, requiring mirroring when switching sides.
Mirroring is achieved by inverting the quaternion’s X and W components (multiplying them by -1), which effectively mirrors the rotation across the hand's axis.

Removable Custom Presets

Users can remove custom hand pose presets through the UI.
This is handled by a generic wrapper around the preset system, allowing removal functionality across any preset list (e.g., hands, hair, poses).

Result

The Hand Pose Preset system provides an efficient and intuitive way to manage hand gestures, saving users significant time when posing characters in both Scene Graph (SG) and XR modes. It integrates seamlessly with Storyboarder’s other preset systems and offers flexibility for creating custom poses and managing mirrored hand gestures.

Pose Mirroring System

Challenge

Storyboarder needed a simple and effective way to mirror a character's pose, swapping the left and right sides of the character’s body. This feature improves workflow efficiency by allowing users to quickly create symmetrical poses or reverse existing ones without manual adjustments.

The challenge was ensuring the mirrored pose was accurate, consistent, and non-destructive, while maintaining compatibility with the existing pose preset system.

Solution

Quaternion-Based Bone Mirroring

The core of the mirroring process involves inverting bone rotations:
- For each bone in the character’s skeleton, its quaternion X and W components are multiplied by -1.
- This effectively mirrors the rotation of each bone across the relevant axis, flipping the pose from left to right (or vice versa).

Bone Renaming and Mapping

After mirroring the bone rotations, the system renames bones to reflect the side switch:
- Bones originally named LeftHand are renamed to RightHand, and vice versa.
- This ensures the mirrored pose maintains consistent naming conventions and is properly reflected in the system.

State Update and Store Synchronization

Once bone rotations are mirrored and bone names updated, the store is updated with the new pose data.
This ensures the mirrored pose is immediately reflected in the UI and available for further editing or saving as a pose preset.

Result

The Pose Mirroring feature allows users to instantly reverse a character’s pose, saving time and effort when working with symmetrical designs or quickly exploring alternate compositions. It integrates seamlessly with the Pose Preset Inspector, maintaining a smooth and intuitive workflow.

BoneHelper System

Challenge

Storyboarder required a visualization tool to represent a character's bone structure, helping users understand and interact with skeletal rigs—especially useful in XR and complex pose editing scenarios.

The previous version of the BoneHelper created an individual mesh for every bone, which caused significant performance issues, particularly in XR environments where rendering large numbers of objects became a bottleneck.

Solution

Singleton BoneHelper Class

Refactored the BoneHelper into a singleton class to ensure only one instance manages all bone visualizations in the scene.
This design reduces memory usage, ensures consistent control, and prevents multiple instances from conflicting or duplicating effort.

InstancedMesh Optimization

Rebuilt the bone rendering system to utilize THREE.js InstancedMesh, grouping all bone meshes into a single draw call.
This drastically improves rendering performance, especially in XR, by minimizing the number of individual mesh objects that need to be processed by the GPU.
The HelperBonesPool dynamically manages the number of bones shown, adapting to the specific character or skinned mesh being visualized.

Intersection and Interaction Handling

InstancedMeshes in THREE.js share a single geometry, which limits their ability to handle per-instance interaction and raycasting.
To resolve this, implemented invisible bone meshes that are separate from the instanced visual bones. These invisible meshes handle raycasting and intersection detection, enabling accurate user interactions with individual bones.

Duplicate Initialization Prevention

Added a UUID-based caching mechanism that tracks which SkinnedMeshes have already been initialized.
Prevents redundant initialization of skeletons and ensures the BoneHelper only processes each unique skeleton once, improving stability and efficiency.

Result

The optimized BoneHelper system allows Storyboarder users to visualize and interact with character skeletons in an efficient and scalable way. Performance in both Scene Graph (SG) and XR contexts is dramatically improved, enabling smooth interaction even with complex rigs and high bone counts.

Shot Explorer System

Challenge

Storyboarder needed a tool to automatically generate camera shots based on standard shot sizes and camera angle mechanics. This feature, known as Shot Explorer, allows users to quickly explore different framing options without manually repositioning the camera for every shot. The system needed to:

Randomly generate different shot compositions while maintaining cinematic rules.
Be implemented as a separate Electron window that communicates with Scene Graph (SG).
Support action batching and delayed execution, so that scene changes made while Shot Explorer is closed are applied when it’s reopened.

Solution

Shot Explorer as a Separate Electron Window

Shot Explorer operates as an independent window, allowing users to experiment with shots without disrupting the main scene.
SG’s React middleware intercepts all relevant actions and forwards them to Shot Explorer.
If Shot Explorer is not active, actions are queued until it is reopened, ensuring no data loss.

Random Shot Generation with Cinematic Rules

Each generated shot consists of a randomly chosen camera angle and shot size.
After the base parameters are set, additional rules are applied, each with a probability of execution.
The rules include: a. Orbiting (100% chance)
Moves the camera around the character with a weighted chance of choosing different positions:
- Front (40%)
- Left (25%)
- Right (25%)
- Back (10%)
After selecting a side, the system randomly picks an angle within that region. b. Roll (10% chance)
Applies a camera roll between 5° and 35°, tilting the frame slightly for a dynamic effect. c. Vertical One Third (70% chance)
Adjusts the camera angle so that the character stays in the left or right third of the frame, depending on their facing direction. (Insert reference image here) d. Horizontal One Third (100% chance)
Ensures that the character’s head remains in the upper third of the camera frame for proper composition. (Insert reference image here) e. Area Shot (New Feature in shot-explorer-new-shots Branch)
Introduces additional refinements for shot selection, improving framing and balance.

Action Batching and Performance Optimization

Implemented proper action batching in Shot Explorer, reducing redundant calculations and improving generation speed.
This enhancement was developed in the shot-explorer-new-shots branch, refining how shots are calculated and applied.

Result

The Shot Explorer tool provides an efficient way to generate cinematically valid camera angles, saving users time and effort while composing scenes. By integrating probabilistic shot rules, the system ensures that generated shots feel natural and well-framed, following established film composition techniques.

Pose Capture System

Challenge

Storyboarder XR needed a way for users to quickly and intuitively pose characters, especially for hands and head positioning, in a realistic and natural way. Manually adjusting these body parts through traditional controls was time-consuming and often led to less natural results. The goal was to implement a system where users could temporarily take direct control of a character’s head and hands using XR controllers, enabling more lifelike and accurate poses, with minimal setup and an intuitive workflow.

Solution

Temporary Pose Capture Mechanic

The Pose Capture system allows users to temporarily control a character’s head and hands in XR by teleporting the XR camera to the character’s head position and attaching control points to the XR controllers.
The session lasts for 5 seconds, giving the user enough time to adjust the pose without lingering control conflicts.

Camera Teleportation to Head Position

Utilizes a standard XR teleportation function to reposition the camera to align with the character’s head.
The teleportation ensures the user's view is aligned with the character, making it easier to adjust head positioning naturally.

Controller Attachment and Orientation Management

The control points for the character’s hands are attached to the XR controllers.
To maintain realistic joint movement while connected to the controllers, a relative angle formula is used:
- This preserves the current object rotations while also allowing the hands to follow controller movements accurately.
Manual calibration ensures the controllers are consistently oriented to face forward, aligning with the character’s natural pose expectations.

Mirror Object Feedback

A mirror object is included in the scene—a simple object with a camera that renders the user’s view to a RenderTarget. -This texture is applied as a material, effectively showing the user a third-person perspective of their adjustments in real time, enhancing spatial awareness during pose capture.

Auditory Feedback: Ticking Sound

A ticking sound plays during the pose capture session.
This provides audio feedback, helping users keep track of time and know when the 5-second window is about to end.

Result

The Pose Capture system makes it significantly faster and more intuitive to create realistic character poses in XR, particularly for head and hand positioning. The direct control via XR controllers ensures naturalistic adjustments, while the **time-limited window **prevents conflicts with other control systems and encourages efficient posing.

All projects

Case study

Storyboarder

The goal is to create an app which allows to you create a story by images, or 3d models.

Electron
Three.js
react
WebXR
js

Storyboarder Github Youtube

Start of development:: 2016
I am in the project:: 2018-2021

Contribution to project

Implement inverse kinematic
Improved 3D editing tool

Make it more user friendly

A client asked to add feature to help user create story faster so as result

Make changing character pose simplier Due to adding an IKRead more

Faster landingDue to creating a simple logic of landing selected object to nearest flat surfaceRead more

Overcoming language barrierDue to adding localizationRead more

Picking through transperancy Due to implementation of GPU pickerRead more

Drawing on background, really?Due to adding drawing ability to cube mapsRead more

Make character fancyDue to allowing adding accessories to charactersRead more

Do they have feelings...Due to adding custom emotion expression to characterRead more

A barber experienceDue to adding ability to change hairstylesRead more

Camera like in moviesDue to implementing an advanced camera control featuresRead more

Control you handsDue to implementing a hand pose presetsRead more

Loook in to mirrorDue to mirroring you pose completlyRead more

X-ray for bonesDue to creating helper to visualize bonesRead more

Amazing shots with one clickDue to implementing a shot-explorerRead more

Assume control of characterDue to attaching camera to character head and controllers to character handsRead more

And my contribution to the project has ended

I decided to persue game developer career so shamefully i need to say goodbye to this project

All projectsLet’s talk