Cubemaps and Texture Drawing System

Challenge

Storyboarder required support for cubemap backgrounds and a system for drawing directly onto textures, including background images, cubemaps, and image objects. The key challenges included:

Automating cubemap setup from image assets with varying layout patterns.
Allowing users to draw directly onto cubemap backgrounds, despite cubemaps not being physical objects in the scene.
Handling drawing interactions both on 3D objects and backgrounds in the Scene Graph (SG) while maintaining cross-module synchronization.
Managing real-time texture updates without overwriting original assets, ensuring changes propagate correctly through the app.

Solution

Cubemap Background System

Leveraged THREE.js's native cubemap support, which requires six individual images (one for each side of the cube).
Implemented an automatic pattern recognition system to correctly interpret various cubemap layout patterns (e.g., cross layouts, strip layouts).
- This system identifies and assigns each image region to the correct cubemap face, reducing manual setup and error risk.
Cubemap backgrounds are configured and applied through the World Settings panel in Storyboarder.

Drawing on Textures

Introduced a drawing system that allows users to paint on both image objects and background textures, including cubemaps (outside of XR mode).
Image Objects:
- Performed raycasting to detect intersections with image planes and retrieve their UV coordinates.
- Converted UVs to 2D canvas coordinates, allowing drawing operations on a CanvasTexture that updates the image in real time.
- Handled cases where the mouse leaves the object by implementing continuous drawing, extrapolating the stroke based on the last known direction and position if the mouse button remains pressed.
Cubemap Drawing:
- Addressed the challenge of cubemaps not being physical objects by creating a small invisible cube around the camera.
- Raycasted against this proxy cube to simulate interaction with the cubemap texture.
- Drawing inputs are then applied to the corresponding CanvasTexture, maintaining accurate spatial mapping between the view and the texture.
All drawing operations are centralized in the LargeSceneManager, maintaining a list of drawable textures and processing updates consistently across modules.

Texture Management and Update Workflow

When a drawing session is completed, a temporary image is generated from the updated canvas and passed to the React store.
- This ensures texture updates are reflected throughout the application without immediately overwriting the original assets.
Finalized textures are only committed and saved when the scene is saved, ensuring non-destructive editing and safe handling of texture assets.

Limitations Drawing on cubemaps and textures is not supported in XR mode due to technical limitations, but drawn elements remain visible when switching back to SG.

Result

The Cubemap and Texture Drawing System provides artists and storyboard creators with an intuitive, flexible tool to customize their environments directly within the app. By automating cubemap setup and offering real-time drawing capabilities, users can enhance their scenes with custom backgrounds and annotations—without leaving Storyboarder.