Season 7 · Article 04

Interactive Canvas Textures

Draw your UI into a 2D canvas. Map that canvas onto a 3D mesh. Raycast mouse events back to 2D coords and forward them to the canvas's event handlers. Result: HTML-feeling UI on a physical 3D surface.

1. The pattern

const uiCanvas = document.createElement('canvas');
uiCanvas.width = 1024; uiCanvas.height = 512;
const ctx = uiCanvas.getContext('2d');
drawUI(ctx);
const tex = new THREE.CanvasTexture(uiCanvas);

const screen = new THREE.Mesh(
  new THREE.PlaneGeometry(2, 1),
  new THREE.MeshBasicMaterial({ map: tex })
);

2. Event forwarding

renderer.domElement.addEventListener('pointermove', e => {
  const mouse = screenToNdc(e);
  ray.setFromCamera(mouse, camera);
  const hits = ray.intersectObject(screen);
  if (hits.length) {
    const uv = hits[0].uv;  // 0-1 in canvas space!
    const cx = uv.x * uiCanvas.width;
    const cy = (1 - uv.y) * uiCanvas.height;
    handleCanvasMove(cx, cy);
    tex.needsUpdate = true;
  }
});

The Three.js raycaster returns UV coordinates on the hit surface. That UV maps 1:1 to the canvas. Perfect event forwarding.

3. Live demo — a drawable canvas on a 3D surface

A plane in 3D. Hover → cursor indicator. Click + drag → paint on it.

4. React/Svelte as the canvas

// render React to an offscreen div
ReactDOM.render(<MyUI />, offscreenDiv);

// serialize DOM to canvas via html2canvas
import html2canvas from 'html2canvas';
const canvas = await html2canvas(offscreenDiv);
tex.image = canvas;
tex.needsUpdate = true;

Run on every React update. Expensive (~10-50ms). Throttle, debounce, or mirror DOM only on state changes.

5. Three.js has HTMLMesh (experimental)

import { HTMLMesh } from 'three/addons/interactive/HTMLMesh.js';
const mesh = new HTMLMesh(myDomElement);
mesh.position.set(0, 1, -2);
scene.add(mesh);

Auto-snapshots a DOM element into a texture. Updated on demand. Good for VR menus.

6. Keyboard / text input

Text input in canvas-texture UI is hard:

• Global key listener + software keyboard drawn to canvas, OR
• A hidden native <input> that receives focus on click, mirrors text to canvas.

For VR: use the browser's XR session with DOM layers (experimental).

7. Performance

• Texture size: 1024² is plenty. 2048² for fine text.
• Only call tex.needsUpdate = true after actually changing the canvas.
• For frequent updates, use a CanvasRenderingContext2D on an OffscreenCanvas + Worker.

8. Takeaways

• Canvas → texture → mesh: 5 lines.
• Raycast UV is your event forwarding superpower.
• html2canvas for React/DOM mirroring (throttled).
• HTMLMesh addon for quickest path.
• Text input: hidden native input OR virtual keyboard.