Procedural 3D Space Reconstruction in Claude Code: From Memory Notes to Interactive WorldsNote · digest for October 30
Architectural blueprint and 3D voxel projection of a virtual memory castle reconstructed using Claude Code.
Note2 sections≈ 4 min

Procedural 3D Space Reconstruction in Claude Code: From Memory Notes to Interactive Worlds

When developers fire up the terminal agent Claude Code, they usually ask for everyday chores: fix a failing unit test, rewrite a database migration, or tweak a Docker config. Researchers at Skill Leap AI decided to break that pattern and see how the agent handles an unusual challenge sitting right at the intersection of semantic analysis and procedural 3D graphics.

The experiment began with a personal knowledge base: a local folder of more than 400 unstructured text files. Those notes held years of memories, book takeaways, favorite movie lists, quotes, and rough ideas—a classic "second brain." Instead of generating a flat link graph or a standard search UI, they fed the directory to the agent with one goal: turn that personal digital archive into an interactive 3D world—a virtual "memory palace."

From Unstructured Text to Physical Rooms

Working autonomously, Claude Code walked the directory tree, parsed headings, tags, and semantic clusters, and engineered a client-side web app in Three.js and WebGL from scratch. The result was a full medieval castle you can explore in first-person view.

Inside the generated castle, rooms are organized strictly by theme:

  • Cinema Hall: Walls lined with interactive posters and cards for favorite films.
  • Memory Library: A massive bookcase packed with neat rows of volumes, each wired to a Raycaster—click a book with your mouse, and it slides smoothly off the shelf to display the underlying note summary.
  • Gallery of Impressions: Exhibits highlighting major life milestones and key projects.
# Launch the procedural memory generator via Claude Code CLI
claude-code "Просканируй папку ./vault/notes (400 markdown-файлов), кластеризуй по 5 ключевым темам. 
Сгенерируй index.html с Three.js сценой в виде замка: 
- 5 изолированных залов с порталами перехода;
- интерактивные 3D-книжные полки с привязкой Raycasting к тексту;
- контроллер управления от первого лица (PointerLockControls);
- адаптивное освещение и процедурные текстуры стен."

Virtual Hotel: Elevators, Procedural Terrain, and Fast Travel

In a second experiment, they raised the bar even higher: reconstructing a real wedding hotel and honeymoon grounds from scattered descriptions and text notes. Without outside help, Claude Code built a banquet hall, a floral wedding arch beside a virtual lake, guest wings, and a beach zone right in the browser.

What stood out was the depth of interactive mechanics. Rather than dropping static boxes, the agent coded working elevator logic: step into the cab, click a floor, and your character rides up to a balcony overlooking the lake. To navigate the sprawling grounds comfortably, Claude Code also added an interactive Fast Travel minimap that teleports the camera straight to any chosen spot.

// Excerpt from the generated Three.js interaction controller
import * as THREE from 'three';

export class MemoryCastleInteraction {
  constructor(camera, scene, memoryRegistry) {
    this.raycaster = new THREE.Raycaster();
    this.mouse = new THREE.Vector2();
    this.camera = camera;
    this.scene = scene;
    this.registry = memoryRegistry;
  }

  onDocumentPointerDown(event) {
    this.mouse.x = (event.clientX / window.innerWidth) * 2 - 1;
    this.mouse.y = -(event.clientY / window.innerHeight) * 2 + 1;
    this.raycaster.setFromCamera(this.mouse, this.camera);

    const intersects = this.raycaster.intersectObjects(this.scene.children, true);
    if (intersects.length > 0) {
      const target = intersects[0].object;
      if (target.userData && target.userData.memoryId) {
        this.displayMemoryCard(this.registry[target.userData.memoryId]);
      } else if (target.userData.action === 'elevator_floor_select') {
        this.animateElevator(target.userData.targetFloorY);
      }
    }
  }
}

This experiment proves that modern terminal coding agents are evolving into genuinely multidisciplinary builders—capable of weaving messy text archives, spatial geometry, and interactive 3D graphics into a finished product without a single line of hand-written code.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 2