This writeup is a work in progress. I will be updating it with more details in the coming months.
Overview
RealityPaint is an interactive projection mapping system that allows users to paint with light on physical surfaces. Using an Xbox Kinect for depth sensing and a projector for display, the system maps virtual artwork onto real-world geometry in real-time.
This was created as part of HackUMass in November 2024, where we won third place overall.
How It Works
- Depth Mapping: The Kinect scans the environment to create a 3D model of physical surfaces
- Calibration: The projector and Kinect are calibrated to share the same coordinate space
- Interaction: Users can paint textures onto
- Projection: The artwork is projected onto the physical surfaces, conforming to their geometry
The result is artwork that appears to be painted directly onto walls, furniture, and other objects in the real world.
We found a better use case for the project was to allow for people to give presentations with a projector facing them, cropping them out of the projected image to prevent glare and the projection from appearing on their body.
Features
- Zero Glare Presentations: Crop the projected image to prevent glare and the projection from appearing on the body
- Gesture-Based Painting: Draw in the air with hand movements
- Surface Mapping: Artwork conforms to 3D geometry
- Real-Time Processing: Instant visual feedback
Technology
- Xbox Kinect: Depth sensing and skeletal tracking
- Projector: High-resolution display output
- Computer Vision: OpenCV for image processing
- 3D Mapping: Depth point cloud processing for surface detection