Home
04. DEVELOPMENT

Technical Implementation

Bringing HoldLight To Life System architecture, technical deployment, and our team's collaborative engineering efforts.

Squad / Individual Contributions

Chenyu Du

Chenyu Du

Student ID: 2360743
Junyan He

Junyan He

Student ID: 2363304
Jiafei Sun

Jiafei Sun

Student ID: 2361510
Youming Yang

Youming Yang

Student ID: 2361217
Individual contribution evidence: what each member built
Member Main Role Areas Code / System Contribution Design / Research / Content Contribution Testing / Documentation Evidence
Chenyu Du Chenyu Du
Frontend/Backend Computer Vision Route Planning Accessibility Deployment User Research Video Production Portfolio Production
  • Built the main React/Vite frontend and Express/MongoDB backend, including JWT login and user preference storage.
  • Implemented the wall-scanning workflow, Detectron2/TripletNet hold recognition, and route-planning algorithms for same-colour generation and recommended sequences.
  • Developed real-time wall alignment, audio speech guidance, manual hold correction, low-vision settings, and completed deployment configuration.
  • Led the evaluation design through questionnaires, blind and low-vision user interviews, and a data analysis framework.
  • Contributed to poster content and structure so the system work and research evidence were clearly presented.
  • Supported project video and portfolio production.
  • Used questionnaires, blind and low-vision user interviews, and analysis planning as evaluation evidence.
  • Supported the documentation of deployment, accessibility functions, and poster evidence drawn from the implemented system.
Junyan He Junyan He
UI/UX Design Inclusive Interface Portfolio Website Video Production Internal Testing Poster Assets
  • Built and structured the process portfolio website as the project evidence platform.
  • Translated the design direction into working portfolio page structure and presentation layouts.
  • Led visual iteration of the mobile prototype, including the user flow, route preview, and scanning layouts.
  • Focused on low-vision accessibility through high-contrast presentations, large-text layouts, and simplified interactions.
  • Collaborated on video, poster, and portfolio production.
  • Conducted internal testing and used the outcomes to support prototype and portfolio refinement.
  • Contributed to poster writing, visual evidence, and portfolio documentation.
Jiafei Sun Jiafei Sun
Human Pose Recognition Technology Evaluation Technical Testing Research Support Poster Visuals Video Production
  • Selected and adapted MediaPipe after comparing multiple pose-tracking frameworks for indoor climbing performance.
  • Connected pose-related outputs to the climbing scenario so the prototype could consider climber body movement and position during guidance.
  • Conducted comparative technology research on pose-tracking frameworks before implementation.
  • Refined questionnaires and evaluation materials, and contributed to poster visuals and video production.
  • Participated in technical testing of the pose component and internal system trials.
  • Supported evaluation material refinement and documented pose-related technical evidence for the project presentation.
Youming Yang Youming Yang
Market Research Evaluation Support User Testing Data Organization Documentation
  • Supported system refinement by identifying usability, accessibility, and interaction-flow issues during internal prototype testing.
  • Contributed design implications from user research data back into the prototype refinement process.
  • Investigated the problem space and conducted competitive analysis of climbing applications to identify the lack of accessible solutions.
  • Organized questionnaire and user research data, translating findings into actionable design implications.
  • Joined internal testing sessions to identify usability, accessibility, and interaction-flow issues.
  • Documented testing findings, project outcomes, and refinement decisions for the portfolio and poster.

Development Timeline

From Research Insight to Final Prototype

01

03.06Team Formation

The project team was created and the initial collaboration structure was confirmed.

This stage established the basis for user research, UX design, frontend development, backend logic, and evaluation work.

02

03.10Theme Idea Fixed

The team confirmed the core product direction: an accessible climbing support system for blind and low-vision users.

The design focus shifted toward route preview, brief voice guidance, explicit fallback, and inclusive interaction flow.

03

03.12BLV Interview & Division

The first BLV interview helped the team understand route-reading barriers, audio guidance needs, and uncertainty during climbing.

The team then divided responsibilities across frontend, backend, research, inclusive UX design, and testing preparation.

04

03.24First System Prototype

The first frontend-backend integration produced an initial working prototype with a connected scan-to-guidance flow.

This made it possible to test whether route preview, cue delivery, and fallback could work as one coherent user journey.

05

04.01BLV Pilot Testing

The first pilot test with a BLV participant showed that long voice instructions were difficult to process during climbing.

Iteration 1 simplified the audio wording into shorter, direction-first cues and reduced unnecessary guidance details.

06

04.10Team Testing

Internal team testing revealed that several interface areas were visually overloaded and not clear enough for quick use.

Iteration 2 simplified the interface and made the main scanning, preview, and guidance actions easier to identify.

07

04.20BLV Retest & Iteration 3

A second BLV test and interview exposed the need for faster hold correction during route setup.

Iteration 3 added multi-select controls for adding or removing detected climbing holds more efficiently.

08

05.06Final Version Optimised

The final version was polished through interface tuning, flow adjustment, and interaction cleanup.

The product was prepared as a more stable and presentable accessibility-focused climbing prototype for final submission.

High-Fi Prototype

High-Fi prototype overview showing the final HoldLight mobile interface screens.
Final Hi-Fi Overview. This overview shows the final mobile interface direction and the main HoldLight prototype screens.
User Can Do

Start from scanning

The main journey begins with route scanning rather than a dense dashboard, making the core task easier to find.

Requirement Supported

Preview + short guidance

The flow supports route preview, route correction, short guidance, and recovery controls.

Accessibility Value

Lower cognitive load

The final prototype reduces feature density and focuses on the moments that matter for blind and low-vision beginner climbers.

Core Feature Set

Feature 01

Scan & Route Preview

User problem: route shape is hard to access visually.

Implemented feature: camera scan, route confirmation, and simplified preview.

Requirement supported: confidence-building route preview.

Current limitation: recognition still needs confirmation in real gym conditions.

Feature 02

Short Audio Guidance

User problem: screen prompts are not practical while climbing.

Implemented feature: concise Web Speech API cue output with repeat and pause.

Requirement supported: low-overload audio guidance.

Current limitation: noisy gym environments may still reduce comprehension.

Feature 03

Accessibility & Fallback Controls

User problem: uncertainty can become unsafe or confusing.

Implemented feature: text size, contrast, repeat, pause, and request-help controls.

Requirement supported: safe fallback and recovery.

Current limitation: human support is still required for safety-critical decisions.

System Architecture and Data Flow

HoldLight is structured as a mobile-first web application that combines browser-side interaction with backend vision processing. The React + Vite frontend manages authentication, onboarding, dashboard, accessibility, speech, camera access, and the Climb Assist flow. Local browser capabilities such as getUserMedia, MediaPipe Pose, speech synthesis, audio, and haptics support real-time interaction on the user's device. Requests are routed through static hosting and an API proxy to a Node + Express backend, which handles users, notifications, climb scans, climb sessions, guidance logs, and vision requests. A Python vision runtime performs wall detection, hold grouping, color analysis, and planar calibration, while MongoDB stores user, scan, session, and guidance data. The system supports two main pipelines: wall scanning and live guidance.

HoldLight system architecture diagram showing users, React and Vite browser frontend, browser local capabilities, reverse proxy and static hosting, Node Express API, MongoDB, Python vision runtime, and two data pipelines for wall scan and live guidance.
System architecture of HoldLight, showing the frontend, browser local capabilities, static hosting, Node/Express backend, MongoDB storage, Python vision runtime, and the wall scan / live guidance pipelines.
Interaction State Evidence

Input, State and Feedback Loop

HoldLight handles interaction as a loop between user input, temporary frontend state, backend session records, and feedback. Scan actions, accessibility preferences, route confirmation, and climb controls update the current interaction state. These states then drive visual feedback, speech cues, fallback messages, and post-climb evidence such as repeated cues, corrections, and guidance logs.

01 Input Capture

User actions enter the system

Input is collected through scan/upload actions, accessibility preference controls, route confirmation, and in-climb controls such as pause, repeat, and help.

  • Scan wall / upload image
  • Select voice and accessibility preferences
  • Confirm route and start session
  • Pause, repeat, or ask for support
02 Frontend State

The interface tracks the current interaction

The browser frontend keeps temporary states for camera permission, scan progress, selected route, cue detail level, pause state, and the current guidance step.

  • Camera ready / scanning / scan complete
  • Selected route candidate
  • Brief or detailed cue mode
  • Current next-hold guidance step
03 Session Records

Confirmed actions become stored evidence

After confirmation, key states are sent through the Node/Express API and stored as climb scans, route candidates, climb sessions, guidance logs, and fallback events.

  • Confirmed scan result
  • Manual route correction
  • Repeated cue event
  • Fallback or low-confidence moment

Technical Notes / AI Logs

Tech Stack

Mobile web prototype

  • Browser frontend for mobile interaction and accessibility controls.
  • Node / Express API for backend coordination.
  • MongoDB for persistent app and session data.
  • Python vision runtime for camera-based recognition experiments.
  • Web Speech API for speech feedback and audio cues.
Verification + AI Logs

Setup and evidence trail

  • README / setup notes document how to run the prototype locally.
  • The repository includes /ai-logs for AI-assisted design and coding records.
  • Verification includes manual user-flow checks, responsive layout checks, and accessibility review.
  • AI-assisted suggestions were reviewed before adoption.