1. Background
Stroke is a global health challenge and is the second leading cause of death and the third leading cause of disability worldwide (1). Despite treatment interventions, 30% to 66% of survivors experience arm dysfunction after 6 months (2 - 4). This disability severely affects patients’ independence and quality of life.
The current paradigm of upper limb rehabilitation faces 2 major challenges: operational limitations among professionals and motivational gaps among patients. In traditional approaches, monitoring treatment progress relies mainly on qualitative and subjective assessments, which, under high workloads, increases the risk of human error and hinders the collection of accurate biomechanical data (5, 6). This issue is particularly important in elbow joint rehabilitation, which requires accurate monitoring of ROM.
The repetitive nature of therapeutic exercises can also cause boredom, which may reduce patient motivation and treatment adherence. Gamification technologies can increase patient motivation and improve adherence by incorporating interactive elements such as scoring systems and challenge levels (7, 8).
To bridge gamification and physiotherapy, 2 main technological approaches have been developed: wearable sensor-based systems and noncontact vision-based systems. Wearable systems enable precise measurement of joint angles using sensors such as accelerometers and gyroscopes and have been reported to be effective in the rehabilitation of stroke patients (9). However, their high cost, requirement for complex calibration, constraints on natural movement, and potential skin discomfort have challenged their widespread adoption (10, 11).
In contrast, machine vision-based systems offer a noninvasive, low-cost approach that uses conventional RGB cameras, rather than red-green-blue-depth (RGB-D) systems, to monitor movement without wearable equipment (12, 13). However, challenges such as joint self-occlusion and depth ambiguity affect the accuracy of joint angle calculation (14). RGB-D systems offer higher accuracy, but their dependence on proprietary hardware has limited their widespread use (15).
Markerless pose estimation models based on deep learning, such as OpenPose (16), DeepPose (17), and MediaPipe Pose, enable the extraction of motion kinematics from simple RGB images and have attracted attention because of their scalability and hardware simplicity. MediaPipe Pose estimates joint angles with high accuracy (18, 19), but it faces challenges in distinguishing isolated movements from compensatory movements and in addressing perspective changes, and its accuracy decreases in the absence of depth information (18, 19).
2. Objectives
Accordingly, there is a need to develop a noninvasive, lightweight, machine vision-based framework that continuously calculates elbow ROM and corrects perspective errors using conventional RGB cameras. This study presents a proof-of-concept framework for elbow rehabilitation that integrates interactive gaming, markerless position estimation, and a proposed machine learning architecture designed to compensate for vision errors, while simultaneously increasing patient motivation and providing quantitative data that can be used for clinical monitoring. The main innovation of this study is the integration of gamification, markerless position estimation, and depth ambiguity error correction using machine learning within a single framework.
3. Methods
The proposed framework was designed at two distinct levels: (1) a systematic analysis layer for perspective effects and (2) a patient–system interaction layer.
3.1. Perspective Analysis Simulator
To systematically analyze the effect of camera angle changes on elbow angle calculation, a standalone web-based simulator was developed using HTML, CSS, and JavaScript. Its environment is shown in Figure 1. This simulator enables investigation of different scenarios of arm and forearm displacement relative to the camera without changing the joint angle. In this controlled environment, changes in the pixel lengths of the vectors and their ratios are analyzed, and the behavior of perspective-induced angular error is examined. The interactive visualization tool and its source code are available in a GitHub repository, with a permanent archive hosted on Zenodo.
Figure 1.
The effect of camera angle on the length of the projected arm and forearm vectors, illustrating simulated projection effects under body rotation. Increasing the body rotation angle toward the camera (body rotation) reduces the projection length of the forearm vector, whereas changing the arm elevation angle (arm elevation) affects the projection length of the arm vector.
The simulation results form the basis for designing the machine learning-based error correction model proposed in this framework. Given the nonlinear nature of perspective error, which exhibits a parabolic relationship with the rotation angle, a Random Forest Regressor was selected to estimate the error. To ensure a clear distinction between the system layers, this machine learning model is explicitly presented as a proposed theoretical component and was not trained or validated on empirical or clinical datasets in the current study. The model inputs include three features: (1) The ratio of the arm vector length to the base value, (2) the ratio of the forearm vector length to the base value, and (3) the ratio of the arm length to the forearm length. The model output is a correction value in degrees that is applied to the angle calculated by MediaPipe (18, 19). Although implementation and training of this model require a real dataset, and no data were collected for its training in the present work, its governing logic is based on the systematic analysis of perspective error behavior in the developed simulator. The main advantage of this proposed approach is its independence from depth reconstruction or complex 3D modeling, which facilitates future implementation in clinical applications.
To ensure reproducibility, the geometric simulation parameters and mathematical formulation are defined as follows:
3.1.1. Projection Modeling
Assuming an orthogonal camera model along the Z-axis, the baseline projected lengths of the arm (L1) and forearm (L2) equal their true lengths (l1 base = L1, l2base = L2). When subjected to body rotation and arm elevation, the 2D projected lengths (l1, l2) contract nonlinearly:
The MediaPipe-derived 2D angle is calculated using the dot product of these projected vectors. The angular error plotted in Figure 2 is defined as follows:
Figure 2.
A, Measurement error according to body rotation angle. The red line shows the raw error, and the green dashed line shows the theoretical error after ideal calculation-based correction. B, Changes in arm image length according to elevation angle. The red line shows the projection percentage, and the green dashed line shows projected baseline stability after ideal correction.
3.1.2. Simulation Ranges
To generate the specific curves in Figure 2, the simulator's operational parameters were evaluated across the full ranges of body rotation and arm elevation: body rotation θϵ[-90°×(+90)] and arm elevation φϵ[0°×(+180)].
3.1.3. Feature Engineering and Correction
Three normalized ratios were computed per frame as inputs for the proposed Random Forest Regressor:
The final corrected angle was then calculated by subtracting the predicted error:
3.2. Patient-System Interaction Layer
In the patient interaction environment, implemented using the Python, Pygame, MediaPipe, OpenCV, and Tkinter libraries, key upper-limb joints, including the shoulder, elbow, and wrist, were extracted using the MediaPipe markerless pose estimation algorithm (Figure 3).
Figure 3.
Points extracted by MediaPipe and the method used to calculate ROM, identifying the exact landmarks used for geometric tracking.
The framework is conceptually tailored for subacute stroke rehabilitation and requires basic cognitive capabilities to interact with the clinical dashboard and game parameters.
Due to the inherent limitations of RGB image-based systems in addressing perspective changes and joint displacement relative to the camera, an initial calibration protocol is defined at the beginning of each treatment session. In this step, the user places the elbow joint in a reference position, approximately perpendicular to the image plane, and the pixel lengths of the arm and forearm vectors, along with their ratios, are recorded as baseline parameters. These values serve as a geometric reference for analyzing subsequent changes. The 2D coordinates of these points in each frame are used as the basis for kinematic calculations, and the elbow angle is calculated as the angle between the arm vector (shoulder-elbow) and the forearm vector (elbow-wrist). This angle is used as the initial estimate of ROM (Figure 3).
Before the session begins, the clinician can set treatment parameters through a dashboard (Figure 4). These parameters include target ROM in degrees, target repetitions, and allowed speed. These parameters are affected by the height of and the distance between obstacles. The settings are applied to the game engine in real time and adjusted dynamically. This allows the clinician to customize the treatment protocol according to the patient's stage of rehabilitation.
Figure 4.
Adjustment of clinical parameters before the game begins by the physiotherapist, representing the clinician dashboard interface of the developed prototype.
4. Results
The results are presented in two complementary parts: first, system error behavior in response to situational changes and the role of machine learning-based correction, based on geometric simulation; and second, a demonstration of the practical implementation of the game-based rehabilitation framework.
4.1. System Error Behavior and Correction Analysis: Simulation-Based Findings
System error behavior in response to geometric changes was conceptually analyzed using the simulator. In the baseline case, before any correction was applied, the relationship between body rotation toward the camera and elbow angle measurement error was investigated. As shown in Figure 2, increasing the body rotation angle increased the measurement error nonlinearly, producing a convex trend. This behavior indicates the sensitivity of 2D image-based systems to perspective changes and to the approximation of vectors relative to the camera line of sight.
The effect of increasing the arm elevation angle on the projected forearm image on the image plane was also investigated. As shown in Figure 2, the relationship between arm elevation angle and changes in the projected length of the forearm is attributable to compression of the depth component of movement in the 2D image. The projected length is maximal when the arm is horizontal, at 0 and 180 degrees, and minimal when the arm is vertical, at 90 degrees. This phenomenon reduces the accuracy of elbow angle estimation even when the actual joint angle remains constant.
After application of the correction solution based on initial calibration and normalized ratios of vector lengths, system error behavior was re-examined. As shown in Figure 2, within a certain range of body rotation, the simulated error was theoretically controlled, suggesting that the mathematical framework can mitigate severe nonlinear deviations under ideal conditions. These conceptual results indicate that the proposed solution can define a more stable operating area for monitoring elbow ROM without explicit depth reconstruction or specialized hardware.
It is important to emphasize that the graphs presented in this section are derived from geometric simulations and do not represent experimental or clinical data. These results are presented to explain expected system behavior and the design logic of the proposed framework in response to the inherent challenges of 2D machine vision in rehabilitation applications.
4.2. Practical Implementation of the Game: Software-Based Framework Demonstration
As shown in Figure 5, the game character is a bird whose vertical motion is continuously mapped to the elbow angle, and environmental obstacles can be traversed based on the target ROM. This figure demonstrates the feasibility of practical implementation of the proposed framework and the integration of motion monitoring in a motivational environment.
Figure 5.
By changing the ROM of the elbow joint, the bird changes position and can pass through the obstacle, demonstrating implementation feasibility and system framework integration rather than clinical patient testing.
In accordance with the proof-of-concept design of the study, no clinical or human-subject evaluation was performed, and the results do not indicate measurement accuracy in real users.
5. Discussion
A key finding of this study is the demonstration of the inherent sensitivity of joint angle estimation in 2D image-based systems to changes in perspective. Simulation analyses showed that, even when the true joint angle is constant, body rotation or arm elevation can lead to nonlinear increases in measurement error. This finding is consistent with observations reported in previous RGB-based studies and is one of the main reasons for limited clinical confidence in this class of systems (15, 18, 20, 21).
To address this limitation, the proposed framework uses a lightweight empirical approach based on initial calibration and normalized ratios of vector lengths, rather than explicitly reconstructing depth or using complex 3D models (22). Conceptual simulation results indicate that this approach can prevent sharp error growth within a certain operating range and provide a more stable region for monitoring elbow ROM.
From a user interaction perspective, integrating continuous angle monitoring with an interactive gaming environment creates a direct link between the therapeutic goal and the game logic. Unlike many existing rehabilitation games based on discrete events or collision detection, progress in the present framework depends directly on achieving a specific ROM. This design could potentially prevent compensatory movements aimed solely at game success and keep the patient focused on movement quality (23-25).
Unlike many existing game-based systems in which the patient is left without expert supervision and simply plays an entertaining game, the clinician remains central to decision-making in the framework proposed in this article (26-28). Using a clinical dashboard (Figure 4), the rehabilitation specialist determines key parameters, such as target ROM, target repetitions, and allowable movement speed, before the start of the session, according to the patient's stage of treatment.
5.1. Study Limitations and Future Directions
Despite these advantages, important limitations should be considered. First, the presented results are based on geometric simulations and cannot be considered evidence of the actual accuracy of the system in clinical settings. Second, the present framework focuses only on the elbow joint and uniaxial movements; generalization to multiaxial joints will require further investigation.
This study demonstrates that, even without relying on specialized hardware or complex 3D modeling, a potentially useful framework for elbow joint rehabilitation can be developed through an informed design combining machine vision, gamification, and geometric analysis. However, this simulation-based proof-of-concept study lacks empirical validation. Future work will train the proposed Random Forest model on clinical datasets and benchmark its real-time accuracy against a gold-standard Vicon motion-capture system or inertial measurement unit sensors using root mean square error and Bland-Altman analysis. In addition, the current geometric simulation does not capture real-world environmental and physiological factors, such as variations in ambient lighting, variability in camera placement, or complex compensatory trunk and shoulder movements.
