Grove - A gamified mobile App that helps neuro-impaired patients self-monitor upper-limb dysfunction in an accessible, engaging, and fun way

My Roles -

🎨 End-to-end App Design
💼 Project Initiator & Team Leader
🔍 Literature & User Research

🎨 End-to-end App Design
💼 Project Initiator & Team Leader
🔍 Literature & User Research

For -
With -

3 Engineers,
1 HCI Professor,
Healthcare Experts

Duration -

Aug. 2023 - Now

Under Implementation

Overview.

iOS App currently under implementation

Patients with upper-limb dysfunction induced by neurological disorders now rely on clinic visits to assess their upper-limb functions, which is inconvenient and hinders a transparent view of their recovery progress. Thus, our team in Georgia Tech Contextual Computing Group designed Grove, a mobile app that enable patients to self-assess upper-limb function conveniently, gain insights into their recovery progress, and stay motivated in long-term self-monitoring.

The iOS App is currently in the exciting implementation phase! 🎉

My role.

I led the user-centered, design & tech bidirectional exploration

As project leader, I led a cross-functional team to research user requirements and explore a suitable technical solution, closely collaborating with healthcare experts and patients. As the sole designer, I conceptualized and designed the mobile app in an iterative process, integrating stakeholder feedback from co-design and testing sessions.

Key abilities:

Design Leadership

Multi-Stakeholders Management

Design for Behavior Change

Game Design

01. Background & Problem Context

01. Background & Problem Context

01. Background & Problem Context

Background.

Patients with neurological upper-limb dysfunction can now self-rehab at home with the VibroTactile Stimulation Gloves

Patients with neurological disorders like Multiple Sclerosis (MS) and Stroke, often suffer from upper-limb dysfunction. To help these patients recover their upper-limb function in a more low-cost and accessible way, the VibroTactile Stimulation (VTS) gloves were developed by our Georgia Tech's Contextual Computing Lab, to enable patients self-rehabilitate their upper-limb function at home, reducing the overhead of clinic visits.

Problem.

While continuous upper-limb function monitoring is valuable, inaccessible assessments and lack of engagement make it challenging for patients

Healthcare experts emphasize the importance of continuous monitoring of upper-limb recovery progress as it enhances their visibility into patients' conditions and improves patients' self-awareness of their recovery journey.

Yet, patients must still visit clinics in person to assess their upper-limb recovery progress. These clinic-centric assessments are inaccessible, especially for underserved communities, and impede timely and transparent tracking of recovery. Moreover, given that the recovery journey spans months or even years, patients often lack engagement in long-term symptom self-monitoring.

Product Goal.

How might we make upper-limb self-monitoring more convenient and accessible for patients while promoting sustained engagement throughout their self-recovery progress?

02. Medical & User Requirements Research

02. Medical & User Requirements Research

02. Medical & User Requirements Research

Stakeholder Interviews.

Investigate the potential & requirements for transforming clinic-based assessments into more convenient self-assessments

Our secondary research suggested the potential for at home, self-assessments for upper-limb functionality. To further investigate such potential, I designed and co-led multi-stakeholder interviews to gain comprehensive perspectives about:

  1. Requirements, existing practices, and challenges in patients’ self-monitoring of upper-limb dysfunction induced by neurological disorders like MS and Stroke

  2. Ideal technical methods that support feasible and accessible at home, self-assessment.

  3. Potential strategies to encourage patients to stay engaged in long-term self-monitoring.

Key Finding 1.

3 Key challenges in sustained self-monitoring of upper-limb dysfunction

Stakeholder interviews and secondary research suggested 3 major barriers to at home, self-monitoring of upper-limb dysfunction:

1.

1.

Inaccessible clinic-centric assessments

Current assessments for upper-extremity function are clinic-centric, relying on clinicians' experience-based diagnoses. However, patients with limited access to clinical resources face challenges in timely tracking their recovery progress.

2.

2.

No effective tools to precisely self-track symptoms

Due to the subtle and gradual nature of symptom changes, patients find it challenging to accurately track changes in their upper limb functionality at home without effective tools.

3.

3.

Lack motivation & adherence in self-monitoring

Maintaining long-term self-monitoring is crucial. Yet, patients frequently struggle due to a lack of awareness regarding its value or simply forgetting to engage in the practice.

Key Finding 2.

Promising to adapt experience-based clinical assessments into more accessible, objective self-assessments using the mobile device

We learned that current clinical assessments for upper-limb function primarily focus on measuring the range of motion for various upper-limb joints (Fig. 1-5), as well as motor coordination, accuracy, and speed through activities like finger tapping (Fig. 6) and the finger-to-nose test (Fig. 7).

Clinic Assessments for Upper-Limb Dysfunction Induced by Neurological Disorders

While all these assessments necessitate the involvement of a healthcare expert for diagnosis, experts strongly support the concept of converting clinical assessments into phone-based self-assessments using phone cameras and sensors for upper-limb movement detection, as they would be more accessible and quantifiable.

Key Finding 3.

Combining objective self-assessments with self-reporting subjective symptoms provides a more comprehensive view

In addition to assessing upper-limb functionality objectively, domain experts emphasized patients' self-reporting of subjective symptoms like behavior, mental health, and quality of life, for both patients and doctors to gain a comprehensive view.

03. User-centered Technical Solution Design

03. User-centered Technical Solution Design

03. User-centered Technical Solution Design

Technique Exploration.

What techniques would be suitable for patients' upper-limb function self-assessment?

Developers in our research team have developed various techniques for assessing different facets of upper-limb functionality. I designed and led the expert interviews to evaluate these techniques based on their usefulness, feasibility, and accessibility.

Technical solution.

💡 Utilizing Camera-based motion capture technique would be effective, accessible, and feasible!

Expert interviews suggested that the hands-free nature of camera-based computer vision techniques makes them more accessible for patients with upper-limb impairments to complete self-assessments. The sensor-based solutions, however, are either not feasible or not useful for our target users.

To prove the validity of CV-based method for upper-limb self-assessment, we tested with other body movements involved in clinic tests using MediaPipe, which was proved to be feasible and accurate.

04. Core Design Concept Building & Validation

04. Core Design Concept Building & Validation

04. Core Design Concept Building & Validation

Design Implications.

💡 A mobile app using motion capture technique to help patients conveniently self-assess upper-limb function at home while fostering motivation and sustained commitment

After synthesizing our research findings, I proposed the overall product concept and the following design requirements:

1.

Self-Awareness

Help patients gain a clear view of their subtle and gradual changes in symptoms and improve their self-awareness in their journey of recovery.

2.

Accessibility

Enable patients to assess their upper-limb functionality in an accessible and easy way at home, instead of visiting the clinics.

3.

Motivation & Commitment

Motivate patients to stay motivated and committed to symptom self-monitoring, particularly regarding their upper-limb functionality.

brainstorming.

Brainstorming App features to satisfy design requirements in accessibility, visibility, motivation & commitment

To come up with solutions that well satisfy design requirements while taking medical/technical/user experience requirements into consideration, I organized the brainstorming session that involved our multi-functional research team (including healthcare experts, AI experts, designers). We brainstormed product features, categorized and evaluated based on their usefulness, impact, urgency, and technical feasibility.

core features.

💡 We've identified 6 core features to satisfy design requirements, including the innovative gamified assessment feature

Through more literature reiew and discussions, we finally landed the following pivotal features that better align with our design requirements. The gamified assessments idea were highlighted for its potential to not only improve test accessibility, but also quantify subjective evaluation through game scores, and boost patients' motivation and engagement in self-assessment.

Concept Testing.

Illustrating core features with wireframes to solicit stakeholder feedback

I illustrated the core concepts we have, and created the games based on the theme “treasure hunting”. Then, we showed healthcare experts and HCI experts working in relavant fields to seek their feedback (shown in the following graph).

Testing Results.

🥳 Our core features were all highly approved!

Overall speaking, experts agree that all the core concepts we have well satisfy user requirements. Particularly, they highlight that the gamification components and the consistent reward system are great ideas for motivating patients to record their symptoms.

Next step.

💡 Extending the gamified assessment concept to cover comprehensive assessments shows promise!

Domain experts think it's highly promising for us to adapt clinical assessments involving other joint movements, e.g., shoulder, elbow, wrist into such motion capture games, so that patients can rely on the mobile device for comprehensive self-assessments, which made me super excited to further explore the game design in the next design phase.

05. Iterative Game Design

05. Iterative Game Design

05. Iterative Game Design

Design Challenge1.

Q1: What would make an effective, engaging, and cohesive game setting?

Literature review on "gamified design" and "narrative as a motivator" suggested the importance of a cohesive game theme and narrative for effectively promoting users' motivation & engagement, as users may find it meaningless and confusing to play random games. Thus, I proposed key design considerations to ensure that the game narrative/theme is coherent and it meets our design goals:

Game Setting Ideation.

Co-designing the game theme based on design requirements

I led brainstorming sessions to engage our multifunctional team in generating diverse design concepts aligned with our considerations. These sessions employed whiteboards, quick sketches, and iterative discussions to explore, synthesize, and refine ideas.

Game Setting Redesign.

💡 Farming is chosen as the ideal theme due to its diverse array of physical activities and tangible outcomes for real-time feedback

Following evaluation, my concept of farm-themed gamified tests received team approval since farm activities:

  1. Have enough varieties to cover all necessary assessments comprehensively.

  2. Can effectively mirror upper-limb movements

  3. Have tangible and measurable outcome to foster user motivation and engagement.

Additionally, I changed the game layout from horizontal orientation to landscape view, which provides a wider space for motion detection and a more immersive game experience. I sketched out the game concepts based on this theme.

Design Challenge2.

Q2: How to ensure the game meets both medical requirements and user experience needs?

Another challenge I faced was to ensure that the game meet medical needs and satisfy user experience. I asked myself the following questions while establishing the game settings:

  1. Do they intuitively mirror the body movements in upper-limb assessments?

  2. Can they accurately capture measurements needed to reflect patients' symptoms?

  3. Can patients easily understand them and feel engaged in the game?

💡 Build a unique design approach that integrates logical reasoning and emotional expressions

Thus, I created a unique design procedure that integrated multifaceted design considerations. Besides, I facilitated quick feedback sessions across the multifunctional team to arrive at the ultimate game concept for each assessment.

👌 Let's dive into the finger tapping coordination test as an example!

06. Mid-fi Prototyping for end-to-end user journey

06. Mid-fi Prototyping for end-to-end user journey

06. Mid-fi Prototyping for end-to-end user journey

Mid-fi Prototypes.

Refine the design concepts into mid-fi prototypes

Task Personalization & Tracking

Personalize the tasks based on user profile while providing users with the flexibility to customize tasks; Clearly visualize the progress to foster commitment.

Personalize the tasks based on user profile while providing users with the flexibility to customize tasks; Clearly visualize the progress to foster commitment.

Personalize the tasks based on user profile while providing users with the flexibility to customize tasks; Clearly visualize the progress to foster commitment.

Tests for evaluating the range of motion

Evaluate the range of motion for different joints with suitable game settings that intuitively mirror the body movements.

Evaluate the range of motion for different joints with suitable game settings that intuitively mirror the body movements.

Evaluate the range of motion for different joints with suitable game settings that intuitively mirror the body movements.

Tests for evaluating brain and upper limb coordination

Adapt the commonly used clinic test “finger-to-nose” test into a “bird catching” game.

Adapt the commonly used clinic test “finger-to-nose” test into a “bird catching” game.

Adapt the commonly used clinic test “finger-to-nose” test into a “bird catching” game.

Tests for evaluating finger coordination, dexterity, and tapping speed

Adapt the finger tapping tests into “pest catching” game to turn subjective measurements of finger coordination, tapping accuracy into quantifiable measurements.

Adapt the finger tapping tests into “pest catching” game to turn subjective measurements of finger coordination, tapping accuracy into quantifiable measurements.

Adapt the finger tapping tests into “pest catching” game to turn subjective measurements of finger coordination, tapping accuracy into quantifiable measurements.

Longitudinal recovery progress visualization to promote self-awareness

Visualize the longitudinal recovery progress for self-reflection to enhance patients’ self-awareness of their recovery journey

Visualize the longitudinal recovery progress for self-reflection to enhance patients’ self-awareness of their recovery journey

Visualize the longitudinal recovery progress for self-reflection to enhance patients’ self-awareness of their recovery journey

07. Design Evaluation & Iterations

07. Design Evaluation & Iterations

07. Design Evaluation & Iterations

Mid-Fi User Testing.

Qualitative & Quantitative evaluation with multiple stakeholders

I collaborated with another user researcher to design the user testing session. We showed stakeholders the prototypes to seek their feedback. Specifically, our mid-fi user testing has the following goals that match design goals regarding visibility, accessibility, motivation&commitment.

Testing Results.

🥳 Our design received high ratings across all success metrics

I continued to iterate the mid-fi design to high-fi after synthesizing the feedback.

Now, let's jump into the major design iterations I made!

Accessibility.

💡 Enhance the accessibility of self-test games with a more user-friendly guide and improved visual accessibility

Self-Awareness.

💡 Promote patients' self-awareness by refining more intuitive and insightful data visualizations

Motivation & Commitment.

💡 Boost patients' motivation by creating a more transparent and appealing reward system

08. Final Solution

08. Final Solution

08. Final Solution

Final Design.

Grove: Monitor your self-recovery progress with accessible and engaging motion games on your phone

By employing phone-based gamified tests with motion capture, Grove simplifies at-home self-testing, making it accessible and engaging. The gamification component, alongside clear task tracking and insightful self-reflection, fosters lasting motivation and commitment to self-monitoring.

Personalize Self-Monitoring Plan

Generate personalized self-monitoring tasks based on the patient's profile, while enabling them to customize according to their own preferences to help them establish scientific and also feasible self-monitoring routine.

Profile Setting

Generated Tasks Walkthrough

Home: Weekly Tasks

Virtual Farmland Building

Track Self-monitoring Tasks

Clearly visualize the weekly tasks on the homepage to help patients track their current progress and display the virtual farmland to visualize the virtual agriculture rewarded for timely task completion, encouraging patients to stick to the self-monitoring schedule.

Play Motion Games for Upper-Limb Assessments

The patient can self-assess their upper-limb function conveniently by playing motion games adapted from clinic assessments. The game results help patients easily understand their recovery progress. Moreover, the farm-themed games provide a sense of achievement, making patients more motivated.

Motion Games Overview

Arm Selection

Pre-Game: User Guide

During-Game: Farming Activity Simulation

*All game videos are for conceptual demonstration purposes and may not fully depict the actual effects

During-Game: Farming Activity Simulation

*All game videos are for conceptual demonstration purposes and may not fully depict the actual effects

After-Game: Test Results

All Upper-limb Self-Assessment Games

⏯ Hover to play/pause the video or enter the full screen. Want to click through the game yourself? Jump to Figma Prototype

⏯ Hover to play/pause the video or enter the full screen. Want to click through the game yourself? Jump to Figma Prototype

⏯ Hover to play/pause the video or enter the full screen. Want to click through the game yourself? Jump to Figma Prototype

🙋 Shoulder Extension & Flexion Test

🫱 Shoulder Rotation Test

💪 Elbow Extension & Flexion Test

🫳 Wrist Up & Down Test

👃 🫵 Finger-to-Nose Test

🖐 👊 Finger Extension & Flexion Test

👌 Finger Tapping1 - Coordination Test

👌 Finger Tapping2 - Speed Test

Self-Reports Overview

Self-Report Example

Self-Report Contextual Info

Provide well-designed, personalized self-reports for the patient alongside assessment games, so that the patient, healthcare providers, and clinicians can gain a more comprehensive understanding of the patient's condition.

Reflect on Recovery Progress

Visualize longitudinal self-assessment results and self-reported responses to help patients or doctors quickly gain a clear overview of the patients' recovery journey.

All Self-Test Results

All Self-reported Responses

Hi-Fi Prototypes.

Heyyy! Let's click through the interactive hi-fi prototypes!

09. Design Impact & Stakeholder Feedback

09. Design Impact & Stakeholder Feedback

09. Design Impact & Stakeholder Feedback

Current stage.

🎉 iOS App now under implementation!

As the project initiator, I presented the final solution to the research lab, receiving enthusiastic acclaim from the researchers. It's incredibly thrilling that it's slated for development in 2024!

Stakeholder Feedback.

🫶 Warm feedback from stakeholders

Our design was spoken highly by HCI researchers, healthcare experts, and patients that we closely collaborated with throughout this project.

10. Reflection & Next Step

10. Reflection & Next Step

10. Reflection & Next Step

What I learned.

🎮 ➕ 👩‍⚕️ Value of using gamification in healthcare

Healthcare is a serious field demanding in-depth medical expertise and precision in design details. I was excited and also surprised that my gamification concept, which might sound "unserious", received a warm reception from healthcare experts and patients. Besides being useful in the product itself, making users feel valued and engaged to use the product is equally important for product success.

Another exciting aspect that I discovered about gamification was its potential to convert subjective assessments into measurable tasks with structured game mechanics. It reminded me of advice from my mentor at Alibaba:

“Be rigorous and logical in your design process, while being soft in conveying human touch when delivering your design to your end users.”

💬 ➕ 📖 Combine bottom-up and top-down design methods

Continuously reading literature on narrative motivation, health informatics, gamification in self-monitoring, behavior change, and habit formation significantly contributed to my design ideation and refinement. In addition to leveraging user research (bottom-up approach), designers should adeptly synthesize secondary research (top-down approach) into design decisions. I think it's a good habit to keep ongoing secondary research throughout the design journey, extending beyond those directly relevant sources/products for more inspirations.

Next Step.

🫶 Continue to enhance the accessibility through usability tests

To validate the actual feasibility and accessibility of our proposed solutions, we will conduct usability tests after app implementation and clinic trials. Patients with vision loss, cognitive issues, and other types of disabilities will also be involved in future tests to enhance the accessibility and inclusiveness of our application.