Gibalica

Target customers are people who would like to:

• engage in basic physical activities such as “raising both hands up”
• improve the left-right orientation
• have fun playing games such as http://www.pjesmicezadjecu.com/dobre-stare-igre/dan-noc.html (e.g. the user should squat when the app plays „sit down“, and the user should remain standing when it says „stand up“)

The product is intended to improve social inclusion of different group of people with disabilities through the positive changes listed above. Furthermore, all of this will have a ripple effect on confidence and wellbeing of people with disabilities. Specifically, the idea described in this document received a positive feedback from a representative of a Down Syndrome association. The students should investigate and assess the generalizability of the solution for different groups of people such as visually impaired persons and people with hearing difficulties.

WHAT PROBLEM OR CHALLENGE DOES THE PRODUCT SOLVE?

The main problems that are tackled with this product are the lack of physical activities and difficulties related to left and right orientation.

WHICH BENEFIT DOES IT PROVIDE?

The key benefits for the users can be summarized as follows:

1. the curated set of physical and coordination exercises – which can be performed without any special equipment (ideally presented in alternative forms – e.g. videos, audio, animations)
2. real-time feedback – the product should be able to detect user movements and assess whether an exercise is completed or not
3. progress tracking – the app should track the user’s progress on completed exercises
4. entertainment – in the form of interactive games such as http://www.pjesmicezadjecu.com/dobre-stare-igre/dan-noc.html
5. accessibility – the product should be designed for all, meaning, it should also be accessible to the users who may have certain disabilities.

The product is a mobile application (phone and tablet) which at least has a support for the Android operating system.

The main feature of the app revolves around the detection of the user and its pose (e.g. placement of head, hands, legs, torso…) using the live feed from the device’s camera as an input. The mobile device would be placed on a stationary object such as table and its camera (i.e., front-facing, selfie camera) would face the user from head to toes.

Such a feature is then used in several modes available from the main menu.

Many different modes can be considered for implementation, including the following:

• learn mode – the user needs to replicate the “pose” instructed by the app. For example, the app may ask the user to “raise a left hand up” and the user should raise a left hand.
• challenge mode
• free mode – the user makes a pose and the app informs (visually and audibly) about the detected pose
• training mode – the user can do workouts such as “10 squats”, “left-right orientation”, “choreography” (concept similar to: https://en.wikipedia.org/wiki/Simon_(game)#Gameplay)
• game mode – e.g. the user should squat if the app prompts “night” and should stand when it prompts “day”

The basic flow of any mode (except the free mode) is as follows:

1. user onboarding – instructing what the user should do before the activity starts
   1. find a location where you can place the device and move freely
   2. place the device on a still surface such as table and point the screen towards you
   3. make three steps back
   4. make sure the front-facing camera can see you fully from head to toe
   5. the app should prompt the user when it is ready to go (e.g. when the user makes the “T-pose” – https://knowyourmeme.com/memes/t-pose)
2. main activity – consists of series of tasks. E.g. in “learn mode”, the app should prompt the user to make a pose “left hand up”. When the left hand is raised, the app may reward the user with points, sound cues or something and then the next task
3. follow up – the screen which will present the user with the outcome of the activity (e.g. statistics, rewards) and offer navigation (e.g. restart or return to main menu)

Note: The free mode does not include “follow up”.

The graphic design should be simplistic and engaging. The user should feel like he/she always interacts with an animated character (female, as per the app’s name).

UI interaction should focus on two key modalities:

• simple UI on-screen buttons
• voice commands

WHAT MAKES IT STAND OUT?

The product aims to fill the gap in the current market of fitness-related applications which may not have the desired level of immersion and interactivity. For example, there are fitness applications which can be used to manually track the number of performed squats (through tapping the screen with a finger), but this product will be able to recognize when the user makes the pose (e.g. performed squat), which makes the app highly innovative. Furthermore, a unique selling point of the app are its accessibility features (e.g. navigating the app with voice commands, different forms of the content representation such as animation, image, or audio).

IS IT FEASIBLE TO DEVELOP THIS PRODUCT?

It is feasible to develop this product as it is a software product which can be developed on any PC with medium-range specifications. There exists the technology stack which can be used to fully develop the product, see the next section for more details.

Given that the product has a strong emphasis on innovative user experience and interaction mechanisms (e.g. voice commands and pose detection), perhaps the most useful contribution from the IT company would be to provide guidance on the UX design (and/or graphics design) of the proposed solution. Furthermore, given the underlying complexity of the solution, the company could help with the architectural design of the application.

In general, the company can use this unique opportunity to meet its CSR (Corporate Social Responsibility) goals. Through the joint collaboration with university and target users (e.g. people with disabilities), the company would be able to receive knowledge and experience in innovative applications of mobile AI such as pose estimation.