Challenge

Our amazing friends at YR Media has published an intriguing interactive story on AI titled Can You Teach AI to Dance? which got us thinking whether something as organic and complex as a dance can be quantified, measured and mathematically studied with the help of AI. In this project you are challenged to create an AI based app that can track the movements of a dancer and recognize some basic dance moves.

If you haven’t set up your computer and mobile device for App Inventor, go to the “Setup your Computer” tab to set up your system. Otherwise, go directly to the “Awesome Dancing with AI” tab to start the tutorial.

Setup your computer

Awesome Dancing with AI (Level: Intermediate)

Introduction

Dancers Do you like dancing? Are you good at it? Would you like to improve? Is there an algorithm to describe, identify or measure an amazing dance? Can dance moves be quantified and measured? Can AI be used to help you improve your dancing skills? In this project you will learn how to use the new AI technology PoseNet to track key points of your body to create a skeletal model and develop some basic methods to quantify, measure and identify some dance moves.

Posenet Demo Image Credit


Important: Please note that for this project you cannot use the Emulator to test your app as the Emulator cannot run MIT App Inventor extensions such as the PoseNet Extension. To make sure that your mobile device has the needed hardware capability for PoseNet, use AI2 Companion on this .aia test file.

Graphical User Interface (GUI)

A possible GUI has been created for you in the starter file. Please change the properties of the components as you wish to get the look and feel you want. However, please do not rename the components, as this tutorial will refer to the given names in the instructions.

GUI of AI Dance

The SwapCameraButton toggles the camera view from Front to Back as the user wishes. CanvasLiveButton toggles the Canvas background from a solid black color to a live camera view. ResetButton sets the Dance Score back to zero.

The WebViewer component is where the camera view will be cast and the Canvas component is where the skeletal model of the body will be created either against a black background or a live camera view background. The dimensions of the WebViewer component and the Canvas component have to match in order to facilitate body tracking. It is recommended that you do not change the default values given in this starter file.

Note that at the very bottom of the Components panel is the PosenetExtension which is the AI technology that we will be using to track key points of a body and which will help you to build a skeletal version of the body.

PoseNet key points

The key points of the body tracked by PoseNet are: eyes, ears, nose, shoulders, elbows, wrists, hips, knees and ankles.

Posenet key points of the body

Whenever PoseNet is able to track a body key point, it will return a list of two elements representing the x and y-coordinates of the key point. When PoseNet is unable to track a body key point, it will return an empty list.

Preliminary GUI code

You are also given some preliminary code for the GUI. Study these to make sure that you have a general idea what they do.

Posenet Ready Error These two blocks are used to communicate the status of PoseNet as either “Ready” or in case of error, display the error message.

Reset Button The ResetButton resets the Dance Score to zero each time it is clicked.

Swap Camera Button The SwapCameraButton toggles the camera view from its default “Front” view to “Back” and vice versa.

Live Canvas Button The CanvasLiveButton adjusts the value of a Boolean variable which will be used to display the PoseNet constructed skeleton either on a live background or on a solid black background.

Helper functions (1)

You are also given some helper functions.

getX and getY functions extract the x and y-coordinates of a given point. A point is simply a list of two elements that PoseNet returns representing the x and y-coordinates of the given point’s location.

getX and getY

Helper functions (2)

defined is a Boolean function that returns true if the given point is a list of two elements (representing the x and y-coordinates of the point’s location) and false otherwise. If PoseNet is unable to detect any of the body key points, it will return an empty list for each of the missing key points.

procedure defined

allDefined is a Boolean function that checks whether all the points in a given list of points are defined. If all the points are defined, it returns true and otherwise, if any of the points is not defined, it returns false. We will use this helper function to collectively check if certain key points of the body are properly tracked and returned by the PoseNet extension. If the posture of the body is such that perhaps due to poor lighting, messy background, baggy clothes etc. any key point of the body is not trackable by PoseNet, an empty list will be returned to indicate the failure to detect this key point.

procedure allDefined

Helper functions (3)

distance is a function that computes the Euclidean distance between two points on the Canvas when the two points are defined. It uses the Pythagorean formula:

distance formula

which you may recall from high school Geometry and Algebra.

distance function

Helper functions (4)

drawPoint procedure draws a small circle of 4 pixel radius on the Canvas at the location of a given defined point. This procedure will help us draw the key points of the body such as the joints. We use the 4 pixel radius to make the point visible to the eye.

procedure drawPoint

drawLine procedure draws a line segment on the Canvas between two defined points. We will use this procedure to draw the bones of the skeletal representation of the body.

procedure drawLine

Pose Update

When PoseNet extension detects that the body it is tracking has changed its position, it triggers the following PosenetExtension1.PoseUpdated event. This event handler’s code has also been created for you even though the code for each of the procedures that are invoked (drawKeyPoints, drawBody, drawHead etc.) are all empty and will soon be created by you. Please note that the if then statement in this event handler makes sure that if the user clicked the CanvasLiveButton, the camera live feed is broadcast on the Canvas which by default has a solid black background color.

when Pose Updated

Constructing the Skeleton

Now you will write some of these procedures.

drawKeyPoints procedure will draw a red circle depicting each key point of the body returned by PoseNet:

procedure drawKeyPoints hint

procedure drawKeyPoints solution

Try this on your own but if you get stuck you can click the Hint button.

drawBody procedure will draw the skeleton of the body, by drawing yellow lines to depict bones between the appropriate key points:

Note: PosenetExtension1.Skeleton returns a list of “bones” where each “bone” is a list of two key points that are its endpoints. For example a lower leg bone in the skeleton would be a list consisting of an ankle key point and a knee key point.

lower leg bone

procedure drawBody hint

procedure drawBody solution

Try this on your own but if you get stuck you can click the Hint button.

Optional: Drawing a circular head

If you would like to have a circular head above the skeletal shoulders then complete the drawHead procedure. Otherwise there will only be some key points above the shoulder designating the ears, eyes and the nose without the circular head, just like shown in the animation on the first page of the tutorial.

drawHead procedure:

procedure drawHead hint

procedure drawHead solution

Note: When the computation becomes too large horizontally, you can use the “External Inputs” menu option (via a right-click) to shrink the space needed for your operations. To go back to the default option, use “Inline Inputs” as shown.

External and Inline inputs

Try this on your own but if you get stuck you can click the Hint button.

Test your App

Now you will use the AI Companion to check that your app works well. Please note that an Emulator cannot be used in the testing as it does not support MIT App Inventor Extensions like PoseNet.

Check that your app can track a body (yours or someone else’s) and have the skeleton correctly constructed joining the key points of the body. For best results with PoseNet, make sure that the body is well lit and is in front of a background of a solid single color. Baggy clothes may also interfere with the tracking of the body key points.

If you are using a mobile phone (instead of a tablet) your screen might be too small to display everything at once. In this case, as the “Scrollable” property of the Screen is checked by default in this project, you can adjust the screen by scrolling to show what you wish. You can also choose to make the HorizontalArrangement1 (where the shadow dancers images are) invisible. If you are still short of screen space you can also choose to make WebViewer1 invisible by unchecking its Visible property. If you do so, beware that the Canvas1 dimensions (250X300) must still exactly match the dimensions of the WebViewer1.

Detecting Dance Moves

Some simple dance moves that we will explore are:

Hands in the Air Move

Knee Up Move I, II

Travolta Move I, II

If you are too young to remember John Travolta’s famous Saturday Night Fever dance move, click the button.

Image Credit: Wikipedia


T-Pose Move

Now you will write some code to detect some of these dance moves. As these dance moves are detected, the Dance Score of the dancer being tracked will automatically be incremented.

Hands in the Air Move

procedure checkForHandsintheAirMove:

X Y Coordinate System

procedure checkForHandsintheAirMove hint

procedure checkForHandsintheAirMove solution

Try this on your own but if you get stuck you can click the Hint button.

Knee Up Move

procedure checkForKneeUpMove:

procedure checkForKneeUpMove hint

procedure checkForKneeUpMove hint

Try this on your own but if you get stuck you can click the Hint button.

Travolta Move

Now why don’t you try to define the Travolta moves and define the procedure checkForTravoltaMove yourself?

One way to define the Travolta move is either the right wrist is above the right ear and the left wrist is below the left shoulder OR the left wrist is above the left ear and the right wrist is below the right shoulder.

Are you sure you are ready to see a solution?

procedure checkForTravoltaMove hint

Try this on your own but if you get stuck you can click the Hint button.

T-Pose Move

If you want a good challenge, try to define the T-Pose move where the dancer stretches left and right hands horizontally to form the shape of the letter “T”.

Arm stretch

To ensure that the hands are held horizontally the wrist and shoulder y-coordinates should be very close on both sides. Also, a stretched arm means that the distance between the right wrist and the right shoulder should be very close to the sum of the distances from the right wrist to the right elbow to the right shoulder. And similarly on the left side.

Are you sure you are ready to see a solution?

procedure checkForTPoseMove hint

Note: If you know some Trigonometry, it may be possible to simplify some of this code by utilizing angular relationships of the key points.

Try this on your own but if you get stuck you can click the Hint button.

Test your App

Check your app thoroughly that if you (or someone else you are tracking with PoseNet) do any one of the dance moves, the move is correctly detected and the Dance Score goes up. As mentioned before, for best results with PoseNet, make sure that the body is well lit and is in front of a background of a solid single color. Baggy clothes may also interfere with the tracking of the body key points.

Test your app on some YouTube dance videos, where a solo dancer is against a clear background, to see how your app is able to track the dancer.

Congratulations you have now built your first AI based app that can track the movements of a person and recognize some basic dance moves.

Expand Your App

About Youth Mobile Power

A lot of us spend all day on our phones, hooked on our favorite apps. We keep typing and swiping, even when we know the risks phones can pose to our attention, privacy, and even our safety. But the computers in our pockets also create untapped opportunities for young people to learn, connect and transform our communities.

That’s why MIT and YR Media teamed up to launch the Youth Mobile Power series. YR teens produce stories highlighting how young people use their phones in surprising and powerful ways. Meanwhile, the team at MIT is continually enhancing MIT App Inventor to make it possible for users like you to create apps like the ones featured in YR’s reporting.

Essentially: get inspired by the story, get busy making your own app!

The YR + MIT collaboration is supported in part by the National Science Foundation. This material is based upon work supported by the National Science Foundation under Grant No. (1906895, 1906636). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Check out more apps and interactive news content created by YR here.