MULTIMEDIA

Continuous self-feeding by monkey A, showing 7 consecutive successful trials. The monkey's cortical control is 4-dimensional, including 3 dimensions of endpoint control plus gripper control

As the monkey makes a reach toward an initial target with the prosthetic arm, the target is displaced so that a direct move to target would knock the food off the presentation device. The monkey then moves the arm endpoint in a curved path to avoid the collision, and successfully obtains the food

When a target is presented, the monkey ignores the target and instead moves the arm so as to be able to lick the gripper fingers. This emergent behaviour is outside the task requirements and is a result of embodied control

When a marshmallow ends up barely between the monkey's lips at the end of a successful reaching and retrieval, the animal is unable to get the food into its mouth without a helping "hand", so it uses the robotic arm to push the food into its mouth

Continuous self-feeding by monkey P, where the cortical control is 3-dimensional, i.e. endpoint control. The gripper is controlled as a dimension dependent on endpoint movement: it opens when the arm moves forward and closes when the arm is held stable or moved backward

Monkey is directly controlling a 3-dimensionally moving prosthetic robot arm to feed itself

This is a video showing a replay of a pre-recorded session of neural prosthetic control. The older arm, made by Keshen, is powered by motors at the joints, whereas the new Barrett arm is powered by a series of motors and pulleys. It is noticable that the same control singal yields a smooth trajectory of the Barrett arm, whereas the Keshen arm demonstrates jerkier movement. This demonstrates that the mechanical quality of the prosthetic arm plays an important role in quality of control

This is an 11 sec movie of the first success of moving an anthropomorphic robot arm through free space using simultaneously recorded units from the brain. This signal was comprised of 26 units recorded from primary sensory and motor cortices in a rhesus monkey. These experiments were performed at Arizona State University and funded by the Neural Prosthesis Program at NIH. These are actual real-time recordings made as the animal reached to a series of pushbuttons. The brain signal is transduced to robot movement with less than a 20 ms delay

Monkey sees the robot's position in a VR display and moves the robot with brain control while its own arms are restrained

Demonstration of center-out movements with anthropomorphic robot arm

Demonstration of two robots interacting in a feeding task using a human controlling anthropomorphic arm

The cursor is initially controlled by the hand position, but later in the movie it is controlled only by the brain-derived signal ("brain powered"). This was within the first few days that the monkey had been exposed to this task and we were using 24 simultaneously recorded units in motor cortex processed with the population vector algorithm

Was recorded the day after movie 1. Notice that the animal is moving its arm during the brain controlled portion, but in subsequent moves it puts its arm down

Was recorded several weeks later

This is an animation of population vectors calculated in 10 m sec bins while a monkey traced spirals on a touch screen. The blue line represent the preferred direction of each M1 neuron in the population vector scaled by the instantaneous firing rate of that neurons. The white line plots the trajectory of the population vector (which matches the trajectory of the arm)

This is an animation of population vectors calculated in 10 m sec bins while a monkey traced an ellipse. The blue line shows the actual trajectory of the hand. The blue arrow shows the tangential velocity of the hand.The yellow lines over the head show the preferred directions of approximately 200 neurons in primary motor cortex scaled by their instantaneous firing rates. The yellow arrow shows the population vector. Note how the population vector slightly proceeds the tangential velocity of the hand

This is a video of an actual trial in which the monkey traces an ellipse. The sounds are action potentials from one neuron in primary motor cortex. Note how the neuron is most active when the hand moves to the left and slightly down.That direction is the preferred direction for this neuron

This animation depicts the position of the hand (magenta dots) and the eye (yellow stars) while the monkey traced an ellipse (blue tube). Note that eye movement is saccadic and typically fixates on the corners of the ellipse. The eye position remains anchored to a point on the ellipse until the hand has reaches the straightaway of the ellipse and saccades to catch up with the hand just before it reaches the next corner