Reach#

../../../_images/reach1.gif

Description#

This environment was introduced in “Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research”.

The environment is based on the Shadow Dexterous Hand, which is an antropomorphic robotic hand with 24 joints. The goal of the task is for the fingertips of the hand to reach a predefined target Cartesian position. The hand has a total of 20 motor controlled degrees of freedom out of the 24 joints. The thumb has 5 joints and 5 DoF while the rest of the fingers have 4 joints and 3 DoF (each finger’s distal joint is coupled with a tendon to its middle joint just like a human hand, so that the middle joint angle is always greater or equal to the distal joint angle). The control frequency of the actuators is of f = 25 Hz. This is achieved by applying the same action in 20 subsequent simulator step (with a time step of dt = 0.002 s) before returning the control to the robot.

The kinematics of the Shadow Dexterous Hand resembles that of the human hand. The robot hand has 2 degrees of freedom for the wrist to perform the radial/lunar deviation movements (WRJ1) and flexion/extension (WRJ0). Each finger has three joints in common. The joint closer to the palm is called metacarpophalangeal (MCP) and has a total of 2 degrees of freedom each. In the robot they are defined as FFJ3, MFJ3, RFJ3, LFJ3, and THJ2 (forefinger, middle finger, ring finger, little finger, and thumb respectively) for the adduction/abduction degree of freedom, and FFJ2, MFJ2, RFJ2, LFJ2, THJ1 for the flexion/extension DoF. The middle joint in the fingers is known as proximal interphalangea (PIP), which in the robot hand correspond to FFJ1, MFJ1, RFJ1, and LFJ1. This joint is also responsible for flexion/extension. The last joint in common is the most distant to the palm, called distal interphalangeal (DIP) and in the robot hand FFJ0, MFJ0, RFJ0, and LFJ0. This joint is not actuated but coupled to the PIP joints by tendons in MuJoCo.

In the robot hand an extra joint is added to the little finger LFJ4 in order to perform the opposition movement with the thumb. Also the the human thumb has two different joints than the rest of the fingers. The carpometacarpal (CMC) joint located close to the palm area, THJ4 and THJ3in the robot. And the interphalangeal joint which is in the same location as the DIP but in this case actuated. This joint is the THJ0 in the robot hand.

Action Space#

The action space is a Box(-1.0, 1.0, (20,), float32). The control actions are absolute angular positions of the actuated joints (non-coupled). The input of the control actions is set to a range between -1 and 1 by scaling the actual actuator angle ranges. The elements of the action array are the following:

Num

Action

Control Min

Control Max

Angle Min

Angle Max

Name (in corresponding XML file)

Joint

Unit

0

Angular position of the horizontal wrist joint (radial/ulnar deviation)

-1

1

-0.489 (rad)

0.14 (rad)

robot0:A_WRJ1

hinge

angle (rad)

1

Angular position of the horizontal wrist joint (flexion/extension)

-1

1

-0.698 (rad)

0.489 (rad)

robot0:A_WRJ0

hinge

angle (rad)

2

Horizontal angular position of the MCP joint of the forefinger (adduction/abduction)

-1

1

-0.349 (rad)

0.349(rad)

robot0:A_FFJ3

hinge

angle (rad)

3

Vertical angular position of the MCP joint of the forefinger (flexion/extension)

-1

1

0 (rad)

1.571 (rad)

robot0:A_FFJ2

hinge

angle (rad)

4

Angular position of the PIP joint of the forefinger (flexion/extension)

-1

1

0 (rad)

1.571 (rad)

robot0:A_FFJ1

hinge

angle (rad)

5

Horizontal angular position of the MCP joint of the middle finger (adduction/abduction)

-1

1

-0.349 (rad)

0.349(rad)

robot0:A_MFJ3

hinge

angle (rad)

6

Vertical angular position of the MCP joint of the middle finger (flexion/extension)

-1

1

0 (rad)

1.571 (rad)

robot0:A_MFJ2

hinge

angle (rad)

7

Angular position of the PIP joint of the middle finger (flexion/extension)

-1

1

0 (rad)

1.571 (rad)

robot0:A_MFJ1

hinge

angle (rad)

8

Horizontal angular position of the MCP joint of the ring finger (adduction/abduction)

-1

1

-0.349 (rad)

0.349(rad)

robot0:A_RFJ3

hinge

angle (rad)

9

Vertical angular position of the MCP joint of the ring finger (flexion/extension)

-1

1

0 (rad)

1.571 (rad)

robot0:A_RFJ2

hinge

angle (rad)

10

Angular position of the PIP joint of the ring finger

-1

1

0 (rad)

1.571 (rad)

robot0:A_RFJ1

hinge

angle (rad)

11

Angular position of the CMC joint of the little finger

-1

1

0 (rad)

0.785(rad)

robot0:A_LFJ4

hinge

angle (rad)

12

Horizontal angular position of the MCP joint of the little finger (adduction/abduction)

-1

1

-0.349 (rad)

0.349(rad)

robot0:A_LFJ3

hinge

angle (rad)

13

Vertical angular position of the MCP joint of the little finger (flexion/extension)

-1

1

0 (rad)

1.571 (rad)

robot0:A_LFJ2

hinge

angle (rad)

14

Angular position of the PIP joint of the little finger (flexion/extension)

-1

1

0 (rad)

1.571 (rad)

robot0:A_LFJ1

hinge

angle (rad)

15

Horizontal angular position of the CMC joint of the thumb finger

-1

1

-1.047 (rad)

1.047 (rad)

robot0:A_THJ4

hinge

angle (rad)

16

Vertical Angular position of the CMC joint of the thumb finger

-1

1

0 (rad)

1.222 (rad)

robot0:A_THJ3

hinge

angle (rad)

17

Horizontal angular position of the MCP joint of the thumb finger (adduction/abduction)

-1

1

-0.209 (rad)

0.209(rad)

robot0:A_THJ2

hinge

angle (rad)

18

Vertical angular position of the MCP joint of the thumb finger (flexion/extension)

-1

1

-0.524 (rad)

0.524 (rad)

robot0:A_THJ1

hinge

angle (rad)

19

Angular position of the IP joint of the thumb finger (flexion/extension)

-1

1

-1.571 (rad)

0 (rad)

robot0:A_THJ0

hinge

angle (rad)

Observation Space#

The observation is a goal-aware observation space. It consists of a dictionary with information about the robot’s joint and finger states, as well as information about the goal. The finger tip observations are derived from Mujoco bodies known as sites attached to the body of interest such as the finger tips. The dictionary consists of the following 3 keys:

  • observation: its value is an ndarray of shape (63,). It consists of kinematic information of the block object and gripper. The elements of the array correspond to the following:

Num

Observation

Min

Max

Joint Name (in corresponding XML file)

Site Name (in corresponding XML file)

Joint Type

Unit

0

Angular position of the horizontal wrist joint

-Inf

Inf

robot0:WRJ1

-

hinge

angle (rad)

1

Angular position of the vertical wrist joint

-Inf

Inf

robot0:WRJ0

-

hinge

angle (rad)

2

Horizontal angular position of the MCP joint of the forefinger

-Inf

Inf

robot0:FFJ3

-

hinge

angle (rad)

3

Vertical angular position of the MCP joint of the forefinge

-Inf

Inf

robot0:FFJ2

-

hinge

angle (rad)

4

Angular position of the PIP joint of the forefinger

-Inf

Inf

robot0:FFJ1

-

hinge

angle (rad)

5

Angular position of the DIP joint of the forefinger

-Inf

Inf

robot0:FFJ0

-

hinge

angle (rad)

6

Horizontal angular position of the MCP joint of the middle finger

-Inf

Inf

robot0:MFJ3

-

hinge

angle (rad)

7

Vertical angular position of the MCP joint of the middle finger

-Inf

Inf

robot0:MFJ2

-

hinge

angle (rad)

8

Angular position of the PIP joint of the middle finger

-Inf

Inf

robot0:MFJ1

-

hinge

angle (rad)

9

Angular position of the DIP joint of the middle finger

-Inf

Inf

robot0:MFJ0

-

hinge

angle (rad)

10

Horizontal angular position of the MCP joint of the ring finger

-Inf

Inf

robot0:RFJ3

-

hinge

angle (rad)

11

Vertical angular position of the MCP joint of the ring finger

-Inf

Inf

robot0:RFJ2

-

hinge

angle (rad)

12

Angular position of the PIP joint of the ring finger

-Inf

Inf

robot0:RFJ1

-

hinge

angle (rad)

13

Angular position of the DIP joint of the ring finger

-Inf

Inf

robot0:RFJ0

-

hinge

angle (rad)

14

Angular position of the CMC joint of the little finger

-Inf

Inf

robot0:LFJ4

-

hinge

angle (rad)

15

Horizontal angular position of the MCP joint of the little finger

-Inf

Inf

robot0:LFJ3

-

hinge

angle (rad)

16

Vertical angular position of the MCP joint of the little finger

-Inf

Inf

robot0:LFJ2

-

hinge

angle (rad)

17

Angular position of the PIP joint of the little finger

-Inf

Inf

robot0:LFJ1

-

hinge

angle (rad)

18

Angular position of the DIP joint of the little finger

-Inf

Inf

robot0:LFJ0

-

hinge

angle (rad)

19

Horizontal angular position of the CMC joint of the thumb finger

-Inf

Inf

robot0:THJ4

-

hinge

angle (rad)

20

Vertical Angular position of the CMC joint of the thumb finger

-Inf

Inf

robot0:THJ3

-

hinge

angle (rad)

21

Horizontal angular position of the MCP joint of the thumb finger

-Inf

Inf

robot0:THJ2

-

hinge

angle (rad)

22

Vertical angular position of the MCP joint of the thumb finger

-Inf

Inf

robot0:THJ1

-

hinge

angle (rad)

23

Angular position of the IP joint of the thumb finger

-Inf

Inf

robot0:THJ0

-

hinge

angle (rad)

24

Angular velocity of the horizontal wrist joint

-Inf

Inf

robot0:WRJ1

-

hinge

angular velocity (rad/s)

25

Angular velocity of the vertical wrist joint

-Inf

Inf

robot0:WRJ0

-

hinge

angular velocity (rad/s)

26

Horizontal angular velocity of the MCP joint of the forefinger

-Inf

Inf

robot0:FFJ3

-

hinge

angular velocity (rad/s)

27

Vertical angular velocity of the MCP joint of the forefinge

-Inf

Inf

robot0:FFJ2

-

hinge

angular velocity (rad/s)

28

Angular velocity of the PIP joint of the forefinger

-Inf

Inf

robot0:FFJ1

-

hinge

angular velocity (rad/s)

29

Angular velocity of the DIP joint of the forefinger

-Inf

Inf

robot0:FFJ0

-

hinge

angular velocity (rad/s)

30

Horizontal angular velocity of the MCP joint of the middle finger

-Inf

Inf

robot0:MFJ3

-

hinge

angular velocity (rad/s)

31

Vertical angular velocity of the MCP joint of the middle finger

-Inf

Inf

robot0:MFJ2

-

hinge

angular velocity (rad/s)

32

Angular velocity of the PIP joint of the middle finger

-Inf

Inf

robot0:MFJ1

-

hinge

angular velocity (rad/s)

33

Angular velocity of the DIP joint of the middle finger

-Inf

Inf

robot0:MFJ0

-

hinge

angular velocity (rad/s)

34

Horizontal angular velocity of the MCP joint of the ring finger

-Inf

Inf

robot0:RFJ3

-

hinge

angular velocity (rad/s)

35

Vertical angular velocity of the MCP joint of the ring finger

-Inf

Inf

robot0:RFJ2

-

hinge

angular velocity (rad/s)

36

Angular velocity of the PIP joint of the ring finger

-Inf

Inf

robot0:RFJ1

-

hinge

angular velocity (rad/s)

37

Angular velocity of the DIP joint of the ring finger

-Inf

Inf

robot0:RFJ0

-

hinge

angular velocity (rad/s)

38

Angular velocity of the CMC joint of the little finger

-Inf

Inf

robot0:LFJ4

-

hinge

angular velocity (rad/s)

39

Horizontal angular velocity of the MCP joint of the little finger

-Inf

Inf

robot0:LFJ3

-

hinge

angular velocity (rad/s)

40

Vertical angular velocity of the MCP joint of the little finger

-Inf

Inf

robot0:LFJ2

-

hinge

angular velocity (rad/s)

41

Angular velocity of the PIP joint of the little finger

-Inf

Inf

robot0:LFJ1

-

hinge

angular velocity (rad/s)

42

Angular velocity of the DIP joint of the little finger

-Inf

Inf

robot0:LFJ0

-

hinge

angular velocity (rad/s)

43

Horizontal angular velocity of the CMC joint of the thumb finger

-Inf

Inf

robot0:THJ4

-

hinge

angular velocity (rad/s)

44

Vertical Angular velocity of the CMC joint of the thumb finger

-Inf

Inf

robot0:THJ3

-

hinge

angular velocity (rad/s)

45

Horizontal angular velocity of the MCP joint of the thumb finger

-Inf

Inf

robot0:THJ2

-

hinge

angular velocity (rad/s)

46

Vertical angular position of the MCP joint of the thumb finger

-Inf

Inf

robot0:THJ1

-

hinge

angular velocity (rad/s)

47

Angular velocity of the IP joint of the thumb finger

-Inf

Inf

robot0:THJ0

-

hinge

angular velocity (rad/s)

48

x coordinate of the tip of the forefinger

-Inf

Inf

-

robot0:S_fftip

-

position (m)

49

y coordinate of the tip of the forefinger

-Inf

Inf

-

robot0:S_fftip

-

position (m)

50

z coordinate of the tip of the forefinger

-Inf

Inf

-

robot0:S_fftip

-

position (m)

51

x coordinate of the tip of the middle finger

-Inf

Inf

-

robot0:S_mftip

-

position (m)

52

y coordinate of the tip of the middle finger

-Inf

Inf

-

robot0:S_mftip

-

position (m)

53

z coordinate of the tip of the middle finger

-Inf

Inf

-

robot0:S_mftip

-

position (m)

54

x coordinate of the tip of the ring finger

-Inf

Inf

-

robot0:S_rftip

-

position (m)

55

y coordinate of the tip of the ring finger

-Inf

Inf

-

robot0:S_rftip

-

position (m)

56

z coordinate of the tip of the ring finger

-Inf

Inf

-

robot0:S_rftip

-

position (m)

57

x coordinate of the tip of the little finger

-Inf

Inf

-

robot0:S_lftip

-

position (m)

58

y coordinate of the tip of the little finger

-Inf

Inf

-

robot0:S_lftip

-

position (m)

59

z coordinate of the tip of the little finger

-Inf

Inf

-

robot0:S_lftip

-

position (m)

60

x coordinate of the tip of the thumb finger

-Inf

Inf

-

robot0:S_thtip

-

position (m)

61

y coordinate of the tip of the thumb finger

-Inf

Inf

-

robot0:S_thtip

-

position (m)

62

z coordinate of the tip of the thumb finger

-Inf

Inf

-

robot0:S_thtip

-

position (m)

  • desired_goal: this key represents the final goal to be achieved. In this environment it is a 15-dimensional ndarray, (15,), that consists of the 15 cartesian coordinates of the desired final finger tip position [x,y,z]. The elements of the array are the following:

Num

Observation

Min

Max

Site Name (in corresponding XML file)

Unit

0

Target x coordinate of the tip of the forefinger

-Inf

Inf

target0

position (m)

1

Target y coordinate of the tip of the forefinger

-Inf

Inf

target0

position (m)

2

Target z coordinate of the tip of the forefinger

-Inf

Inf

target0

position (m)

3

Target x coordinate of the tip of the middle finger

-Inf

Inf

target1

position (m)

4

Target y coordinate of the tip of the middle finger

-Inf

Inf

target1

position (m)

5

Target z coordinate of the tip of the middle finger

-Inf

Inf

target1

position (m)

6

Target x coordinate of the tip of the ring finger

-Inf

Inf

target2

position (m)

7

Target y coordinate of the tip of the ring finger

-Inf

Inf

target2

position (m)

8

Target z coordinate of the tip of the ring finger

-Inf

Inf

target2

position (m)

9

Target x coordinate of the tip of the little finger

-Inf

Inf

target3

position (m)

10

Target y coordinate of the tip of the little finger

-Inf

Inf

target3

position (m)

11

Target z coordinate of the tip of the little finger

-Inf

Inf

target3

position (m)

12

Target x coordinate of the tip of the thumb finger

-Inf

Inf

target4

position (m)

13

Target y coordinate of the tip of the thumb finger

-Inf

Inf

target4

position (m)

14

Target z coordinate of the tip of the thumb finger

-Inf

Inf

target4

position (m)

  • achieved_goal: this key represents the current state of the fingers, as if it would have achieved a goal. This is useful for goal orientated learning algorithms such as those that use Hindsight Experience Replay (HER). The value is an ndarray with shape (15,). The elements of the array are the following:

Num

Observation

Min

Max

Site Name (in corresponding XML file)

Unit

0

Current x coordinate of the tip of the forefinger

-Inf

Inf

robot0:S_fftip

position (m)

1

Current y coordinate of the tip of the forefinger

-Inf

Inf

robot0:S_fftip

position (m)

2

Current z coordinate of the tip of the forefinger

-Inf

Inf

robot0:S_fftip

position (m)

3

Current x coordinate of the tip of the middle finger

-Inf

Inf

robot0:S_mftip

position (m)

4

Current y coordinate of the tip of the middle finger

-Inf

Inf

robot0:S_mftip

position (m)

5

Current z coordinate of the tip of the middle finger

-Inf

Inf

robot0:S_mftip

position (m)

6

Current x coordinate of the tip of the ring finger

-Inf

Inf

robot0:S_rftip

position (m)

7

Current y coordinate of the tip of the ring finger

-Inf

Inf

robot0:S_rftip

position (m)

8

Current z coordinate of the tip of the ring finger

-Inf

Inf

robot0:S_rftip

position (m)

9

Current x coordinate of the tip of the little finger

-Inf

Inf

robot0:S_lftip

position (m)

10

Current y coordinate of the tip of the little finger

-Inf

Inf

robot0:S_lftip

position (m)

11

Current z coordinate of the tip of the little finger

-Inf

Inf

robot0:S_lftip

position (m)

12

Current x coordinate of the tip of the thumb finger

-Inf

Inf

robot0:S_thtip

position (m)

13

Current y coordinate of the tip of the thumb finger

-Inf

Inf

robot0:S_thtip

position (m)

14

Current z coordinate of the tip of the thumb finger

-Inf

Inf

robot0:S_thtip

position (m)

Rewards#

The reward can be initialized as sparse or dense:

  • sparse: the returned reward can have two values: -1 if the fingers haven’t reached their final target position, and 0 if the fingers are in their final target position (the fingers are considered to have reached their goal if the 2-nom between the achieved goal vector and the desired goal vector is lower than 0.01).

  • dense: the returned reward is the negative 2-norm distance between the achieved goal vector and the desired goal vector.

To initialize this environment with one of the mentioned reward functions the type of reward must be specified in the id string when the environment is initialized. For sparse reward the id is the default of the environment, HandReach-v1. However, for dense reward the id must be modified to HandReachDense-v1 and initialized as follows:

import gymnasium as gym

env = gym.make('HandReachDense-v1')

Starting State#

When the environment is reset the joints of the hand are initialized with the following angles (rad):

Joint Name (in corresponding XML file)

Angle (rad)

robot0:WRJ1

-0.16514339750464327

robot0:WRJ0

-0.31973286565062153

robot0:FFJ3

0.14340512546557435

robot0:FFJ2

0.32028208333591573

robot0:FFJ1

0.7126053607727917

robot0:FFJ0

0.6705281001412586

robot0:MFJ3

0.000246444303701037

robot0:MFJ2

0.3152655251085491

robot0:MFJ1

0.7659800313729842

robot0:MFJ0

0.7323156897425923

robot0:RFJ3

0.00038520700007378114

robot0:RFJ2

0.36743546201985233

robot0:RFJ1

0.7119514095008576

robot0:RFJ0

0.6699446327514138

robot0:LFJ4

0.0525442258033891

robot0:LFJ3

-0.13615534724474673

robot0:LFJ2

0.39872030433433003

robot0:LFJ1

0.7415570009679252

robot0:LFJ0

0.704096378652974

robot0:THJ4

0.003673823825070126

robot0:THJ3

0.5506291436028695

robot0:THJ2

-0.014515151997119306

robot0:THJ1

-0.0015229223564485414

robot0:THJ0

-0.7894883021600622

For the target cartersian position of the fingers there are two possible initializations chosen randomly. With a probability of 10 % the episodes goal will be to keep the initial position of the finger tips for an indefinete perido of time. The initial position of the finger tips will then be:

Finger Tip

Coordinate

Position (m)

Forefinger

x

0.99

Forefinger

y

0.8

Forefinger

z

0.15

Middle

x

1.02

Middle

y

0.8

Middle

z

0.15

Ring

x

1.04

Ring

y

0.81

Ring

z

0.155

Little

x

1.07

Little

y

0.82

Little

z

0.16

Thumb

x

0.95

Thumb

y

0.84

Thumb

z

0.16

In the other possible episode intializaitons one of the fingers is randomly selected to meet the tip of the thumb over the palm of the hand. The rest of the finger tips must maintain the initial positions mentioned before.

Episode End#

The episode will be truncated when the duration reaches a total of max_episode_steps which by default is set to 50 timesteps. The episode is never terminated since the task is continuing with infinite horizon.

Arguments#

To increase/decrease the maximum number of timesteps before the episode is truncated the max_episode_steps argument can be set at initialization. The default value is 50. For example, to increase the total number of timesteps to 100 make the environment as follows:

import gymnasium as gym

env = gym.make('HandReach-v1', max_episode_steps=100)

Version History#

  • v1: the environment depends on the newest mujoco python bindings maintained by the MuJoCo team in Deepmind.

  • v0: the environment depends on mujoco_py which is no longer maintained.