1 Introduction

The knowledge of human limb motion has an important relevance in robotics when a robot is wanted to perform human-like behaviours. The robotic teleoperation is the action to control a robot at a distance.

One of the most important method to teleoperate robots is through haptic devices. These devices can be joysticks or special suits like gloves or full-body suits that are equip with inertial measurement unit (IMU) and are able to transfer force and positions to a robot in order to perform an specific motion [¹⁵, ¹⁴, ¹², ⁹, ¹⁹, ¹⁰, ¹, ¹⁶, ¹⁸].

The disadvantage of these methods is the sensors noise that can carry out low precision on the final motion of the robot. Other remarkable disadvantage is the suits are made for a fixed size and not all person can use them.

Another method is capture specific points of the body through artificial vision and artificial intelligence, afterwards use them as the set-points for the robot. The motion capture can be with an algorithm that search for a specific color in an image, for example a mark on person’s clothes.

Also, the augmented reality, that is the combination of real word images and information generated by a computer, in combination with haptic devices improves the teleoperation capabilities [⁸, ¹⁷, ¹¹].

In Health Sciences, the robots teleoperation is applied to rehabilitation and therapy techniques such as therapies for children with autism [³, ⁵, ⁴]. In the present paper, a method to mimic the human limbs motions using three-dimensional kinematic model and a fuzzy logic control algorithm.

The kinematic model receives as input the points array from an artificial intelligence algorithm [²] which correspond to the locations of human joint, this array contains the points obtained by three-dimensional analysis of 4 plain images that are taken from 4 different points of view (front, back, right side, left side) instead of a 3-D approximation from a single point of view of a bi-dimensional image [²⁰].

Then, the angles of limbs are obtained by three equations which are derived from kinematic model. These angles correspond to the motions that human limbs can perform on the different axis and are the set point of the fuzzy control algorithms of the robot. The fuzzy logic algorithms are Mamdani fuzzy inference systems with ”if, else” rules engine.

On the Fig. 1 is depicted a general flow diagram of this method. The artificial intelligence algorithm [²] obtains the information of four cameras equipped with two different sensors, RGB sensor and depth sensor.

Fig. 1 NAO robot teleoperation method 

The images captured by the 4 RGB sensors are processed with a Convolutional Neural Network which creates feature maps of input image and infers 2-D key points for person in the image. The 2-D key points are the locations of person’s joints. After that, the 3-D points are calculated making the relation between four set of 2-D points.

Two depth sensors work as security system that activate the RGB data acquisition till the person is in the middle of cameras array. This system ensures that person position in the images is similar in order to minimize error when the 3-D points are calculated.

The artificial intelligence algorithm [²] provides a robust method to detect the joint’s location. The measurements are not affected by the lighting changes and it was not necessary to use special equipment on the body, that is the main advantage of the algorithm.

2 Method

2.1 Kinematic Model

The kinematic model was based on the five basic motions that the NAO robot [⁶] and humans can perform [¹³, ⁷]. These movements are described below:

• – Flexion. It is angle decrease at one joint.

• – Extension. It is angle increase at one joint.

• – Abduction. It is the motion of a limb away from the midline of the body.

• – Adduction. It is the motion of a limb toward to the midline of the body.

• – Rotation. It is the motion around a longitudinal axis of a bone, it can be internal or external.

The motions of upper limbs parts considered in this method were shoulders and elbow motions; the shoulders motions are Flexion / Extension (y), Abduction / Adduction (z) and Rotation (x).

The elbows motions are Flexion / Extension (z). Similarly, the motions of lower limbs are hips motions and knees motions.

The hips motions are Flexion / Extension (y), Abduction / Adduction (z) and Rotation (x). The knees motions are Flexion / Extension (y) as is shown on the Fig. 2.

Fig. 2 Upper and lower limb joints 

The Roll, Yaw and Pith rotations matrixes were selected to describe the previous motions. The Equation 1 is the rotation matrix on the (z) axis (Yaw) and the Equation 2 is the rotation matrix (y) (Pitch), these equations are used to represent the elbows, knees, shoulders and hips Abduction / Adduction and Flexion / Extension motions respectively.

The Equation 3 is the rotation matrix on the (x) axis (Roll) and represents the Rotation motions performed by the shoulders and hips:

Rz=[Cφ−Sφ0SφCφ0001], (1)

Ry=[Cθ0Sθ010−Sθ0Cθ], (2)

Rx=[1000Cψ−Sψ0SψCψ], (3)

where Cφ, Cθ and Cψ represent cos⁡ φ, cos⁡ θ and cos⁡ ψ respectively and Sφ, Sθ and Sψ represent sin⁡ φ, sin⁡ θ and sin⁡ ψ respectively. The angles φ, θ and ψ are the rotation angles on the axis (z), (y) and (x) respectively.

Due the shoulders and hips are able to perform rotations on the 3 axis, the 3 previous equations were multiplied to obtain a single rotation matrix, equation 4:

Rxyz=[A11A12A13A21A22A23A31A32A33]. (4)

where:

A11=CφCθ,

A12=CφSθSψ−SφCψ,

A13=SφSψ+CφSθCψ,

A21=SφCθ,

A22=CφCψ+SφSθSψ,

A23=−CφSψ+SφSθCψ,

A31=Sθ,

A32=CθSψ,

A33=CθCψ.

The artificial intelligence algorithm [²] is able to locate 15 key points. Three of these key points were used as reference point (0, 1, 14) and the others key points were used to estimate the joints angles. These key points are shown on the Fig. 3.

Fig. 3 Key points of human body 

The kinematic model for the upper and lower limbs is described by the following equations:

PMD→=RzP34→+RxyzP23→+P2→, (5)

PMI→=RzP67→+RxyzP56→+P5→, (6)

PTD→=RyP910→+RxyzP89→+P8→, (7)

PTI→=RyP1213→+RxyzP1112→+P11→, (8)

where the Equation 5 describes the final position of the right arm with the vector PMD→. The equation is composed of a translation P2→ from the point 0, P0=[0 0 0]T, to the point 2.

A second translation from the point 2 to the point 3 creates the vector P23→. This second vector has a rotation described by the matrix Rxyz.

Finally, the vector P34→ and the matrix Rz represents the translation from the point 3 to the point 4 and his rotation, respectively.

This procedure was applied to the other limbs, where Equation 6 corresponds to the left arm, Equation 7 to the right leg and Equation 8 to the left leg.

In Tables 1 and 2 are shown the maximum angle that human limbs can perform before get injured. These angles are know as comfort angles [¹³, ⁷]. The vector value of each limb was calculated by the Equation 9:

Vext→=[(Xe−X0)(Ye−Y0)(Ze−Z0)]=[XextYextZext], (9)

where (X0, Y0, Z0) are the limb origin points and (Xe, Ye, Ze) are the end points. For example, the forearm origin point is the elbow an the end point is the wrist, hence Vext→ is the final vector with the point (Xext, Yext, Zext).

Table 1 Comfort angles of right limbs 

Part	Movement	Axis	ID	Angle Range
shoulder	flex / ext	Y	q1	-150° to 40°
	abd / add	Z	q2	-150° to 20°
	rotation	X	q3	-70° to 60°
elbow	flex / ext	Z	q4	10° to 150°
hip	flex / ext	Y	q6	130° to 15°
	abd / add	Z	q8	-45° to 20°
	rotation	X	q7	50° to 45°
knee	flex / ext	Y	q8	0° to 155°

Table 2 Comfort angles of left limbs 

Part	Movement	Axis	ID	Angle Range
shoulder	flex / ext	Y	q9	-150° to 40°
	abd / add	Z	q10	-20° to 150°
	rotation	X	q11	-70° to 60°
elbow	flex / ext	Z	q12	-150° to 10°
hip	flex / ext	Y	q13	-130° to 15°
	abd / add	Z	q14	-20° to 45°
	rotation	X	q15	-50° to 45°
knee	flex /ext	Y	q16	0° to 155°

To obtain each motion angle, the directional cosines were selected to estimate the rotation angles qn on the different axis, where qn is the ID of angles shown in the Tables 1 and 2.

The director cosine for the rotation angle at the (x) axis is the the Equation 10 and for the (y) and the (z) axis are the Equations 11 and 12 respectively:

qn=cos⁡−1(Yext|Vext→|), (10)

qn=cos⁡−1(Zext|Vext→|), (11)

qn=cos⁡−1(Xext|Vext→|), (12)

where Yext corresponds to the (y) point from the vector Vext→ and |Vext→| is magnitude of that vector. In the same way, Xext and Zext correspond to the (x) and (z) points from vector Vext→ respectively and |Vext→| is magnitude of that vector.

2.2 Control Algorithm

The control algorithm used to control the robot’s joint speed was a Fuzzy PD (Proportional Derivative) controller.

This controller is a Mamdani Fuzzy Inference System which is based on ”if-else” inference engine. For this application, the hip rotations were excluded due the NAO robot [⁶] is not able to perform this motion.

The controller’s set-points were bounded to the angles values in the Tables 1 and 2 in order to prevent a damage at the robot joints. In the first step, the controller, Fig. 7, assigned fuzzy values to error and differential error through input membership function.

Fig. 4 Error membership functions 

Fig. 5 ΔError membership functions 

Fig. 6 Output memberships functions 

Fig. 7 Fuzzy Controller 

The error was estimated subtracting the set-point and the angle measured by the sensors in robot’s joints at an instance of time (e(t)) and the differential error is the error is Δe(t)=e(t)−e(t−1).

Then, the fuzzy values from the inputs were processed by an inference engine. Finally, the crisp values, that controls the robot’s joints, were estimated by the defuzzification block and sent them to robot’s actuators.

The error and the differential error were in radian and could be positive and negative, due this, five triangular memberships functions were assigned. For example, the membership functions from the right shoulder are shown on the Fig 4 and 5 .

Each motion angle have five triangular memberships functions for the error and other five for the differential error. The name assigned to the membership functions are the follow:

• – eng. Big Negative Error.

• – enb. Low Negative Error.

• – ec. Zero Error.

• – epg. Big Positive Error.

• – epb. Low Positive Error.

The robot’s speed motion was controlled with values from 0 to 1, which 0 is zero speed and 1 is the maximum speed. Five triangular membership functions were defined in this range.

On the Fig 6 is shown an example from the right shoulder. Similarly to the input, each motion angle has five output membership functions. The name assigned to the membership functions are the follow:

• – vc. Zero Speed.

• – vb. Low Speed.

• – vm. Medium Speed.

• – va. High Speed.

• – vmx. Maximum Speed.

The inference engine was defined with the rules shown in the Table 3. The fuzzy rules were designed to smoothly decrease the speed when the joint approaches to the set point. These behaviour is shown on the control surface in Fig. 8.

Table 3 Fuzzy inference rules 

e / Δe	eng	enb	ec	epb	epg
eng	vmx	va	vm	va	vmx
enb	va	vm	vb	vm	va
ec	vm	vb	vc	vb	vm
epb	va	vm	vb	vm	va
epg	vmx	va	vm	va	vmx

Fig. 8 Control surface 

Finally, the crisp value that was sent to the robot was estimated with Center of Gravity (CoG) method. This defuzzification method is shown in the Equation 13:

x=∑i=1nμc¯(xi)⋅xi∑i=1nμc¯(xi),a (13)

where x is the crisp value, μc¯ is the membership function and xi output variable.

3 Results

The tests were divided into two: stable posture tests and unstable posture test. When the body’s center of mass is located at the support base is called as stable posture.

The human is able to perform this kind of posture without loss the balance. If the body’s center of mass is not located at the support base, then it is an unstable posture. For this test, the unstable posture was divided into free unstable posture and intervened unstable posture.

When a person lifts the robot in order to avoid the feet touch the ground is an intervened unstable posture, if the robot’s feet touch the ground is a free unstable posture. The errors were estimated subtracting the final angle for each joint of the human limbs and robot limbs. These angles are shown in the Tables 1 and 2:

The Equation 14 is the Relative Angle Error ERMA(n) and the Equation 15 is the Total Position Error ERTP. At the Equation 14, kn is the angle from robot’s joints and qn is the angle from the human’s joints. The Equation 15 is the average of each Relative Angle Error ERMA(n) from the 14 key point.

On the fists test, the person performed an stable posture. The person only extended his arms and did not move the legs. The Table 4 contains the errors from each limb’s angle, in this Table P-R is refereed to Person-Robot.

In the Fig. 9, Fig. 10 and Fig. 11 are show examples of the approximated location of human’s key points, the skeleton created by the artificial intelligence algorithm [²] and the final robot’s posture, respectively.

Fig. 9 Key points locations on human 

Fig. 10 3-D Skeleton 

Fig. 11 Robot’s final posture 

The second test was a stable posture too. But in this case, the person extended his arms completely in order to perform a “T” posture, the legs remained at the same position of the previous posture.

The Table 5 contains the errors from each limb’s angle, in this Table P-R is refereed to Person-Robot. On the third test, the person raised one leg and extend the arms slightly and the robot stayed on the ground because it was a free unstable posture.

The robot’s anti-fall safety system located at the at the feet was activated automatically lowering the legs to the ground in order to avoid a fall. The Table 6 contains the errors from each limb’s angle, in this Table P-R is refereed to Person-Robot.

The fourth test was an intervened unstable posture, hence a second person lifted the robot in order prevent the activation of robot’s anti-fall safety system.

The first person raised one leg, flexed one arm and raised the same arm in order to locate the wrist near to the chest. The Table 7 contains the errors from each limb’s angle, in this Table P-R is refereed to Person-Robot.

On the final test the fist person raised one leg and both arms and the robot stayed lifted by the other person in order to perform an intervened unstable posture. The Table 8 contains the errors from each limb’s angle, in this Table P-R is refereed to Person-Robot.

Finally the Average Relative Errors for each test was calculated and shown in the Table 9.

Only the Average Relative Errors of the arms was used to compared with the results of the state of the art, due the total error of the posture was not included as a result on the referenced paper. Both results are shown on the Table 10.

4 Conclusion and Future Work

With the detailed information of limbs motions provided by the kinematic model was possible to estimate the joint angles without complex equations. These angles were used as control system’s set points that controls the joint’s motion speed.

The maximum and minimum joint’s angles from human and robots are slightly different, due the maximum angles that the robot can perform were lower than the angles that a human can perform.

Even with this difference, the final posture errors resulted only 1% different than the result obtained in a similar work of the state of the art. The method proposed on the state of art [²⁰] used a standard library to locate the human joints based on the information of the RGB sensor and the depth sensor.

The 3-D location of the joints were estimated only with one plane image. This is the main disadvantage of the method due the algorithm cannot estimate the 3-D position of a joint if the depth sensor cannot measure the complete body of the person, for example, if the RGB sensor and the depth sensor can only measure the left side of the human body, the 3-D location of the joint and the teleoperation cannot be performed due the information of the missing parts cannot be inferred.

Additionally, the OpenNI library requires a special illumination because the location of the joints cannot be estimated if the RGB image does not have a specific range of colors.

The proposed method in this paper, the 3-D analysis of the person posture was made with the information of 4 plain images from different points of view. Hence, the 3-D location of the joints are more reliable than the method where the 3-D location is estimates using only one image.

Furthermore, the limitation which the person’s joints must be located in front of 2 sensor was eliminated. The artificial intelligence algorithm [²] provided the joints location no matter where the person was located on the range of the 4 cameras. The measurements were not affected by the lighting changes due the artificial intelligence algorithm was trained with different kind of images, those images included bright and dark images. The robot’s final posture from Test 3 was affected by the anti-fall safety system programmed by the manufacturer. This can be seen in the error values from the hip and knee.

Thank of the human intervention, the robot could perform a complex posture avoiding the anti-fall safety system activation getting better errors values of the hip and knee. The human body can keep the balance naturally and unconsciously, but the robot cannot do it. That it is the main reason why the manufacturer included the anti-fall safety system.

The fuzzy control algorithm had a favourable performance moving smoothly the robot’s limbs to the set-points. The joints speed was decreased when the angle measured by the joint sensor got close to the set-point.

The fuzzy membership functions were correctly set due the maximum operation range and the speed configuration of the NAO limbs were knew that are included in the technical documentation of the manufacturer.

The future research will be focused on test and develop new deep learning models and fuzzy control algorithms to improve the measurement of human limbs and motion control, also the hands and finger motion will be included on the future 3-D models. The main application to focus will be the medicine and mining.

For medicine, the application to focus will be the surgeon robotics. Nowadays, the surgeon robots are teleoperated through haptic devices and specialist surgeons require many training hours to properly operate it. The motion capture through computer vision to teleoperate a robot will reduce the training time, due the surgeons will need only move their hands as they normally do at surgeon procedure, the haptic devices are not going to be required anymore.

For mining, the research will be focus to eliminate the risk to get injured or dead at work or explore a mine. The labor will teleoperate a robot from a safe distance, the robot in the mine will mimic the labor’s movements. If the mine collapse, explode or has a water leak, the human life will be safe and the only lost will be the robot.