nova página do texto(beta)
Inglês (pdf)
Artigo em XML
Referências do artigo
Tradução automática
Enviar este artigo por email
Citado por SciELO 
Similares em
SciELO 
Permalink
1 Introduction
Digital skills are increasingly becoming a must have requirement of skilled labor in our society. The demand for digital skills will increase in the coming years, so it is necessary to invest in infrastructure, digital services, teacher training, among others, to be prepared for the future [1]. The new generations must be trained not only in linguistic but also in digital literacy [2].
Serious games are increasingly being used for education and training of a certain skill. A serious game is a game designed with an objective different than just entertaining its users [3]. They have been developed for training in a wide range of topics, such as medicine [4], intelligent buildings [5], and even in tactical combat [6]. Similarly, gamification is a term used to describe those game elements such as rewards and competition [7], looking to motivate the use of a certain tool. Gamification in education and learning most commonly uses affordances signaling achievement and progression [8].
Creating learning experiences for children through digital programming activities in their early years provides a strong foundation for developing productive technology users and thinkers of the future [9]. The term Computational thinking (CT) was made popular by Jeannette M. Wing [10], who defined CT as a process that involves solving problems, designing systems, and understanding human behavior, based on the fundamental concepts of computer science. These skills are useful for everyone to integrate into the working market and be active participants of an increasingly digital society. Currently, there are many proposals about what the main concepts of the CT are. The four concepts mostly commonly mentioned in the literature [1,11–15] are: Abstraction, Algorithms, Pattern Recognition, and Decomposition.
On the other hand, learning is affected by academic emotions [16], which involve coordinated psychological processes, and in turn, include affective, cognitive, physiological, motivational, and expressive components [17]. Like human beings, for an educational system to have the ability to recognize and express emotional states, it must have similar mechanisms at its disposal to perceive the physical world through different sensors [18]. Knowing the emotional state of a student in real-time allows an educational system the ability to modify its interaction with the student. In this way, students can remain in an emotional state that is optimal for their cognitive process.
The main contribution of this work is the integration of the various important CT concepts such as pattern recognition, algorithms, abstraction, and decomposition, in a learning tool with different kinds of exercises, which allow students to learn gradually while having fun. Another important contribution is the integration of emotion recognition and motivational techniques through gamification to achieve better interaction with students, depending on the affective state they present when using the tool.
This paper is organized as follows: Section II discusses the state-of-the-art of topics related to our work. Subsequently, in Sections III we present the Intelligent Learning Environment (ILE) with their components. Section IV shares the validation results of our ILE. Finally, we conclude the paper, and define our future steps in Section V.
2 Related Work
2.1 Learning Environments for Computational Thinking
In CT literature, it is common to use coding activities as the main teaching tool [19], while, to a lesser extent, other approaches analyze issues such as design and creativity [20]. Authors [21] present a web-based game for gamifying CT learning with the support of collaborative learning. The environment consists of giving instructions to robots using visual programming, where the next elements of CT are covered: Algorithms, abstraction, decomposition, reformulation, recursion, and testing. Similarly, authors [22] designed and implemented an application to promote computational thinking with robots named Roboliterate. They focused on learning engagement and found that some users engaged in his interest for a longer time, compared to other learning environments that did not use interactions with robots.
2.2 Facial Expression Recognition
In general, there are two fundamental approaches used in the literature for the classification of emotions [23–25]. The first, a categorical approach, specifying that emotions are discrete and separately identifiable. The second, a dimensional approach, where emotions are placed in space that has a limited set of dimensions.
In the work published by Ministry et al. [26], the authors used a categorical approach for emotion recognition. First, they propose a modified local binary patterns algorithm to generate an initial facial representation. Then, an evolutionary particle swarm optimization (PSO) to perform feature optimization. Their algorithm achieves an accuracy between 94.66% and 100% for seven facial expressions. Likewise, authors [27] used stationary wavelet entropy to extract features, a single hidden layer feedforward Neural Network (SLFN), and a proprietary algorithm to obtain an accuracy of 96.80±0.14% in the classification process. Similarly, there are efforts to recognize learning-centered emotions using a convolutional neural network, obtaining up to 95% accuracy [28].
Facial emotion recognition literature has a lot of work using controlled environments. However, there are many challenges in in-the-wild environmental conditions [29], such as posture (non-frontal image), illumination (low image quality, use of the infrared image) and age invariance (muscle movements vary with age) in uncontrolled environments [30]. Authors [31] used facial recognition to measure and quantify the effect of gamification on users’ emotions.
2.3 Engagement in Education
The success of gamified systems depends on their ability to engage users by eliciting both positive and negative emotions [32].
The goal of educational gamification is to maximize student engagement, welcome positive competitiveness, and make learning fun [33]. Some authors emphasize the use of meaningful gamification [34, 35], which involves using specific elements for each user, with the goal of long-term learning, rather than using short-term rewards.
Some works have studied the influence of gamification elements in the emotional states of students [8, 36–38]. Also, the use of gamification techniques on drivers has been explored, to study the influence of emotional states [39].
3 The Intelligent Learning Environment
The main goal of this work was to develop an intelligent learning environment aimed at teaching computational thinking, using cutting-edge techniques such as recognition of emotions focused on learning and motivational techniques used in education. This section describes the system architecture, modules, and mechanisms that give intelligence to the proposed system, as well as its user interface.
3.1 General Architectural Design
The ILE was designed as a Web application, mainly for it to be portable (to be able to be used from any device). The ILE was designed using the layered architectural pattern Model View Controller (MVC).
This pattern is mainly based on the principle of separation of concerns. So, it separates the data, the business logic, and how the information is presented to the user.
The Views are responsible for presenting the information to users and interact directly with the Controllers. The Controllers are responsible for responding to events and invoking requests to the Model.
The Model is responsible for managing the data used in the application. Fig. 1 shows the general architecture of the learning environment. On the left, the three main layers in which the system is organized are shown. On the right, the different modules used are shown.
3.2 Modules
Due to the nature and scope of the project, many functionalities were separated into separate modules (see Table 1) to obtain the final learning environment. In this way, modules can be reused by other applications, if necessary.
Table 1 Modules used within the learning environment
| Description | |
| Graphics Engine | A module that encapsulates the whole process necessary to generate 3D graphics in web browsers. This module uses BabylonJS [40] as the graphic engine. |
| Image Processing | This module is responsible for obtaining images from the camera of the device used while using the learning tool. Also, it processes the images obtained to detect the faces of the students and apply filters for better image quality. For this task, we rely on OpenCV.js [41]. |
| Visual Programming | A client-side library for creating block-based visual programming languages and editors. For this purpose, we decided to use Blockly [42], mainly because is configurable, expandable, and open source. |
| Machine Learning (Emotion Recognition) | A module to classify and recognize five different kinds of emotions focused on learning: Bored, Committed, Excited, Concentrated, and Interested. This component uses convolutional neural networks for the emotion classification process. |
| Assistance | This module uses user usage data, to determine if students require intervention during their learning process. |
| Gamification | This module integrates four elements (points, badges, levels, and progress bar) that are commonly found in games to the learning environment, to motivate students to continue using the learning tool. |
3.3 Adding Intelligence to the Learning Environment
One of the main goals of the ILE was to create an intelligent system capable of providing personalized instructions to each of the students.
The ILE can perform different interventions in real-time automatically, while the students use the tool. To achieve this, an algorithm that processes three main elements was used. The data used by the algorithm are:
Real-time information (elapsed time and executions of exercise) is obtained while the student solves an exercise in the ILE.
Additionally, student emotions are historical information of each of the students, previously stored by the machine learning module in the database. The intervention selector algorithm describes the process for choosing an appropriate.
intervention for a student, given the three parameters. The type of intervention was assessed according to the experience of a group of programming teachers at different levels of education. If the device where the tool is used, does not have a camera, or if the student decided not to allow the use of the camera, the system randomly chooses between motivational and informative interventions.
Three types of interventions can be presented to a user. Table 2 describes such interventions, which may contain text, images, and videos. In general, interventions seek to maintain and direct users to an optimal state of learning. The application is configurable so that both the use of the registered emotions of the students and the use of the interventions can be activated or deactivated. If the use of students’ emotions is deactivated, the system will provide both types of interventions (informative and emotional) but choosing randomly between both with a 50% probability for each of the interventions.
Table 2 Types of interventions
| Intervention Type | Description |
| Initial | This refers to the information presented only at the beginning of the exercises that require it, without the need to evaluate any information from the user. Initial interventions cannot be deactivated since they are necessary to explain the tasks that must be performed at the beginning of each exercise. |
| Informative | These interventions seek to reinforce students' knowledge of the subject matter of the current exercise, based on relevant information that will facilitate the completion of the exercise. |
| Motivational | They use the gamification module to present elements such as trophies, information on the user’s progress, among others, to motivate users to continue using the application. |
3.3.1 Recognizing Facial Emotions of the Users
The first step in the process of emotion recognition is the detection of users’ faces. A cascade classifier based on Haar characteristics was used to achieve this. Then, each face found is processed by applying filters to improve image quality.
Finally, each image is processed by a Convolutional Neural Network to recognize five different classes: Bored, Committed, Excited, Concentrated, and Interested [43]. The hyperparameters of a convolutional neural network were optimized using genetic algorithms. The final model precision for the five categories resulted in an 82% accuracy using cross validation.
The five emotions that are recognized by this classifier are focused on learning, which allows us to use this information to manipulate the application's interaction with the student in real-time.
This model was created using Keras (an interface capable of running on TensorFlow) and later transformed to be used in web applications using TensorFlow.js. TensorFlow.js [44] is a library for developing and training machine learning models in JavaScript. Therefore, the machine learning model can be used directly into the web browser using two files: one containing the topology and the other containing the final weights of the neural network. Each emotion recognized by the machine learning model is stored in the database. In this way, this valuable information is used in decision-making within the ILE.
3.3.2 Gamification
The ILE implements gamification techniques to motivate students, for which 3 elements are used:
Points. Each solved exercise generates a certain amount of points; This depends on the time required to solve the exercise and the number of executions required. Students can repeat the exercises to try to get a better score in each of them.
Trophies. There are trophies that are obtained by accumulating a certain amount of points. When solving each of the exercises, it is verified if the student has reached the necessary score for each of the trophies; if a new trophy was obtained, the system displays a pop-up window indicating it to the user.
Position Table. Since the ILE is intended to be used in real courses, students must select the group to which they belong at the time of registration. Each student will be able to visualize the score that each of their classmates has so far and the trophies they have managed to obtain. This is intended to encourage them to be competitive and interested in using the tool frequently.
3.4 User Interface
The main goal of the ILE is the students learn the four main topics of computational thinking. Therefore, the learning environment was divided into four sections: Algorithms, Pattern Recognition, Abstraction, and Decomposition. Each of these sections has its dynamics (e.g. puzzle, mini game), so the learning environment varies in some cases.
Fig. 2 shows the graphical interface. In Fig. 2 (A), a 3D world is shown, where the main objective is to make the character of the game move towards the goal. To achieve this, the student must create an algorithm using the blocks provided.
Similarly, in Fig. 2 (B), a first-person view of the main character can be seen. Each of the hemispherical objects represents an exercise in the abstraction subject, which can be accessed by approaching them. In Fig. 2 (C), the goal of the exercise is to create a mini game using the existing blocks. In this case, students can modify the amount of 3D objects that will be generated, the speed at which they will move, as well as the size of them.
Finally, in part (D) of Fig. 2, an exercise for learning the subject pattern recognition is shown. The goal of the exercise is that the students modify the properties of the 3D objects using sliders, making the 3D object comply with the pattern shown in the background.
We strived to make the interface of the learning environment look more like a game (using 3D graphics) than a learning tool. That is why the ILE can be classified also as a serious game. In this way, we sought that the learning environment had a wide acceptance by students, aimed primarily at K-12 children.
Interventions are shown as pop-up messages and are registered within the application. The system manages student information, seeking to show interventions of the appropriate type, with different information on each occasion.
4 Evaluation
Two independent experiments were performed to assess the learning environment presented. Before experiments, a preliminary evaluation was conducted to help us improve the application.
Then, a cognitive evaluation was carried out to measure statistically the learning gain from students who used the learning environment.
Finally, a technology acceptance test was performed to analyze the degree of acceptance of the learning environment by students.
4.1 Participants
The preliminary evaluation was carried out with fourteen graduate students in the field of computer science and 76 K-12 students.
Regarding the cognitive evaluation, an experiment was carried out at the Francisco Zarco elementary school (see Fig. 3), located in the city of Culiacán, Sinaloa, Mexico. A total of 102 K-12 children (Age = 9-12 years, M = 10.37, SD = 0.628) participated in the experiments.
4.2 Experimental Design
Preliminary tests helped validate the correct operation of the application and aspects related to its usability. These tests consisted of the use of the learning tool by computer experts, as well as by end users.
The process consisted of performing 10 exercises within the application and then answering a brief survey that served as feedback. These preliminary experiments served to make improvements and correct errors.
Subsequently, the cognitive evaluation was applied to statistically measure the learning gain from students who used the tool.
The goal of the cognitive evaluation was to compare the learning obtained by students who used the learning tool, contrasted to students who learned from traditional classroom material.
To quantify the learning gain, a four-stage experiment was designed. First, a pre-test evaluation was applied to all the participants of the study. Then, in the intervention phase, participants were divided into two groups (experimental and control) using pre-test results, so that they were statistically balanced.
The experimental group used ILE to learn, while the control group learned through traditional material used in class. Then, a post-test was carried out by all the participants of the experiment. Finally, a technology acceptance test was carried out by the experimental group.
The intervention phase consisted first of explaining the purpose of the experiment to the students. Subsequently, a brief orientation was given on the general use of the tool (mainly navigational aspects of the tool). Finally, students were asked to work with section one, which covers topics related to algorithmic logic. In the intervention phase, each of the students was given one hour to complete all the exercises available in that section. The pre-test and post-test consisted of an evaluation of ten questions on logic issues and basic control structures used in programming.
These tests were performed in 30 minutes. Table 3 shows sample questions used in these tests. Finally, the technology acceptance validation consisted of a survey that was applied to all the students who used the ILE.
Table 3 Subset of questions used in pre-test and post-test
| Question | Solution |
| How many times does the instruction [x = x + 3] must be repeated to make x have the value of 15, if the initial value of x is zero? | a) 3 times b) 4 times c) 5 times d) 6 times |
| What message will be printed when the following
instructions are
executed? age = 10 If age ≥ 18 Then Print "You're of age" Else Print "You are a minor" End If |
a) age = 10 b) You're of age c) You are a minor d) None |
This survey consisted of 10 questions, two for each element analyzed within the TAM model [45].
5 Results and Findings
5.1 Cognitive Evaluation
The results of the cognitive evaluation are presented in table 4. At the end of the experiments, 14 students were discarded because they did not attend the three stages of the experiment, resulting in a total of 88 students with whom to perform the statistical analysis, which corresponds to 44 students for each of two groups: the control and the experimental group.
Table 4 Basic statistics of evaluations
| Test | GRP | N | Media | Std Dev | SEM |
| Pre | Exp. Ctrl |
44 44 |
3.818 4.00 |
1.419 1.161 |
.214 .175 |
| Post | Exp. Ctrl |
44 44 |
4.590 4.636 |
1.896 1.526 |
.286 .230 |
Data obtained from the pre-test satisfied a normality sample (Sk = 0.565, K = 0.429), that indicates that the difference in students' prior knowledge comes from a normally distributed population. Therefore, a parametric test was used for hypothesis testing.
An independent t-test was conducted to compare the students' prior knowledge (Pre-test) among the students in the experimental group and the control group. Table 5 shows the results of the t-test. The result indicated that there was no statistically significant difference (t (88) = -.658, p =.512) between the students in the Experimental Group (M = 3.82, SD = 1.419) and the Control Group (M = 4.00, SD = 1.16).
Table 5 T-test results for independent samples
| Independent Samples test | |||
| Test type | t-test for Equality of Means | ||
| T | df | Sig. (2-tailed) | |
| Pre-test | -.658 | 86 | .512 |
| Post-test | -.124 | 86 | .902 |
Similarly, the post-test data shows no statistically significant difference (t (88) = -.124, p =.902). The post-test evaluation score mean for the control group is slightly higher (M = 4.636, SD = 1.526) than that of the experimental group (M = 4.590, SD = 1.896).
However, calculating the difference in means (MD) between both evaluations, the control group had an average increase in score of MD =.636 while the experimental group reported an increase of MD =.772, which may indicate a greater gain in knowledge for the experimental group.
We should make more experiments with different statistical techniques in order to find the source of the no significant difference.
It is important to mention that during the experiment, a total of 2139 student emotions were inferred by the system (table 6). Most of these emotions belong to the committed and interested category (56.66%), followed by excited and concentrated (42.22%), and only a few instances of bored (1.12%). Based on these results, we estimate that students were engaged in the learning process while they were using the ILE. Therefore, according to the algorithm presented in Fig. 2, a greater number of informative interventions were presented to the students.
5.2 Technology Acceptance Model
Perceived enjoyment (PE) was added as an external factor in the TAM evaluation, as it is a factor of interest for this study. PE emphasizes the usage process and reflects the pleasure and enjoyment associated with using a system [46]. The results from the TAM survey are shown in table 7.
Table 7 Results of the TAM elements in the surveys
| TAM Element | Acronym | Cronbach´s Alpha Value |
| Attitude Towards Use | ATU | 0.71259499 |
| Intention to Use | IU | 0.72520171 |
| Perceived Ease of Use | PEU | 0.63245997 |
| Perceived Enjoyment | PE | 0.80812517 |
| Perceived Usefulness | PU | 0.609256 |
The Cronbach’s alpha value for all the analyzed elements is greater than 0.6. As a standard, Cronbach’s alpha values greater than 0.7 are considered acceptable, while values between 0.6 and 0.7 correspond to questionable results. Perceived enjoyment (PE) resulted in the highest Cronbach’s alpha value (>0.8) of all TAM elements; This indicates that the students had fun while learning topics related to computational thinking and it is consistent with respect to results shown in table 6.
In the same way, both ATU and IU obtained acceptable results, indicating that students were engaged during their learning in the experiments and they would be interested in using the tool in the future. On the other hand, the results of PEU and PU correspond to uncertain results. Results of PEU can be interpreted as the system being complex and somewhat difficult to use. At the same time, PU values may indicate that there was confusion on the part of the students, not expecting that they were learning by interacting with a serious game.
6 Conclusions
This paper presented a novel ILE for teaching the main topics of computational thinking. This learning environment integrates facial expression recognition, a 3D graphic interface, and gamification. The use of facial emotion recognition allows the learning environment to have timely information for decision making. With this, it seeks to keep students in a positive affective state. On the other hand, the use of gamification techniques, and the design of a 3D user interface, seek to motivate students to use the tool for a longer amount of time.
This document reports the result of a cognitive test performed on K-12 students, as well as a technological acceptance test. Technological acceptance tests obtained good results but could be improved. Based on both the opinions collected through the surveys and the attitude (emotions) shown by the students during the experiments, it was observed that the students were motivated when using the educational environment. Therefore, we can conclude that using a serious game to learn computational thinking might be positive.
On the other hand, the results of the cognitive evaluation show that we had a learning gain with no significant difference. We need to add more experiments in order to get more and exact conclusions about this result.
Previous studies have evaluated the introduction of programming and computational thinking in children of different ages [14], [47] using different computational tools.
Our work explores the integration of three main elements, which no other known CT teaching tool integrates emotional recognition to try to direct the emotions of students in real-time, a 3D graphics engine in a learning environment, and gamification techniques to motivate the students. Academic emotions involve various processes that can affect learning. In this work, an algorithm was used to try to manipulate and direct the emotions of the students. However, we need more data to be able to draw a conclusion related to the manipulation of emotions during learning CT concepts. There are several limitations with the existing study that need to be addressed in future work.
First, the socioeconomic status of students is not considered to determine if this influences performance and motivation to learn. A second limitation is a short period used in the intervention sessions with the students. Since the objective of the ILE is to teach CT, longer periods should be considered, which are suitable for a short and long-term learning process. Thus, we need to do more experiments with a greater amount of exercises, more students, extended learning periods, and use students from schools with different backgrounds (e.g., private schools).
