2.1 HPC Cluster Architecture

The first step is the implementation of an HPC cluster. Initially it was built with the following characteristics: 7 nodes with Intel Core i7 processors at 3.6 Ghz, 12GB of DDR4 memory, SATA III hard disk at 2 TB at 7200 RPM and average reading speed at 156 MB/s, a GigabitEthernet network card and a 24-port Gigabit Ethernet switch. CentOS version 7 was implemented as the operating system. As well as the xfs local file system and the Lustre parallel file system in conjunction with ext4 (figure 1)

Fig. 1 HPC Cluster Architecture 

The xfs was used for the cases where the CPU implementations that by default come with the Dlib ^[5] and OpenCV frameworks are compared; It is noteworthy that here there is no modification to the applications, so the image is stored and processed in a single node, thus preventing access to remote servers.

Part of our parallel strategy is the data location, therefore we define the use of Lustre as a parallel file system and likewise, a collection of images¹ of multiple sizes was used. The following configuration was implemented:

2.2 Data Location

In this second step, the goal is to distribute the image across multiple data servers. Lustre operations are implemented using a client/server model where clients send Input/Output operation initially to metadata servers to obtain the parameters of the objects (offset, size, id and others) as well as the type of operation (write/read) to be executed.

If a write operation is requested, lustre allows the image located in the client's local buffer to be sent in parallel across multiple object storage targets (OSTs) in a round-robin fashion (Figure 2).

Fig. 2 Write image file 

That is, an image can be divided into multiple pieces (1) that will be sent and stored in different OSTs within the luster system.

Once the image is located on disk, it can be accessed through a read operation directly from the OSTs:

2.3 Facedetector_MPI Description

a. Reading Image

The third step aims to describe the application developed. Our tool is responsible for carrying out an operation of reading the image to the object servers, this request is made in grayscale (figure 3) regardless of whether it is color or not. It is noteworthy that we chose to perform the operation using a single 8-bit channel instead of the three channels (RGB) used by other tools; since it allows us to avoid any inconvenience when manipulating the pixels, in addition to reducing the access time to it. Remember that an image is just an array where each cell represents a pixel, so reading the image is saved in a variable of that type.

Fig. 3 Read Image file 

b. Scaling Image

Once the image is loaded in gray scale, the size is extracted in pixels (width and height). Subsequently, one of the contributions that improve our tool is the implementation of a function called scalar_img(). This function is responsible for scaling the image to a factor of 2 when the image is small (≤1920 and ≤1080).

This allows the face detector to search for faces around 30×30 pixels allowing to reduce the search time and improving the level of accuracy. The second occurs when the images are of a larger size (≥4096 × ≥2160), then a factor of 0.5 is used, which will reduce the image to 50% of its original size. If none of the above conditions are met, the image remains unchanged. A look at the scaling algorithm can be seen in the algorithm 1.

The algorithm 1. scalar_img description 

c. Image Dispersion

A further contribution on this work is to allow an increase in the parallelism at the process level. The use of the message passing interface library is implemented for this matter; that is, the application is replicated on the multiple processing nodes.

The division of the image (size_img/num_procs) is given horizontally (figure 4a). This is due to the fact that such division allows to maintain a greater number of traits or particular characteristics such as the eyes and nose of a face compared to a vertical division (figure 4b), where the features or signs of the faces can be affected by the cuts of the image (figure 5), which is translated into an increase of false negatives. In other words the faces are not detected even when they are present so the success rate is in the detection of faces is reduced.

Fig. 4 Image dispersion 

Fig. 5 Division of a face 

A scatter routine (figure 6) is used to designated root process (lustre client 01) sending data to all lustre clients (processes) in a communicator, in order to sends chunks of an image to each node to process the chunk of the received image.

Fig. 6 Scatter routine 

2.4 Face Detection

In this step, the objective is to explain how face detection is carried out. It takes advantage of the multi-threading offered by MPI, a big advantage of MPI is the ability to specify noncontiguous accesses in memory, so the parallelization is even greater. Each of the nodes executes the face detection algorithm on the piece of image that was assigned to it.

The Haar-like feature classifier was used for front faces. It is noteworthy that such a classifier is included in the OpenCV implementation and was previously trained with multiple images.

It should be noted that the process for training a classifier is that the sample images called positive include these characteristics, as well as different pixel sizes.

In addition, to complement the training it is necessary to use images that do not contain faces. For the extraction of the Haar characteristics, each of these has a unique value that is obtained by subtracting the sum of the light parts from the sum of the dark parts, with which eyes, nose and lips can be identified.

To reduce the computational cost required, an integral image ^[3] is implemented, like Crow's summed area table ^[13]. Wherein, the value of i (x, y) in the summed area table is the sum of all the pixels to the left and above (x, y), see equation (2):

After loading the classifier and generating the matrix where the image is stored, a new vector is created where the detected faces will be stored. In order to calculate the minimum requirements to preserve the possible faces detected, the minimum and maximum size required (30×30) is defined. The algorithm detects faces of different sizes on the image and draws a box on the detected face.

A face outside the indicated parameters is ignored. The preliminary result is sent to the master or main node to show the result (figure 7) through a gather routine which takes elements from each clients (processes) and gathers them to the root process. Subsequently these are ordered by the rank of the process that each of them received.

Fig. 7 Gather routine 

The results obtained will show the total number of faces detected (FD), as well as the possible false positives (FP) and false negatives (FN) of each image; This allows us to calculate the success rate (HR), using the following expressions (3, 4), where A is the number of successes:

3.1 Image Read Time Performance

In the executions made in xfs (Dlib and OpenCV) the client accesses his local filesystem; There is therefore no remote access. In contrast, in executions in Lustre (Facedetector_MPI), the client accesses two remote servers (table 1).

As you can see the reading accesses with Lustre (figure 8 and 9) in the case of small files (<1MB) present a significant penalty against local accesses, so there is no significant improvement. The results shown in the figures do not reflect the advantage of accessing multiple servers in parallel. It should be noted that in the tests performed the servers only attend to the client's requests, so they are not running any other application.

Fig. 8 Throughtput 

Fig. 9 Latency 

Therefore, there is the conviction that to increase the performance to a better extent the implementation of a greater number of servers is necessary and this is noticed when the image size is larger than the stripping unit (1MB).

An increase in bandwidth is seen; This is due to the distribution of the image on 2 servers, since we will have multiple processes in parallel, simultaneously accessing part of the image, reducing access times by approximately 50% compared to the implementation in OpenCV and being similar in Dlib although the latter was optimized at a level 3 (-O3).

Better still is the fact of being able to manage an image with different clients, because each one of them can access a part of the image managing to increase the parallelism even more and consequently a reduction in access times.

3.2 Face Detector Algorithm Performance

Table (2) shows the average execution times (5 per image) obtained in the face detection algorithms that comes by default in Dlib and OpenCV, as well as the results of the modified algorithm in Facedetector_MPI for 2 and 4 clients.

When analyzing the results, we realize that there are some differences in the execution times of the algorithm. Figure 10 shows how the operations for smaller images in some cases are similar for the 3 implementations.

Fig. 10 Run time of face detector algorithm 

However, as the image grows, it is possible to appreciate how the execution time is considerably reduced in Facedetector_MPI.

This increase in performance is achieved thanks to the interaction of multiple processes that execute in parallel the algorithm of face detection on the assigned piece of image. You might think that this increase in benefits compromises the success rate of the face detection algorithm, however this does not happen, and it is here that the implementation of scaling becomes relevant.

3.3 Hit Rate

We take some final measurements and add a comparison between Dlib, OpenCV and Facedetector_MPI with 2 and 4 clients. Table (3) shows the lowest number of faces in the images (TF) (4 to 128), as well as the results obtained in: total number of faces detected (FD), false positives (FP) and false negatives (FN) of each of the implementations.

If we look at the data, we can realize that even in some cases, the number of faces detected is greater (FP) with respect to TF and this in the case of OpenCV adds more time in the execution of the face detection algorithm.

This is because some characteristics have been found within the image that are interpreted as faces, but they are not. To prevent this from happening in our implementation, we take advantage of the previously designed image scaling in order to better identify the characteristics of the faces and thus reduce the number of false detections.

Figure 11 clearly shows how the hit rate is similar in 5 (student01, student02, astro, stair and graduation) of the 9 cases having a hit rate between 94 and 100% over the 3 implementations, However, Facedetctor_MPI it is superior (gpl01, gpl03, isum) about 40 to 60% effective in 3 cases and only one (gpl02) shows a 20% reduction compared to the other implementations.

Fig. 11 Hit rate 

Although the results show certain differences, we can see how the added parallelism strategy improves performance and exploits existing computing resources, without compromising the effectiveness of face detection on an HPC cluster.