nueva página del texto (beta)
Inglés (pdf)
Artículo en XML
Referencias del artículo
Traducción automática
Enviar artículo por email
Citado por SciELO 
Similares en
SciELO 
Permalink
1 Introduction
The scheduler is a crucial component of the Linux kernel responsible for choosing the next thread to run and allocating to a processor core for execution [35, 37, 1, 13, 22]. This process involves primarily a load-balancing procedure that provides the thread migration between the cores of a CMP system. For a modern Linux scheduler that is aware of CMP systems to carry out load balancing is extremely important and quite challenging.
“The load-balancing procedure is based on a number of criteria of varying relative importance. The scheduling algorithm policy determines the importance of each of the criteria. Unfortunately, it is impossible to design an algorithm that fits in all the criteria simultaneously; trying to improve performance according to one criterion would adversely affect the expected performance by another” [14].
Nowadays, the Linux scheduling policy is designed to obtain the best possible performance while ensuring a fair allocation of the processor cores’ time among the normal (non-real-time) tasksfn. It is known as Completely Fair Scheduling (CFS) policy [20, 24, 28]. However, this policy collaterally can cause a relentless execution of the load-balancing procedure and an excessive number of thread migrations [21].
It results in several disadvantages, such as an increased cache invalidation, scheduling latency, and power consumption [8, 21]. Let us briefly expose how the Linux scheduler’s main functions work to gain a better understanding of the leading role of load balancing in CFS performance. A modern Linux kernel scheduler is composed of two functions:
The
The
A task temporarily yields its core’s time without being blocked—the task remains in the TASK_RUNNING state—by calling the
The
RT tasks are assigned the highest static priorities in the system (by default range from 0 to 99) in order to receive enough processing time to meet critical time constrains.
CFS tasks are user tasks (including those of
A run queue (
On the per-core run queue a CFS subqueue
To balance CFS tasks, the Main Scheduler first checks the per-core
Once this pull-load balancing is done, the Main Scheduler picks the next task to run and performs the context switch. At this point, the Main Scheduler must be sure that there are no RT tasks in the current run queue waiting to be dispatched.
In order to be executed promptly, the Main Scheduler calls the function
Just after this push-load balancing is done, the Main Scheduler becomes preemtable and checks if the reschedule flag bit
Main Scheduler’s functions involved in load balancing are shown in Fig. 1. When a processor core is allocated to a new task, the previous task that was running on that core has either gone to the ready or blocked state [31], and can be migrated to another core when a new pull-load balancing operation is performed.
Whenever load imbalance is detected, the load-balancing procedure is triggered to distribute the system load among the cores in a homogeneous multi-core processor, which results in excessive task migrations and the consequent drawbacks mentioned earlier.
On the other hand, in accordance with the literature [6, 11, 33, 44], threads’ contention for shared resources on a multi-core processor is the major cause of system performance drop. This paper proposes and evaluates a proactive load-balancing (PLB) algorithm for Linux on CMP systems to avoid a decrease in performance due both to contention among the threads for shared resources and excessive thread migrations.
Our PLB algorithm keeps a high level of system performance by proactively averting contending threads from running concurrently as well as by reducing unnecessary thread migrations at runtime. We propose runtime-updated IPC thresholds, which are the basis of the operation of the proactive load balancing. Also, a complementary support algorithm to migrate threads on CMP systems is designed.
The PLB algorithm takes advantage of the Performance Monitoring Unit (PMU) accessible from each core of a modern multi-core processor, and the perf event tool, a powerful profiling subsystem included in the Linux kernel since version 2.6.31. To meet our proactive load-balancing criterion, our algorithm carries out the following actions at runtime:
-
Configuration of the per-core PMU counters to read different performance-event samples simultaneously at constant time intervals; namely, those corresponding to the Instructions Retired, Unhalted Core Cycles and Thread Migrations events.
Instructions Retired is the number of fully executed instructions and Unhalted Core Cycles is the number of cycles executed on the core (when the core was not in HALT state), i.e. it shows the total elapsed cycles. These events are sampled for each application thread as a part of the workloads launched separately.
-
Obtaining the Instructions Per Cycle (IPC) statistic which reflects the system performance by using the samples of the Instructions Retired and Unhalted Core Cycles events. In this way, IPC is obtained as follows:
Comparison of the subsequent IPC values with the current IPC-event threshold while the workload is running.
Reacting proactively to avoid performance ramp-down based on the result of this comparison.
These actions allow to proactively decide whether a running thread must be migrated to another core, and whether its current performance threshold value (previously-sampled IPC value) needs to be updated depending on the result of the comparison.
If a current thread’s IPC count is below of its corresponding performance threshold value, the thread is migrated. If this count is higher, then the current performance threshold value is updated to this new IPC count (i.e., runtime-updated IPC thresholds are used).
In this way, only when a runtime-updated IPC threshold is not reached, our algorithm triggers the migration of contending threads in order to find couples of co-running threads that do not contend (or contend as little as possible) for shared resources on the cores, and therefore, leading back to a high level of system performance.
We thereby say that the algorithm obeys a criterion that proactively avoids contending threads from running concurrently.
Threads composing each workload are initially bound to a single core (e.g., core0) as the startup configuration. Our PLB algorithm is implemented at user level, which is sufficient for the accurate assessment [2].
In Section 8, we present a comparative table of different workloads when they run on Linux, first using the original (unmodified) Linux load-balancing procedure, and then merging our algorithm into the Linux kernel. This table shows that the number of thread migrations is significantly reduced (by 43.8% on average) without harming system performance when the PLB algorithm is used.
Fig. 2 illustrates the architecture of the proposed PLB algorithm. It shows its main components (rounded boxes) interacting with a dual-core CMP system (squared boxes). Next, we briefly describe each component addressed with more detail throughout this article.
– Resource Monitoring through the PMU: The PMU (Peformance Monitoring Unit) included within each core is composed of a special set of counting registers (Section 3) that we have configured to count the number of instructions that are fully executed (Instructions Retired), elapsed cycles (Unhalted Core Cycles) and (Thread Migrations).
– Runtime Sampling (IPC): PMU registers are used to collect performance events sampled at regular time intervals at runtime (Section 3).
– perf event Kernel Subsystem: PMU registers setup, runtime performance-event sampling and registers reading are all done through perf event (Section 3).
– Proactive Load-Balancing Algorithm: Our algorithm uses runtime-updated IPC thresholds that indicate the minimum IPC values that must be reach at any sampling instant to hold the system performance at a steady high level.
Therefore, we rely on these threshold values to proactively decide whether to migrate a thread (Section 6).
– Thread-Migration Algorithm: If a thread needs to be migrated, our complementary support algorithm designed to migrate threads on a CMP system is invoked (Section 5). Next, the effect of this migration on the system performance is monitored and our PLB algorithm again decides whether to migrate a certain thread in order to maintain high system performance.
Table 1 summarizes the main features of the example CMP system used. The vanillafn Linux kernel version 3.1.2 is run on the Intel Corei5 660 processor with codename Clarkdale based on the Intel Westmere microarchitecture [7, 16] whose virtual cores (i.e., hyper-threading) have been disabled from the BIOS, thus having a CMP system with only 2 physical cores. The remainder of this paper is organized as follows: Section 2 briefly surveys related work. Section 3 describes at length the research framework used. Section 4 is devoted to workload selection.
Table 1 Main features of the example CMP system
| Processor | Intel CoreTMi5 660 @ 3.33GHz |
| Number of Cores 2 physical | |
| Number of Threads 4 (2 BIOS disabled) | |
| L1 Cache 2 x 32 KB | |
| L2 Cache 2 x 256 KB | |
| L3 Cache 4096 KB | |
| Micro-architecture | Intel Westmere (Codename Clarkdale) |
| Main Shared Elements | L3 Cache, IM and PCI-e Controllers, and DMI and FDI interfaces |
| PMUs | Core, uncore and offcore MSRs register sets |
| Memory Size | 4 GB |
| Memory Type | DIMM DDR3 Syncronous 1066 MHz (2 x 2GB) |
| Operating System | Ubuntu-GNU/Linux vanilla Linux kernel 3.1.2-SMP x86 32 |
| Compiler | gcc version 4.5.2-8ubuntu4 |
Section 5 unveils the design stages of our thread migration support algorithm. Section 6 explains in detail the implementation of the algorithm that embodies our proposed proactive approach to perform load balancing. Section 7 delineates the evaluation experiments for the PLB algorithm. Section 8 reports the results. Finally, Section 9 concludes the paper and provides an avenue for future work.
2 Related Work
The design of scheduling algorithms that are aided by statistics collected via multi-core architecture-specific performance monitoring counters at run time to avert shared resource contention has been proposed in previous research. For this, they use either the Oprofile [26] or the Perfmon2 [29] external monitoring tools that are no part of the vanilla Linux kernel.
These algorithms aim to minimize the contention for the different shared resources within a CMP processor, such as the L2 or L3 caches, the system bus, the instruction queue, the core itself, and so on, therefore improving the overall system throughput.
Next, we present some previous work that has been done to implement scheduling algorithms that tackle the problem of shared resource contention in today homogeneous multi-core architectures:
Zhang X. et al. [43] developed a flexible framework for Throttling-Enabled Multi-Core Management (TEMM), which efficiently finds an optimal hardware throttling configuration for a user-specified resource management objective.
“It can support a variety of objectives for fairness, quality-of-service, overall performance, and power optimization. Throttling configuration refers to the settings of the platform-specific registers involved with the duty cycle modulation and dynamic voltage and frequency scaling (DVFS) mechanisms, originally designed for power management within processors.
TEMM searches for a reference configuration based on model predictions, and iteratively refines the search with a broad set of previously executed configuration samples. This search stops when a high-quality throttling configuration that meets the objective is found”. Sáez J.C. et al. [30] designed a non-work- conserving framework (i.e., a core may be idle at any time) to improve priority enforcement based on statistical information collected through hardware performance monitoring counters (PMU).
“When multiple threads run simultaneously, the system tries to detect changes in the behaviour of high priority (HP) threads that comes from negative interactions with other low-priority (LP) threads. Those changes trigger CPU disabling actions that temporarily block the potentially incompatible LP threads”.
Herdrich A. et al. [15] adapted rate-based techniques (clock modulation and frequency scaling) that are employed to address power management and cache/memory Quality of Service (QoS) issues.
The QoS term refers to the ability to guarantee a certain level of performance. Basically, what they do is to regulate the time the core is active and/or its working voltage and frequency (DVFS technique) if it is running a low-priority task that harms the performance of a high-priority task due to system cache or memory contention.
Shi Q. et al. [32] proposed both a load-balancing algorithm based on the construction of scheduling domains by taking shared L2 cache into account and the design of load vectors to weigh the processor core’s workload. Their goal is to reduce L2 cache misses (so main memory accesses are also reduced), and therefore, decrease the total execution time of threads.
Lim Q. et al. [21] implemented an operation-zone-based load balancer to improve the performance of multi-core systems at runtime. It provides three multi-core load-balancing policies based on the CPU employment.
“The cold zone policy loosely performs load-balancing operations; it is adequate when the CPU utilization of most tasks is low. The hot zone policy performs load-balancing operations very actively, and it is adequate for high CPU use. The warm zone policy takes the middle between the cold zone and the hot zone”.
Our research work proposes a proactive approach to perform load balancing of software threads on homogeneous multi-core processors (i.e., CMP). Our proactive approach is primarily based on runtime-updated IPC thresholds that we devised and used in our decision-making model (Section 6) in order to reduce task migrations originated in the Linux scheduler.
On the example CMP machine used (Table 1), our PLB algorithm maintains two different threads from each workload running concurrently as long as it results in the least shared resource contention, and therefore, to the same extent, thread migration is reduced; thus helping to improve system performance.
Our work relies heavily on the performance monitoring subsystem of the Linux kernel, perf_event, to implement our routines that simultaneously monitor different performance events at runtime—thus providing valuable insight into how to use and configure perf_event.
In a first intance, we developed a complete workload-launcher tool that we used both to synchronously launch workloads made up of various CINT speccpu2000 benchmarks on the example multi-core system and to collect the resulting statistical data from a special set of performance-event counters located within each core’s PMU (Performance Monitoring Unit) in the CMP processor.
Our results show that the number of migrations performed on the application threads (benchmarks) that make up the workloads used is significantly reduced (by 43.8% on average) without degradation of performance when our PLB algorithm is utilized.
3 Research Framework
This section describes the research framework used for the implementation of our PLB algorithm. Our research framework consists mainly of both the Performance Monitoring Unit (PMU) included in each core of a multi-core processor and the perf_event profiling tool available in the recent versions of the Linux kernel.
They are used jointly to implement our IPC-based decision-making model as well as to design the procedure for carrying out properly the sampling of different performance events simultaneously at runtime, which are the essential parts of our algorithm. Programming details of the PMU counters and perf event are also explained in this section. On the other hand, the different major program elements which make up the scheduler such as its main data structures and fuctions have been studied at length directly from the Linux kernel.
Basically, we mostly used the TOMOYO Linux Cross Reference [9], a very helpful web-based tool, to navegate and analyze extensively the vanilla kernel scheduler source code. Also, the Open MPI Portable Hardware Locality tool
3.1 The Performance Monitoring Unit
Processors supporting Intel 64 and IA-32 architectures have a Performance Monitoring Unit (PMU) consisting of a collection of Performance Monitoring Counter registers (PMCs) and Performance Monitoring Event registers (PMEs) [2, 25, 11, 17, 39]. PMCs and PMEs are implemented as Model Specific Registers (MSRs). They are accessed via the RDMSR and WRMSR instructions.
PMCs are used to collect event counts or serve as hardware buffers, so they are named Counter MSRs. PMEs are used to indicate what events need to be monitored, so they are named Event Programming MSRs. The number of MSR registers that make up the PMU depends on the processor model.
A monitor is defined to be a combination of a PME for the configuration and one or more PMC registers for collecting data. A counting monitor need only one PMC register.
Therefore, the counting monitors can each be programmed to count one event at a time. A monitoring session consists of several steps that must be followed to collect valid measurements. Those steps can be summarized as follows [25]:
i) Program the monitors (paired PME and PMC registers).
ii) Enable the monitors.
iii) Run the code to be monitored.
iv) Disable the monitors.
v) Collect results.
Each core built on a CMP chip has its own register bank which contains the MSR registers that make up the PMU [17]. The PMU and other registers in the register bank are grouped together to form the architectural state of a process thread.
That is, the architectural state is the set of registers within each core in the CMP processor that holds the state of its respective running subprocess. Therefore, on a CMP processor, each running thread has its own independent architectural state.
When a thread migrates from one core to another, its PMU state also moves. That is, counts of different events collected in the MSR registers in the source core’s PMU are replicated into the same type of registers in the target core’s PMU. It is this important design feature of multi-core processors that allowed us to implement code to follow a thread from one core to another without loosing information on the accounts of events.
3.2 The Linux perf event Kernel Subsystem
Perf_event is a performance monitoring tool merged into the Linux kernel from version 2.6.31 [10, 40, 41]. The principal goal of perf event is to provide Linux with the support needed to effectively utilize the PMU, thus allowing an advanced performance analysis.
Currently, it is a powerful kernel subsystem increasingly used in the research and development of new computer systems such as multi-core architectures. Support for the latest architectures is added according to new kernel versions.
The perf_event tool includes plenty of commands to collect and analyze performance and trace data. It can measure both hardware and sofware events. Software events are those that originate in the kernel. Some examples are: the number of context-switches, cpu-migrations or page-faults.
Hardware events are micro-architectural events such as the number of elapsed cycles, instructions retired, L1 cache misses, etc. The perf event interface (API),
By means of this system call, the PMU registers can be read during the execution of an application. Therefore, event samples can be obtained at runtime. This system call has the following prototype:
A description of its arguments can be found in the perf event documentation (file
It then returns an integer which is the file descriptor
Next, we show how the
An excerpt from our code to measure these events is shown in Table 2. Line (1) specifies the
Table 2 An excerpt from our code to collect instructions, cycles and cpu-migrations events in a single monitoring session
| (1) | attr.type = PERF_TYPE_HARDWARE; |
| (2) | attr.size = sizeof(struct perf_event_attr); |
| (3) | attr.disable = 1; |
| (4) | attr.config = PERF_COUNT_HW_INSTRUCTIONS; |
| (5) | fd = perf_event_open(attr, pid, 0, -1, 0); |
| (6) | fd1 = perf_event_open(attr, pid, 1, -1, 0); |
| (7) | attr.config = PERF_COUNT_HW_CYCLES; |
| (8) | fd2 = perfevent_open(attr, pid, 0, fd, 0); |
| (9) | fd3 = perf_event_open(attr, pid, 1, fd1, 0); |
| (10) | attr.type = PERF_TYPE_SOFTWARE; |
| (11) | attr.config = PERF_COUNT_SW_CPU_MIGRATIONS; |
| (12) | attr.exclude_kernel = 0; |
| (13) | fd4 = perf_event_open(attr, pid, 0, fd, 0); |
| (14) | fd5 = perf_event_open(attr, pid, 0, fd1, 0); |
Line (2) sets the
Lines (5) and (6) define the system calls used to count the number of instructions retired on both cores (the third argument is
Next, both the
Line (12) shows the new value for this attribute. The system calls to measure
4 Workload Selection
Workloads made up of different combinations of three CINT speccpu2000 benchmarks [36] were previously characterized using the vanilla Linux kernel 2.6.32.10 patched with the Perfmon2 profiling tool [29] on an Intel Core2 Duo E6550 multi-core processor [16].
Table 3 presents our workloads and Fig. 3 shows the bar graphs that result from their characterization using some key metrics. These metrics are:
a) Instructions Per Cycle (IPC): fully executed instructions divided by the total CPU cycles: IPC = instructions retired / unhalted core cycles
b) L2_MISSES: data and instruction misses at second level (L2) cache. On the Intel Core2 Duo processor, the L2 cache is a unified cache that is shared by both cores to serve L1 cache misses of instructions and data.
c) BUS_TRANS_MEM:BOTH_CORES: Memory Bus Transactions due to both cores. That is, memory requests initiated by any core on the system bus.
d) INST_QUEUE:FULL: cycles during which the instruction queue is full. The instruction queue is a unit where instructions wait until they are ready for execution. An instruction is ready for execution when its operands have already been computed.
Table 3 Workloads made up of CINT speccpu2000 benchmarks
| CINT2000 workloads | |
| W1 (gzip gcc mcf) | W11 (gzip bzip2 eon) |
| W2 (bzip2 gcc mcf) | W12 (bzip2 eon crafty) |
| W3 (eon gcc mcf) | W13 (gzip eon crafty) |
| W4 (crafty gcc mcf) | W14 (bzip2 gzip crafty) |
| W5 (gzip gcc bzip2) | W15 (eon gzip mcf) |
| W6 (eon gcc bzip2) | W16 (bzip2 gzip mcf) |
| W7 (crafty gcc bzip2) | W17 (crafty gzip mcf) |
| W8 (gzip gcc eon) | W18 (eon bzip2 mcf) |
| W9 (crafty gcc eon) | W19 (crafty bzip2 mcf) |
| W10 (gzip gcc crafty) | W20 (crafty eon mcf) |
As can be seen in Fig. 3, a smaller number of instructions per cycle is executed for workloads W1 to W4 and W15 to W20. Also, the number of L2 cache misses is too large for such workloads. Furthermore, both the number of memory requests and the number of cycles during which the instruction queue is full are also too large for these same workloads.
This indicates that the benchmarks composing workloads W1 to W4 and W15 to W20 contend with at least one of their co-runners for shared resources intensely. In particular: the L2 cache, the system bus and the instructions queue. Therefore, such workloads are regarded as best suited to carry out experiments in which a stress capacity for our CMP system is required.
5 Design of the Complementary Support Algorithm to Migrate Threads
The kernel uses the
First, this system call obtains a cpumask bitmap called
If the core to which the current task is bound is not part of the new mask,
This is done through the
Stage 1: The Linux kernel’s
The
We have called the improved function
Stage 2: Algorithm 1 shows our algorithm called mctopology (i.e., multi-core topology) which is designed to construct from an input mask in hexadecimal notation two domain masks of cores:
The input mask is devised to allow grouping cores that are physically adjacent (and so sharing resources, e.g., a common cache) into domain masks (i.e., scheduling domains) such that threads should preferably be moved between the cores in a domain. To pass the input mask to the kernel to be processed, the
In lines 10 and 11, the number of cores in the system (taken straight from the kernel variable
This section of code first tests if the bit that corresponds to the core at position i (cpui) is set in the default kernel
Depending on the result that has been stored in
As the number of cores increases, the setting of an input mask so that the selection of the adjacent cores be optimal is a matter of key importance.
Stage 3: An algorithm to update the current task’s cpumask bitmap with either the mask
Our
For our example dual-core system (which has two physical cores and hyperthreading disabled), the default kernel cpumask is 0x11 (both cores are online), so when the
In line 7, the current task descriptor is obtained first. In lines 8–17, a domain mask for the
On Linux, threads are always migrated from one run queue to another. Before being migrated, the kernel must suspend the execution of the thread running on the local core and save its task context (including performance statistics) accordingly. These actions are part of a context switching procedure performed in kernel mode. For a thread, its task context is obtained from its task descriptor (
The task descriptor contains all of the data needed to keep track of the thread, whether it is running or not. Some of the primary fields it includes arefn: A
The thread’s virtual address spacefn is divided into several regions of type
Thus, since the thread’s task descriptor is moved from the local run queue to the target run queue, the thread’s task context can be restored and its execution resumed (a thread is always scheduled from the run queue on to the core in user mode [18]).
CPU cycles consumed in all this moving that the kernel does to migrate threads result in pure overhead, because no useful work (IPC) is done [34, 8, 22, 23].
In this light, our PLB algorithm maintains two different threads from each workload running concurrently as long as it results in the least shared resource contention, and therefore, to the same extent, thread migration is reduced; ultimately preventing system performance from declining.
In the next section, we first illustrate sequentially how our algorithm works, and then explain its pseudocode line by line.
6 Proactive Load-Balancing (PLB) Algorithm
6.1 Sequential Diagram
A sequential diagram to depict the interaction of the various threads involved in our algorithm is shown in Fig. 4. In this diagram the leading function (parent thread) called
Next, the child thread starts executing in parallel to the parent until the parent thread stops and waits for the child to complete the preparatory steps prior to launching the application (which will be detailed in the next section) before carrying out the counters’ setup. Using an inter-thread communication technique known as pipes, the parent and child get synchronized by sending messages between them. A null message is sent from the end of a pipe that gets closed, to the end that remains open to read the message. The
This is shown through the points A-C. When the parent thread receives the null message from the child, the parent resumes execution and performs the counters’ setup. Straight afterwards, the parent thread first resets and enables the counters and then starts measuring the runtime parameter through the
Meanwhile, the child thread waits for the parent to finish these steps. Right after the parent thread has started measuring the runtime parameter, it sends a null message to the child by closing one end of a second pipe (D).
The child thread reads the message at the other end of this pipe, resumes execution, and launches the application by means of the
The
Based on these statistics, the parent thread decides whether it is convenient to migrate the current application thread. When the application ends, the terminated status for the child specified by its pid is immediatly available.
Then, the parent thread leaves the
Finally, the parent thread saves the collected statistics in dedicated files, prints them on the screen, and exits. See points E-G. Next, we present our proactive load-balancing algorithm split into two parts: Algorithm 3 and Algorithm 4.
6.2 Detailed Description
Our PLB algorithm requires a tight synchronization between the parent, child and application threads as exposed so far. Algorithm 3 details the synchronization between these threads as well as the first part of the
In lines 4–8, the parent thread spawns a child thread through the
At this point a branch occurs: in line 9, the child begins executing the preparatory steps prior to launching the application thread, and in line 14, the parent thread waits for the child to finish.
In line 10, just after the child thread has completed these steps, it sends a null message to the parent by closing one end of pipe 1. In line 11, the child waits for the parent thread to carry out the counters’ setup.
Then, in lines 15–18, the counters are setup, reset and enabled by the parent thread, whereupon it begins measuring the runtime parameter. In line 19, the parent sends a null message to the child by closing one end of pipe 2, so in line 12, the child launches the application by means of the
As mentioned earlier, the child performs a couple of preparatory steps prior to launching the application thread which consist in:
-
First calling a dummy
Setting the close-on-exec flag
On the other hand, in line 20, the control variable
The decimal part of the IPC values thus obtained is truncated to its hundredth part; this in order to make them more meaningful. Algorithm 4 presents the second part of the
In lines 39–49, for the first pass inside the
If so, the new IPC values are assigned to the IPC-threshold variables (i.e., runtime-updated IPC thresholds are used). On the other hand, in lines 51–62, if the new IPC values are lower than the IPC-threshold values, it means that a significant contention exists between the co-runner threads.
So the current IPC-threshold values are maintained. In line 51, the IPC value of the thread running on core0
Similarly, in line 57, the IPC value of the thread running on core1
As previously stated, the optimal couples of co-runner threads are searched in this way at runtime. The
Finally, when the application terminates, the
7 Evaluation Experiments
This section details the steps performed to evaluate our proactive load-balancing algorithm: First, workload implementations of CINT speccpu 2000 benchmarks [36] which simulate application threads are carried out. Three-benchmark workloads that have adequate stress capacity for the example multi-core system used (Table 1) are employed. Second, perf_event is used to configure the PMU built into each processor core in order to collect the instructions retired and unhalted core cycles hardware events from which we calculate the Instructions Per Cycle (IPC) statistic.
As explained in earlier sections, IPC is the primary metric we employed to manage the migration of process threads. Although there are several other events collectable such as L2 cache misses, rejected L2 cache requests (by the bus queue), completed memory transactions on the system bus, and so on, IPC is a good performance metric to keep simplicity in our conception. The PMU is also configured to collect the cpu-migrations software event at the same time (as described in Section 3).
Third, as mentioned in Section 2, we developed a workload-launcher tool in order to synchronously launch the implemented workloads on the CMP system. The workload-launcher picks the different bechmarks that make up a workload and launches them simultaneously to execution. As soon as a workload finishes, the benchmarks that compose the next workload are launched. When the last workload is executed, the workload-launcher starts over by executing each workload again in a second round. It stops after 10 rounds are completed (i.e., each workload is run 10 times). As discussed earlier, 10 workloads were selected from the 20 that we had previously characterized to stress the CMP system used.
Fourth, in the manner described in the previous step, each of the selected workloads is run 10 times on Linux without modifying the scheduler, and the number of instructions retired, unhalted core cycles and thread migrations are counted at runtime. Fifth, our PLB algorithm is then merged into the Linux scheduler and each of these workloads is run 10 times again. The same events are counted at runtime. Finally, the statistics values obtained are plotted on a bar graph and analyzed. This bar graph is shown in Fig. 5.
8 Results
The number of thread migrations obtained after running each workload 10 times on the example CMP system with the Linux OS installed, first with the vanilla (standar) kernel and then with the PLB algorithm merged into the Linux kernel scheduler is plotted in the bar graph in Fig. 5, which also shows the benchmarks that make up each workload. From this figure it can be seen that the number of migrations performed is significantly reduced when using proactive load balancing.
Table 4 shows in detail the percentage by which the number of migrations decreased for each constituent benchmark (mimicking a software thread) of the workloads used. In this table, below each benchmark, there are three small boxes. In the first box, we have the resulting number of thread migrations for the vanilla Linux scheduler, in the second, the corresponding number for our proactive load-balancing algorithm, and in the third, the percentage by which the migrations decreased. Our PLB algorithm reduces the number of thread migrations by up to 71.7% (
Table 4 Percentage of reduction in the number of thread migrations for each workload
| Thread Migrations | |||||||||
| Workload | Benchmarks (threads) | ||||||||
| W1 | gzip | gcc | mcf | ||||||
| 278 | 136 | 51.1% | 54 | 21 | 61.1% | 348 | 181 | 47.8% | |
| W2 | bzip2 | gcc | mcf | ||||||
| 247 | 137 | 44.5% | 59 | 39 | 35.4% | 372 | 181 | 51.4% | |
| W3 | eon | gcc | mcf | ||||||
| 112 | 62 | 44.6% | 45 | 25 | 44.4% | 317 | 240 | 24.3% | |
| W4 | crafty | gcc | mcf | ||||||
| 413 | 230 | 51.6% | 42 | 26 | 38.1% | 363 | 137 | 62.2% | |
| W15 | eon | gzip | mcf | ||||||
| 134 | 94 | 29.9% | 248 | 150 | 39.5% | 328 | 146 | 55.5% | |
| W16 | bzip2 | gzip | mcf | ||||||
| 244 | 69 | 71.7% | 272 | 82 | 69.8% | 352 | 109 | 69.0% | |
| W17 | crafty | gzip | mcf | ||||||
| 252 | 228 | 9.5% | 223 | 103 | 53.8% | 268 | 146 | 45.1% | |
| W18 | eon | bzip2 | mcf | ||||||
| 126 | 78 | 38.1% | 217 | 123 | 43.3% | 348 | 201 | 42.2% | |
| W19 | crafty | bzip2 | mcf | ||||||
| 408 | 265 | 35.0% | 232 | 105 | 54.7% | 398 | 217 | 45.5% | |
| W20 | crafty | eon | mcf | ||||||
| 279 | 317 | -13.6% | 282 | 224 | 20.6% | 93 | 58 | 37.6% | |
Therefore, our algorithm is forced to migrate
Proactive load balancing:
Thus, the total reduction in the number of migrations
That is, in total there were 3224 fewer migrations.
Therefore, on average the percentage by which the number of migrations was reduced
Without degradation of performance.
9 Conclusion and Future Work
We have presented our proactive load-balancing (PLB) algorithm designed to avoid performance drop in CMP systems due both to an excessive number of thread migrations and shared resource contention. Our PLB algorithm proactively decides whether a running thread must be migrated from its current core to another active core based on performance data read at runtime from system counters (PMU) that we configured to collect counts of selected events.
While the software threads are running concurrently (co-running threads) on a CMP processor, the Instructions Retired and Elapsed Cycles events are collected and used to obtain the IPC parameter which shows the system performance. In the same way, the cpu-migrations event is also gathered to know the number of migrations performed on each thread.
According to the literature, shared resource contention by co-running threads is the most important cause of performance drop. Hence, our PLB algorithm is primarily designed to proactively avoid resource contention. On the example CMP machine, our algorithm maintains two different threads from each workload running concurrently as long as it results in the least shared resource contention, and therefore, to the same extent, thread migration is reduced.
Our PLB algorithm avails itself of the Linux kernel’s perf_event subsystem to configure each core’s PMU, read event counts and monitor software threads. The perf_event subsystem represents an easy access to hardware counters to Linux, which are a key resource for improving system performance. Unfortunately, there is a lack of literature and limited online documentation available for this monitoring tool.
Therefore, an important aspect of our research work is that we have excelled at dredging up most of the vague and obscure configuration facts, and thus shedding light on how to use and set up perf-event; namely, how to set it up in order to count different events simultaneously at runtime. We have introduced a comprehensive view of the methodology used for conducting research to adapt an operating system such as Linux to the modern multi-core architectures, which is a theme of great relevance and interest among computer scientists today. Overall, our results have shown that the number of migrations performed on the threads (benchmarks) that make up the workloads used is significantly reduced (by 43.8% on average) without harming system performance when our proposed PLB algorithm is utilized. To design A.I. algorithms that can be merged into Linux to perform smart scheduling of co-running threads in multi-core architectures so that system performance improves is an interesting avenue for future work.
