The prediction of processor performance has important reference significance for future processors. Both the accuracy and rationality of the prediction results are required. The hierarchical belief rule base (HBRB) can initially provide a solution to low prediction accuracy. However, the interpretability of the model and the traceability of the results still warrant further investigation. Therefore, a processor performance prediction method based on interpretable hierarchical belief rule base (HBRB-I) and global sensitivity analysis (GSA) is proposed. The method can yield more reliable prediction results. Evidence reasoning (ER) is firstly used to evaluate the historical data of the processor, followed by a performance prediction model with interpretability constraints that is constructed based on HBRB-I. Then, the whale optimization algorithm (WOA) is used to optimize the parameters. Furthermore, to test the interpretability of the performance prediction process, GSA is used to analyze the relationship between the input and the predicted output indicators. Finally, based on the UCI database processor dataset, the effectiveness and superiority of the method are verified. According to our experiments, our prediction method generates more reliable and accurate estimations than traditional models.
Currently, processor performance prediction plays a pivotal role in dictating the evaluation of new processor designs. As with the continuous improvement of processor performance, processor design and evaluation have become increasingly complicated [
According to the modeling mechanism, performance prediction methods can be divided into three categories: a black-box model, a white-box model and a gray-box model [
The uncertainty of processor parameters brings challenges to traditional modeling methods. The black-box model can ignore the uncertainty of input and obtain accurate prediction value. However, the reasoning process is opaque, the results are not traceable, and there is no effective insight into the impact of each input on the structure. The white-box model is affected by the system structure and system environment parameters, and the model accuracy cannot be guaranteed. Therefore, the gray-box model is more suitable for the performance prediction of processors, which can guarantee the accuracy and interpretability of the model at the same time.
Belief rules are gray box models extended from IF-then rules and traditional D-S theory. It uses evidential reasoning (ER) as an inference engine to establish the uncertain relationship between input and output [
To verify the interpretability of the prediction method, a sensitivity analysis method is applied in this study. As an uncertainty analysis technique [
The highlights of this study mainly include the followings: (1) A performance prediction model based on HBRB-I is proposed, which may solve the uncertainty of input parameters and make the model inference process transparent. (2) An optimization structure with interpretability constraints is constructed, which ensures the accuracy of the model and increases interpretability of the model. (3) Sensitivity analysis is used to verify the interpretability of the model.
The remainder of this paper is organized as follows. In the second part, the problems in the process of processor performance prediction are analyzed, and a performance prediction method based on HBRB-I and GSA is proposed. The third part constructs the performance prediction model of processor. The definition of interpretable model, the establishment and optimization of prediction model and the analysis of parameter sensitivity are described. In the fourth part, the effectiveness of the method proposed in this paper is verified by a case, and the experimental conclusion is obtained. In the fifth part, the core steps of the method are summarized, and the future work is prospected.
Interpretable performance prediction methods are largely implicated in the design of future processor architectures. The traditional modeling method of performance prediction is affected by the system structure and environmental factors. It is difficult to improve the interpretability of the model while ensuring the accuracy of the model, and the output process cannot be deduced and proved. Therefore, in this section, we build a reasonable and accurate performance prediction model to assist gaining insights into the involved factors of processor resource usage and provide a reliable reference for performance analysis. Along with the problems that may occur in the actual project, we make the following summary.
Question 1: How can the interpretability of performance prediction models be defined? The interpretability of the model is affected by multiple factors. To ensure the interpretability of the processor performance analysis process, based on general interpretability criteria and the actual characteristics of the system, the interpretability of the performance prediction model is defined to meet the basic requirements of the process. The requirements are described as follows:
Question 2: How can a completely and reasonably interpretable HBRB performance prediction model be built? The definition of interpretability is satisfied as the base. Then, this model fully takes into account the causal relationship between inputs and outputs. The initial parameters combined with expert knowledge are set. Finally, the reasoning process of HBRB-I is constructed based on the above preparations. At the same time, to ensure the accuracy of the model, interpretability constraints are added to build the parameter optimization process of this model. The model inference process in this paper is described as follows:
Question 3: How to verify model interpretability? The interpretability of model parameters is affected by the input and output indicators. To clearly understand the uncertainty of the input source and output division, global sensitivity analysis on the input and output of the prediction model is conducted. Then the reasoning process of sensitivity analysis is established. It helps enhance the traceability of the model. The relationship between the model indicators is verified by calculating the sensitivity index of the indicators. The global sensitivity analysis model can be described as follows:
To solve the problem described in Section 2, a performance prediction model based on HBRB-I is proposed with the relative performance of processors as the research target. Besides, the global sensitivity analysis method is used to quantify the uncertainty of the input and output of the model. It helps a lot to verify the relationship between the indicators. The interpretability of the model can be better explained. As shown in
As an expert system [
(1) Integrity of input and output metrics
The integrity of the indicator data is conducive to the reasonable definition of the relationship between the indicators. According to the modeling causal attributes, the indicators can be divided into input indicators and output indicators. The difference between the input and output of the model will directly affect the accuracy of the results. For performance prediction models, at least one reference value should be set for each indicator, and at least one rule should be activated. Additionally, when performing rule reduction, it is necessary to ensure that each input corresponds to at least one rule [
(2) The rationality of rules
The rationality of rules is the basic requirement of model interpretability. Since the setting of the initial rules comes from expert knowledge, the rationality of rules is guaranteed. In addition, this criterion should also be satisfied when performing model optimization.
(3) Normativeness of rule matching degree
The matching degree of the rules represents the distribution characteristics of the indicators that affect the performance of the processor. To ensure the interpretability of the system distribution, the matching degree of each reference value in indicator space is normalized. Usually, the sum of the matching degrees of the reference values of each indicator data should be between [0,1], which is expressed as
(4) Structures and parameters have physical meaning
The overall structure and initial parameters of the model should have actual physical meaning. On the one hand, BRB is established by logical derivation. This conforms to the system principle. On the other hand, the initial parameters are dominated by belief rules. They include attribute weights, rule weights, activation weights, and beliefs [ a. Attribute weight: the importance of the premise attribute relative to other attributes b. Rule weight: Indicates the importance of the rule relative to other rules c. Activation weight: the degree to which the corresponding rule is activated by the input d. Belief: how well the rules are converted into processor performance evaluation levels
(5) Distinguishability of the range of values
The input reference value of the model should be reasonably divided. The performance status evaluation level is represented by different level space ranges. There should be a distinction between different spatial ranges, so that different divisions of the corresponding meaning are different. It can help to meet the actual needs of the system.
(6) Normativeness of Information Transformation
Information conversion refers to the process of converting input and output information into belief distribution. To ensure the equivalence of the information conversion process is to maintain the characteristics of the original information. Yang et al. had developed a method to convert information based on rules and utility. This method could convert information reasonably and completely [
(7) Transparency of model inference
To maintain the interpretability of the rule base, BRB’s inference engine is required throughout the inference process. Besides, each reasoning process should be reasonably calculated and have an obvious causal relationship. This helps to ensure a clear description of the relationship between the input and output indicators. Based on the above conditions, the reliability of the model in practical applications can also be improved. As an inference engine for predicting model pairs, the ER algorithm can not only achieve clear inference and traceability of results, but also be able to explain itself.
In this section, the construction and optimization of the performance prediction model based on HBRB-I are defined and included as the following: (1) According to the mechanism analysis of the indicators, ER is used to evaluate the indicators; (2) Based on the evaluation results of the indicators, the HBRB model is constructed, and the detailed description and reasoning process of the prediction model are given; (3) Combined with WOA, a parameter optimization method is designed with interpretability constrained.
According to the mechanism analysis of the performance indicators, there may be correlations between indicators that affect the performance of the processor. If an indicator is analyzed individually in the process of performance evaluation. It is independent and unconvincing. In addition, the input and output index data in the process will be incomplete when indicators are blindly screened. Eventually, the evaluation results will produce errors. Therefore, it is proposed to use the ER algorithm to build the index evaluation model. The main implementation process is as follows, and
As an expert system with the ER algorithm as the inference engine, the essence of BRB is a gray-box modeling type. Its system structure is composed of expert knowledge summarized by historical observation data and corresponding rules [
The reasoning process of the HBRB model is mainly divided into five steps. First, expert knowledge is combined to convert the input value into the form of belief distribution. Second, the matching degree of the belief rules should be calculated. Finally, ER is used to obtain the output utility value.
The parameters of the initial HBRB performance prediction model are derived from the actual system and expert knowledge. Considering the limited level of expert knowledge, it is difficult to accurately describe the performance state of the processor. Therefore, it is necessary to update the initial parameters of the model in conjunction with the optimization algorithm. In addition, the interpretability of the BRB will be destroyed in the process of parameter optimization. To ensure the interpretability and accuracy of the prediction model, interpretability constraints are added to the WOA algorithm. Therefore, the parameters with interpretability constraints are proposed. The model is shown in
First, the optimization objective function is given according to expert knowledge and the mean square error obtained from the actual training output, as shown in
Based on the above analysis, the problem of BRB model optimization is a global optimization problem with constraints. As a metaheuristic optimization algorithm, the whale optimization algorithm is widely used in engineering [
The current number of iterations is represented by t; the coefficient vectors are
In the process of constructing the HBRB-I performance prediction model, the parameters of the model are derived from expert knowledge and the analysis results of input and output indicators. The expert knowledge is given according to the characteristics of the actual system and can be well explained by itself. The interpretability of the results of subjective index analysis needs to be further verified. Therefore, this section performs a global sensitivity analysis on the input and output metric parameters. Then the interpretability of the model derivation process is demonstrated.
The GSA considers the effect of the interaction between variables on the output when all parameters are changed at the same time. GSA mainly includes the regression analysis method, Morris-based screening method, Sobol method based on variance decomposition, and extended Fourier sensitivity test method [
First, the prediction model function
Based on the above conditions, the partial variance and the total variance are defined. The ratio of the partial variance to the total variance indicates the degree of the parameters’ influence and their interactions with the target response. The relationship between them can be expressed as:
To verify the effectiveness of the proposed method, the computer hardware data of the UCI database is used as the main experimental data. The relative CPU performance data are obtained according to the description of processor cycle time and memory size. The data set contains 209 instances with a total of 7 attributes. The specific parameter descriptions are shown in
No | Parameter | Data type | Data range |
---|---|---|---|
1 | Machine cycle (MYCT) | Integer | [17,1500] |
2 | Minimum memory (MMIN) | Integer | [64,32000] |
3 | Maximum memory (MMAX) | Integer | [64,64000] |
4 | Cache (CACHE) | Integer | [0,256] |
5 | Minimum channel (CHMIN) | Integer | [0,52] |
6 | Maximum channel (CHMAX) | Integer | [0,176] |
7 | Relative performance (PRP) | Integer | [6,1150] |
In this section, a processor performance prediction model based on the UCI computer hardware data set is built.
a. According to the data distribution characteristics of the original data set, the trend changes of the indicators are summarized, and the indicators based on the change analysis are classified. |
b. The physical meaning of the classification indicators is considered, and the classification of indicators are rationally adjusted. |
a. The parameter settings of the evaluation indicators are given, and the evaluation utility value are calculated according to ER. |
a. Combined with Section 3.1, the initial definition of the model is completed. |
b. According to the evaluation results and expert knowledge, the index reference value, reference level and performance status are given. |
c. The relationship between the reference value and the evaluation status is analyzed, and the initial HBRB-I is established. |
a. Interpretability constraints are determined according to interpretability criteria and optimization parameters are set. |
b. The optimization of the initial HBRB-I is completed, and the belief parameter table is given after WOA optimization. |
Combined with the change law of the actual value, the relationship between the processor indicators is analyzed. The forecast indicator trend analysis is shown in
Machine cycle | Storage | Cache | Channel | Relative performance |
---|---|---|---|---|
Increase | Constant | Constant | Constant | ↓ |
Constant | Increase | Constant | Constant | ↑ |
Constant | Constant | Increase | Constant | ↑ |
Constant | Constant | Constant | Increase | ↑ |
Based on the above index classification results, the ER algorithm is used for index evaluation. The reference values are set as shown in
Class | Indicator | Weight | X1 | X2 | X3 | X4 |
---|---|---|---|---|---|---|
Sr | Minimum memory | 0.332 | −1 | −0.9 | −0.6 | 1 |
Maximum memory | 0.211 | −1 | −0.9 | −0.6 | 1 | |
Cache | 0.457 | −1 | −0.9 | −0.7 | 1 | |
Cr | Minimum channel | 0.505 | −1 | −0.9 | −0.6 | 1 |
Maximum channel | 0.495 | 1 | −0.9 | −0.7 | 1 |
Based on the above analysis, the initial HBRB-I establishes a belief rule, which is expressed as follows:
Very small (VS) | Small (S) | Middle (M) | Large (L) | |
---|---|---|---|---|
BRB0 | −1 | −0.96 | −0.8 | −0.12 |
BRB1 | −1 | −0.95 | −0.77 | −0.15 |
To avoid the problem of rule combination explosion, an interpretable hierarchical BRB performance prediction model is constructed. According to the reasoning process shown in
x | Very small (VS) | Small (S) | Middle (M) | Large (L) | |
---|---|---|---|---|---|
BRB0 | St | 1.9 | 3.0 | 4.8 | 7.7 |
Ct | 1.9 | 4.0 | 5.2 | 7.9 | |
BRB1 | Pt | −1.0 | −0.95 | −0.88 | −0.32 |
Mt | −1.1 | −0.94 | −0.88 | 0.1 |
Considering the limitations of expert knowledge, the prediction of the performance state by the initial model is not sufficiently accurate [
Taking the initial parameters of BRB1 as an example, the belief constraints and the optimized belief are shown in
To verify the model interpretability, based on the HBRB-I performance prediction model, Sobol global sensitivity analysis of parameters is performed. The specific process can be described as follows:
MYCT | MMIN | MMAX | CACHE | CHMIN | CHMAX | |
---|---|---|---|---|---|---|
First-order | 0.2868 | −0.0952 | 0.2632 | 0.1519 | 0.0746 | 0.1824 |
Total-order | 0.6444 | 0.4604 | 0.1697 | 0.2212 | 0.2688 | 0.3483 |
Comparing the values of different sensitivities of performance parameters, it is found that the total sensitivity of the machine is significantly higher than other parameters. The first-order sensitivity is generally smaller than the total sensitivity value. The result indicates that a single index parameter has a small impact on the processor performance. The interaction between each index has a great impact on the performance.
To further explore how each index parameter affected the performance model, the linear fitting method is used to fit the functional relationship expression between the predicted performance value and single index parameter. This helps to study the correlation between the index parameter and the performance. Assuming that the index parameters are uniform linearly, distributed within the range during the fitting process, other parameters are averaged. The HBRB-I model is used to obtain the performance prediction value. In addition, the correlation coefficient between each index parameter of the original data set and the relative performance is calculated as
According to the experimental results, the correlation coefficient
In this section, the accuracy and interpretability of the model are mainly compared. The experimental results are combined to prove the effectiveness of the method.
To verify the effectiveness of the method, the method proposed in this paper is horizontally compared with four types of machine learning methods. They are backpropagation network (BPNN), radial basis function neural network (RBFNN), extreme learning machine (ELM) and random forests (RF). For comparison, each method uses the same number of training and test sets, and conducts 10 rounds of experiments. The fitting effect is shown in
WOA&HBRB-I | BPNN | ELM | RBFNN | RF | |
---|---|---|---|---|---|
Accuracy | 83.39% | 83.5% | 62.3% | 77.2% | 84.2% |
To better evaluate the performance of the proposed algorithm, the computation time and convergence of the proposed method are discussed. As shown in
The belief distribution of each rule is shown in
ER&HBRB-I | ER&HBRB | BPNN | ELM | RBFNN | RF | |
---|---|---|---|---|---|---|
Average MSE | 0.0054 | 0.005 | 0.0038 | 0.0402 | 0.0057 | 0.0058 |
Optimal MSE | 0.0047 | 0.0043 | 0.0027 | 0.0377 | 0.0057 | 0.0020 |
Although the accuracy of the HBRB-I model is slightly different from that of machine learning algorithms such as BPNN, ELM, and RF, the HBRB-I model is interpretable and the reasoning process can be retroactive. As for data-driven modeling methods such as BPNN, ELM, and RF, the internal structure is invisible. The inference engine ER deduction is used in HBRB-I. It could reasonably explain the causal relationship between input and output to make the conclusion more reliable. Besides, the HBRB-I model can make full use of expert knowledge to characterize the system and help users better understand the model structure. However, they cannot be achieved by ordinary machine learning methods. Compared with the HBRB model without interpretable definition, the HBRB-I model has no obvious difference in accuracy, and its advantage is reflected in the interpretability. On the one hand, the optimization method of the HBRB-I model has interpretability constraints. When the WOA is initialized, the optimal individual is selected based on expert knowledge instead of random selection in the HBRB model, which effectively utilizes expert knowledge. The rationality of the optimal individual selection is improved. On the other hand, the rule matching degree of the HBRB-I model sets a screening interval. Therefore, the belief distribution of the model satisfies the actual performance prediction system. The HBRB fails to solve the problem of reasonable belief distribution, such as the 14-th rules in
This paper proposes a processor performance prediction method based on sensitivity analysis and an interpretable hierarchical belief rule base. Transparent reasoning engine is used in the model reasoning process to deduce the problem. Besides, the interpretability criterion is added to make full use of the characteristics of the expert knowledge description system. It can solve the problems of ineffective use of expert knowledge and unreasonable belief optimization. Therefore, the performance prediction method proposed in this paper has strong interpretability. The results of predictive model are traceable. The relationship between input and output also verifies the role of interpretable modeling. Compared with other methods, the interpretability enhances the reliability of the system under the condition that the accuracy is basically guaranteed.
The researchers would like to thank Harbin Normal University for funding this project.
No | x1 x2 | Belief constraint { |
The optimized belief { |
|
---|---|---|---|---|
1 | 1 | VSVS | {0.95~1.00 0.00~0.05 0.00~0.05 0.00~0.05} | {0.9530 0.0439 0.0031 0.0000} |
2 | 1 | VS S | {0.00~0.05 0.00~0.50 0.00~0.05 0.50~1.00} | {0.0000 0.4999 0.0000 0.5001} |
3 | 1 | VS M | {0.95~1.00 0.00~0.05 0.00~0.01 0.00~0.01} | {0.9968 0.0032 0.0000 0.0000} |
4 | 1 | VS L | {0.00~0.01 0.95~1.00 0.00~0.01 0.00~0.01} | {0.0000 0.9999 0.0001 0.0000} |
5 | 1 | S VS | {0.00~0.01 0.95~1.00 0.00~0.05 0.00~0.01} | {0.0000 0.9767 0.0233 0.0000} |
6 | 1 | S S | {0.00~0.01 0.95~1.00 0.00~0.01 0.00~0.01} | {0.0000 0.9999 0.0000 0.0001} |
7 | 1 | S M | {0.00~0.01 0.00~0.50 0.00~0.50 0.00~0.05} | {0.0000 0.4950 0.4950 0.0100} |
8 | 1 | S L | {0.00~0.01 0.00~0.01 0.90~1.00 0.05~0.10} | {0.0000 0.0000 0.9278 0.0722} |
9 | 1 | M VS | {0.00~0.01 0.85~0.90 0.00~0.01 0.10~0.15} | {0.0000 0.8861 0.0000 0.1139} |
10 | 1 | M S | {0.00~0.01 0.00~0.01 0.00~0.50 0.50~1.00} | {0.0000 0.0000 0.4999 0.5001} |
11 | 1 | M M | {0.00~0.01 0.10~0.15 0.70~0.76 0.10~0.15} | {0.0080 0.1244 0.7425 0.1251} |
12 | 1 | M L | {0.00~0.01 0.30~0.35 0.30~0.35 0.30~0.35} | {0.0000 0.3333 0.3333 0.3334} |
13 | 1 | L VS | {0.00~0.01 0.00~0.01 0.95~1.00 0.00~0.05} | {0.0000 0.0000 0.9739 0.0261} |
14 | 1 | L S | {0.90~0.95 0.05~1.00 0.00~0.01 0.00~0.01} | {0.9251 0.0749 0.0000 0.0000} |
15 | 1 | L M | {0.00~0.01 0.00~0.01 0.00~0.01 0.95~1.00} | {0.0000 0.0000 0.0001 0.9999} |
16 | 1 | L L | {0.00~0.01 0.00~0.01 0.00~0.50 0.50~1.00} | {0.0000 0.0000 0.4999 0.5001} |
Index | Fit expression | Results | ||
---|---|---|---|---|
MYCT | Negative | −0.3070(−) | Consistent | |
MMIN | Positive | 0.7950(+) | Consistent | |
MMAX | Positive | 0.8630(+) | Consistent | |
CACHE | Positive | 0.6626(+) | Consistent | |
CHMIN | Positive | 0.6089(+) | Consistent | |
CHMAX | Positive | 0.6052(+) | Consistent |
Model | Max iteration | Rules | Execution time |
---|---|---|---|
WOA-HBRB | 800 | 64 | 178.81 s |
WOA-HBRB-I | 800 | 57 | 198.32 s |