The emergence of power dispatching automation systems has greatly improved the efficiency of power industry operations and promoted the rapid development of the power industry. However, with the convergence and increase in power data flow, the data dispatching network and the main station dispatching automation system have encountered substantial pressure. Therefore, the method of online data resolution and rapid problem identification of dispatching automation systems has been widely investigated. In this paper, we perform a comprehensive review of automated dispatching of massive dispatching data from the perspective of intelligent identification, discuss unresolved research issues and outline future directions in this area. In particular, we divide intelligent identification over power big data into data acquisition and storage processes, anomaly detection and fault discrimination processes, and fault tracing for dispatching operations during communication. A detailed survey of the solutions to the challenges in intelligent identification over power big data is then presented. Moreover, opportunities and future directions are outlined.
Recently, with the rapid development of the social economy and technology, the demand for a safe and stable power supply has generated higher requirements [
The emergence of power dispatching automation systems has greatly improved the efficiency of power industry operations, solved many problems faced by the power system, and greatly promoted the rapid development of the power industry [
As the speed of power grid construction accelerates and its scale expands, various dispatching automation systems are continuously built, and the business interaction applications between local dispatching systems and county dispatching systems become more frequent, so the power dispatching data network emerges [
With the construction of a power supply company’s power dispatching data network, the main station has been consecutively connected to all county dispatching services, resulting in further convergence and an increase in information flow in the main station, which puts considerable pressure on the dispatching data network and the main station dispatching automation system [ Data analysis is not sufficiently accurate, and hidden problems are difficult to discover. The timely detection of many hidden problems from the master control and dispatch automation system is difficult through the dispatch automation system. For example, after the integration of local and county consolidation and regulation and control, an increasing number of dispatching objects are consecutively connected to the master station, and the dispatching automation system is stuck and unresponsive, which is likely to be related to problems such as intermittent errors and frequency of messages. Problem data capture and diagnosis are not immediate enough. Intermittent false online, unsuccessful remote control, and partial signals that are not uploaded when tremendous signals are uploaded occasionally occur. Due to the lack of timely and in-depth analysis, the problem cannot be immediately identified, which hinders remote regulation and control and increases work costs. For example, if historical information is checked without a breaker variation signal, the system cannot determine the accident tripping after successfully tripping the reclosing action. This situation affects the efficiency of power grid repair. On the other hand, the outage information cannot be instantly reported. Lack of data support for work such as counterevent analysis. Due to the lack of a black box mechanism, data for many problems are not instantly saved, which causes problems for scheduling automation personnel in learning, fault analysis, and drills.
To effectively identify the safety hazards inherent in the dispatching automation system in real time, it is necessary to improve the automation operation and management level of power supply companies and to lay a solid foundation for the safe, stable, high-quality and economic operation of the power grid through online analysis of the data volume of the dispatching automation system and rapid problem identification research.
The remainder of this paper is organized as follows: The acquisition and structured storage methods of the data flow are surveyed in
Presently, the single storage function enables optimization analysis and fault tracing when retrieving data [
To achieve the integrity of big data acquisition, data flow acquisition equipment depends on the high performance and fast processing capacity of software and hardware, which can quickly and deeply analyze the complete collected data and upload the data information after pattern recognition to the data business analysis platform [
In existing works on the acquisition and structured storage of data flow, there are some open challenges, which are illustrated in Effective and valid data collection. The process of obtaining raw data from an external system or network is referred to as data collection [ Efficient data transmission. Due to the high bandwidth consumption and the energy efficiency, transmitting numerous data to storage facilities becomes challenging [ Reliability and persistency of data storage. Considering the tremendous amount of data, it is challenging to achieve reliability and persistency of data storage while balancing the cost [
The data acquisition methods are listed in
Target | Reference | Methods | Advantage |
---|---|---|---|
Acquisition performance | [ |
Introduced vehicular micro clouds to data collection | Improve the fault tolerance of data collection |
[ |
Data forwarding algorithm | Improves the efficiency of the data delivery ratio through restricting the broadcast messages | |
[ |
Combine the long-term evolution network with vehicular hoc network to optimize the data volume and collection cycle | Optimize the data volume and collection cycle considering system robustness | |
[ |
Data relay mule–based collection scheme | Improve the data acquisition performance | |
[ |
Method of traffic data collection using vehicle-mounted monocular camera | Enhance the flexibility and coverage of traffic data collection | |
Optimizing resource utilization | [ |
Data acquisition scheme supported resource utilization optimization | Optimize the limitation of the wireless communication bandwidth |
[ |
On-demand vehicular sensing framework without infrastructure | Achieve accurate monitoring data and reduce the number of participating vehicles, energy consumption and communication costs | |
Security protection | [ |
Route optimization method | Improve the collection of multimedia data considering dynamic factors |
[ |
Data acquisition system based on blockchain | Utilize the UAV as a relay to gather data | |
[ |
Security-preserving data collection and sharing scheme based on blockchain | Ensure efficient and real-time data collection with security protection |
Hagenauer et al. [
Considering the system robustness, Turcanu et al. [
Ren et al. [
To address the limitation of flexibility and coverage of traffic data collection, a traffic data collection method that uses a vehicle-mounted monocular camera is proposed [
Nie et al. [
Rahman et al. [
Islam et al. [
To address the problem of data security, Kong et al. [
The data storage methods are listed in
Target | Reference | Method | Advantage |
---|---|---|---|
Storage performance | [ |
Distributed database based on Apache | Determine the validity of packets by detecting the payload |
[ |
Multilevel filtering method | Accomplish similarity join of the fuzzy string | |
[ |
Reformative method in DPI based on regular expression | Utilize character intervals to describe multiple consecutive characters and improve transmission efficiency | |
[ |
Semistructured data storage and processing engine | Extract semantic values from a tremendous amount of patient data | |
[ |
Semistructured data in the data flow, based on the DL semistructured tree miner algorithm | Effectively mined and stored data with the time attenuation model |
Power systems generate millions or even billions of status, debug, and error records every day. To ensure the security and sustainability of power systems, it is necessary to quickly process and analyze a large amount of power data to realize real-time decisions. Traditional solutions typically use relational databases to manage power data. However, when the amount of data substantially increases, the relational database cannot effectively process and analyze a large amount of power data.
Jin et al. [
A multilevel filtering method is proposed to accomplish similarity join of the fuzzy string. With the proposed method, Wang et al. [
Based on regular expression, a reformative method in DPI is proposed. In the face of increasingly complex attacks, accurate string recognition has difficulty accurately obtaining features. Regular expressions with flexibility and high efficiency are widely employed in feature fuzzy matching. In the matching process, Sun et al. [
With the intention of handling the problem of increasing medical expenses caused by the swift increase in the quantity and quality of medical data, Satti et al. [
In [
Although some of the previously described challenges in data acquisition and storage are addressed, opportunities remain, as illustrated in Security in data transmission. Due to the limitation of network transmission conditions, data transmission is vulnerable to attack [ Privacy preservation and security assurance. Although data storage has received widespread attention in recent years, as summarized in
Anomaly detection and fault discrimination have been investigated in many existing works [ Lack of training samples. Sufficient training samples are required to construct a model with high performance [ Anomaly detection in time series data. Time-series data, such as weather data and power data, have high requirements for real time [ Privacy preservation in anomaly detection. To more efficiently detect anomalies, online, real-time anomaly detection methods are usually adopted [
The reliability and real-time service scheduling flow directly influence the function [
The anomaly detection methods to improve the stability of data flow are listed in
Target | Reference | Method | Advantage |
---|---|---|---|
Improving stability of data flow | [ |
Anomaly detection method that combines LSTM with a GAN | Improve the stability of time series data |
[ |
Survey of anomaly detection methods in computer network | Discuss the challenges and present the open problems | |
[ |
Feature selection approach with random forest and support vector machine | Improve the stability of feature selection | |
[ |
Adaptive online anomaly detection approach in small samples | Predict the unknown anomalies by classifying the known anomalies | |
[ |
Random forest method combined with feature selection and DL classification | Perform better when facing financial data | |
[ |
Anomaly detection method for |
Identify the anomalies caused by various attacks | |
[ |
Probabilistic anomaly discrimination method for wind turbine | Improve the stability of the wind turbine | |
[ |
Micro anomaly detection in satellite telemetry data | Improve the stability of satellite telemetry data | |
[ |
Unsupervised anomaly detection method in the IoT | Effectively characterize the time series data in the IoT | |
[ |
Anomaly detection approach with signal filtering discrimination | Enhance the security of the automated vehicle transportation | |
[ |
Anomaly discrimination and classification approach | Improve the stability of the automotive industry |
Zhu et al. [
The anomaly detection methods in computer networks were surveyed in [
Li et al. [
Existing anomaly detection methods performed poorly in the absence of training samples. Therefore, an adaptive online anomaly detection approach in small samples was proposed [
To reduce the financial loss of financial statement fraud for investors, Yao et al. [
Li et al. [
The stability of the wind turbine indicates its operating conditions. To develop the condition and anomaly detection for the wind turbine, Zhang et al. [
Sun et al. [
Anomaly detection is necessary for the Internet of Things (IoT). However, the data are generally labeled to discriminate the anomalies. Guo et al. [
Currently, automated vehicle transportation, which is a novel MEC-based scenario, has been emerging. To enhance the security of automated vehicle transportation, Wang et al. [
Numerous data were generated during the production and testing phases in the automotive industry. To evaluate the performance of vehicular systems, potential faults should be discriminated against. By analyzing the connections of vehicular systems, a robust anomaly discrimination and classification approach was presented [
The anomaly detection methods to improve the accuracy of data flow are listed in
Target | Reference | Method | Advantage |
---|---|---|---|
Improving accuracy of data flow | [ |
Bias scoring mechanism for anomalies | Adaptively detect the anomaly |
[ |
Framework to detect the abnormal sequence | Improve the identification capacity of anomaly detection | |
[ |
Anomaly detection in power systems with artificial neural network | Reduce the consumption of online resources | |
[ |
Counting method for the telemetry data features | Extract the features and improve the accuracy of telemetry data | |
[ |
Unsupervised anomaly detection approach | Distinguish between normal data and abnormal data | |
[ |
Weather data analysis framework | Extract the features and modes in complicated weather data | |
[ |
Anomaly detection model for satellite telemetry data with sequence features | Improve the accuracy of satellite telemetry data | |
[ |
Intelligent anomaly detection method | Solve the class imbalance | |
[ |
Survey of the anomaly detection methods with DL | Classify and evaluate the methods | |
[ |
Anomaly detection method combined the autoencoder with LSTM | Improve the accuracy of anomaly detection | |
[ |
Generation and identification model with GAN | Generate samples to train the identification model | |
[ |
Identification and correction method for drilling data | Correct the abnormal drilling data | |
[ |
Extreme gradient boosting framework | Improve the classification accuracy | |
[ |
Abnormal traffic discrimination model | Generate the substation samples |
Anomaly detection in computer networks was investigated in [
Song et al. [
Anomaly detection is necessary to maintain the stability of power systems. The artificial neural network (ANN) could train offline data and reduce the consumption of online resources. Therefore, ANNs can be applied to power systems to detect faults. Yadav et al. [
On-orbit anomaly detection is a primary problem in satellite management [
To identify and detect the anomalies in the process of chemical plants, an unsupervised approach that combines graph theory (GT) with generative topographic mapping (GTM) was presented in [
Weather data analysis can be implemented by the IoT and big data framework. To extract the features and modes in complicated weather data, a weather sensor anomaly detection algorithm using clustering was explored [
To improve the accuracy of satellite telemetry data, Du et al. [
The monitoring and acquisition data in the wind turbine system were imbalanced because of the large amount of data. Therefore, the abnormal data were difficult to accurately discriminate. With the deep neural network (DNN), Chen et al. [
The anomaly detection methods with DL are investigated in detail [
Park et al. [
DL, which is an algorithm driven by neural networks, is rapidly developing. DL models with feature representation are applied to fault detection. However, the misclassification rate was increased when the fault data were limited. To improve the accuracy of anomaly detection, Zhou et al. [
To improve the quality of drilling data, Yang et al. [
To improve the classification accuracy of the scheme for protecting the power transformer, Raichura et al. [
With the development of digitalization, the flow of substation communication networks is increasing. Moreover, abnormal traffic discrimination has been the key to maintaining network security. Yang et al. [
Although some of the previously described challenges in data acquisition and storage are addressed, opportunities to improve the performance of anomaly detection are discussed in existing works and are illustrated in Anomaly detection in small samples [ Online anomaly detection [ Mixed trained samples [
Most existing works focus on service fault-tolerant scheduling. However, with the development of a heterogeneous system and an increase in data, certain challenges about service fault-tolerant scheduling algorithms are surveyed, as illustrated in Dynamic scheduling. The dynamic fault-tolerant scheduling of services effectively reduces the delay and energy consumption caused by resource redistribution and improves resource utilization [ Criticality levels of run-time faults. The criticality levels of run-time faults represent the priority to handle. To enhance the efficiency of service fault-tolerant scheduling, run-time faults with high criticality levels must be addressed first [ Service scheduling in heterogeneous systems. Heterogeneous systems bring convenience to service scheduling and increase the complexity of the systems. Many complex faults occur in heterogeneous systems, which provides new challenges for fault-tolerant scheduling algorithms [
With the development of big data and 5G, numerous data have been generated according to the requirements of users [
Next, we will summarize existing works on service fault-tolerant scheduling from four optimization goals, i.e., reducing energy consumption, decreasing service response latency, improving resource utilization and enhancing the reliability of systems, which are shown in
The service fault-tolerant scheduling methods for decreasing energy consumption are listed in
Target | Reference | Method | Advantage |
---|---|---|---|
Reducing the energy consumption | [ |
Heuristic algorithm based on the earliest finish time of service clusters | Reduce the resources consumption |
[ |
Survey of fault-tolerant service scheduling in CC | Evaluate the performance of existing methods | |
[ |
Energy-aware, efficient heuristic scheme | Solve fault-tolerant scheduling for real-time tasks in heterogeneous systems | |
[ |
Fault-tolerance scheduling method by developing three mode redundancy | Decrease the energy consumption in systems | |
[ |
Fault-tolerant scheduling method with checkpoints | Address the potential faults and decrease the energy consumption in systems | |
[ |
Byzantine fault detection algorithm | Decrease the fault-tolerant overhead |
Currently, executing clustering services will increase the efficiency of scientific workflows (SWf) in cloud servers. Vinay et al. [
The scheduling algorithms in CC focus on high-performance computation and computing costs. However, because of the incomplete scheduling strategies, the execution efficiency of computing tasks is hard to guarantee. Therefore, to build a foundation for constructing an efficient fault-tolerant framework, Pandita et al. [
Nair et al. [
Three mode redundancy (TMR) is used to eliminate faults in homogeneous systems with high energy consumption. Yu et al. [
In mobile cloud computing (MCC), mobile devices are usually resource limited. The scheduling strategy must be updated when scheduling resources change. Lee et al. [
Chinnathambi et al. [
The service fault-tolerant scheduling approaches for reducing the response latency of service are listed in
Target | Reference | Method | Advantage |
---|---|---|---|
[ |
Fault-tolerant service scheduling scheme with checkpoints in CC | Reduce the response latency of service | |
[ |
Fault-tolerance scheduling algorithm based on resubmitting and duplication | Maximize the idle time during service scheduling | |
Decreasing response latency of service | [ |
Dynamic clustered scheduling method | Decrease the execution delay of tasks |
[ |
Heuristic service, fault-tolerant scheduling algorithm based on cross-entropy | Balance the running time of tasks and the lifetime of systems | |
[ |
Multipath service scheduling algorithm | Optimize the reliability and fault tolerance of the IIoT |
With the development of intelligent computing techniques in CC, fault tolerance has become increasingly significant [
Yao et al. [
Considering the failure of computation tasks in CC, Abd Latiff et al. [
Cao et al. [
Applications in the Industrial Internet of Things (IIoT) usually require high reliability and low access latency. Ahrar et al. [
The service fault-tolerant scheduling methods to improve resource utilization are listed in
Target | Reference | Method | Advantage |
---|---|---|---|
[ |
Dynamic resource allocation method with a fault-tolerant mechanism | Improve the resource utilization of systems | |
[ |
Service allocation and communication approach with real-time fault-tolerance | Improve the computation resources utilization | |
Improving resource utilization | [ |
Offline elastic scheduling algorithm | Dynamically regulate resource allocation and increase resources utilization |
[ |
Dynamic elastic fault-tolerant scheduling method | Improve resource utilization in the cloud | |
[ |
Dynamic task allocation and scheduling scheme | Optimize the energy efficiency | |
[ |
Fault-tolerant framework with criticality levels of run-time faults | Avoid the overallocation of computation resources |
Fault tolerance is widely employed in CC. Soniya et al. [
Zhu et al. [
In cloud systems, fault tolerance has been the primary requirement for the execution of computation tasks. Therefore, Ding et al. [
Yan et al. [
Marahatta et al. [
Chen et al. [
The service fault-tolerant scheduling methods to improve the reliability of systems are listed in
Target | Reference | Method | Advantage |
---|---|---|---|
[ |
Fault-tolerant scheduling method in power management system | Operate the deceleration mechanism according to the actual workload | |
[ |
Fault-tolerant task scheduling algorithm in grid computing | Improve the reliability of systems | |
[ |
Fault-tolerance method in criticality-mixed systems | Improve the security of tasks with different critical levels | |
[ |
Fault trace approach for the power grid | Take full advantage of the monitoring data and infer the fault reasons | |
Enhancing reliability of systems | [ |
Heuristic fault-tolerant task scheduling algorithm | Improve the reliability of systems by the number of tolerant permanent faults |
[ |
Adaptive fault-tolerant scheduling algorithm | Select the most appropriate fault-tolerant technique to address the faults | |
[ |
Service fault-tolerant scheduling method in scientific workflows | Improve the reliability of systems | |
[ |
Task clustering algorithm with fault tolerance | Synthetically consider the execution latency and cost of workflows |
Zhang et al. [
Grid computing serves computation-sensitive and long-operating applications. To guarantee the quality of service (QoS), these applications need to tolerate potential faults. Based on the ant colony algorithm, Idris et al. [
In real-time systems and embedded systems, criticality-mixed task scheduling is usually considered. Zhou et al. [
Wang et al. [
The complexity of heterogeneous systems increases the possibility of faults in the systems, resulting in the growing significance of efficient task scheduling strategies with fault tolerance. Hence, Liu et al. [
Alarifi et al. [
Fault tolerance is the primary technique in CC. When the workflow experiences faults, the applications provide a protection mechanism to ensure the safety of the systems. Talwani et al. [
Task clustering can enhance the computational granularity of the scientific workflow to execute tasks with distributed computing resources. Khaldi et al. [
Although some of the previously mentioned challenges in service fault-tolerant scheduling are addressed, opportunities to enhance the efficiency of service scheduling remain, as illustrated in Elastic resource scheduling. The rich resources in data centers are used to process a large amount of data, which consumes resources and power and negatively impacts the natural environment [ Prediction for errors in tasks. Frequent task migration causes additional energy consumption and influences the system performance [ Security in service fault-tolerant scheduling. Service fault-tolerant scheduling requires the detection of multiple attributes of services, possibly including the privacy of the user, which leads to the significance of security [
In this work, a comprehensive and detailed survey of intelligent identification based on power big data is presented. First, the data acquisition and storage process are investigated. Second, the anomaly characteristics and fault discrimination techniques of a massive amount of data are analyzed. Furthermore, the problem of fault tracing for dispatching operations during communication is discussed. This survey is presented to promote the further progress of intelligent identification based on power big data. However, numerous research issues in this area are still open and need further efforts, including optimizing the distributed intelligent identification process and balancing security protection and performance.