For BCI systems, it is important to have an accurate and less complex architecture to control a device with enhanced accuracy. In this paper, a novel methodology for more accurate detection of the hemodynamic response has been developed using a multimodal brain-computer interface (BCI). An integrated classifier has been developed for achieving better classification accuracy using two modalities. An integrated EEG-fNIRS-based vector-phase analysis (VPA) has been conducted. An open-source dataset collected at the Technische Universität Berlin, including simultaneous electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals of 26 healthy participants during n-back tests, has been used for this research. Instrumental and physiological noise removal has been done using preprocessing techniques followed by individually detecting activity in both modalities. With resting state threshold circle, VPA has been used to detect a hemodynamic response in fNIRS signals, whereas phase plots for EEG signals have been constructed using Hilbert Transform to detect the activity in each trial. Multiple threshold circles are drawn in the vector plane, where each circle is drawn after task completion in each trial of EEG signal. Finally, both processes are integrated into one vector-phase plot to get combined detection of hemodynamic response for activity. Results of this study illustrate that the combined EEG-fNIRS VPA yields considerably higher average classification accuracy, that is 91.35%, as compared to other classifiers such as support vector machine (SVM), convolutional neural networks (CNN), deep neural networks (DNN) and VPA (with dual-threshold circles) with classification accuracies 82%, 89%, 87% and 86% respectively. Outcomes of this research demonstrate that improved classification performance can be feasibly achieved using multimodal VPA for EEG-fNIRS hybrid data.
A brain-computer interface (BCI) is a pathway for communication between brain thoughts and computer to achieve hardware control, without any dependence on channels like nerves and muscles. [
EEG and fNIRS are two of the significant non-invasive modalities. Being portable, cost effective and less noisy, counts as the major benefits of these modalities [
fNIRS is one of the emerging BCIs which records brain activity as blood oxygen level changes. It uses near-infrared-range light of wavelength 650~1000 nm to estimate the variations in the concentration of oxygenated hemoglobin (∆HbO) and deoxygenated hemoglobin (∆HbR) [
A hybrid BCI system is usually comprised of two BCIs. It can also be composed of at least one BCI system and another system (ECG and EMG etc.) [
Accuracy is one of the major concerns of most researchers to evaluate brain-computer machines. There is a need to develop a hybrid EEG-fNIRS architecture that can enhance the accuracy for better performance and control of devices. Thus, to further improve the accuracy, in this study, we have devised a novel methodology using an open-source meta-dataset comprising of simultaneous EEG and fNIRS data of 26 healthy subjects, integrated at Technische Universität Berlin is available online (
The rest of this study is organized as follows. Section 2 investigates the literature work and presents a review on the techniques/methods for early response detection using EEG. Section 3 provides details of the dataset and discusses the preprocessing process. Section 4 describes the design details of the proposed method. Section 5 presents the experimental results. Section 6 outlines the conclusion and discusses the limitations of the proposed method.
Vector-phase analysis (VPA) displays the trajectory formed as a result of oxy-hemoglobin (ΔHbO) and deoxy-hemoglobin (ΔHbR) concentration changes [
Different researches have been carried out on the dataset selected for this research, and efforts have been made to improve the classification accuracy [
In this paper, we propose a novel modified multimodal VPA methodology to detect activity in hemodynamic response. We have used state-of-the-art hybrid BCI (EEG-fNIRS) data for the n-back test for the presented methodology. Complete data has been preprocessed using conventional ways to make it noise-free. Initially, both the modalities have been dealt with individually. Hilbert transform has been applied to EEG signals to get the required magnitude and phase values to construct polar plots of all the trials. Then, activity is detected using these polar plots.
Similarly, VPA has been applied to fNIRS signals to construct vector-based phase plot for hemodynamic response detection with a resting state threshold circle as a detection criterion. Finally, an integrated multimodal VPA has been designed with multiple threshold circles, based on the activity completion of each EEG signal trial, to achieve better detection of hemodynamic response. This proposed design yield 91.35% average classification accuracy, which is significantly higher than other techniques mentioned previously.
An open-source dataset has been used for this research (
Each participant was provided with a comfortable armchair to sit in front of a 24’ LCD screen. Distance between the person’s eyes and the screen was 1.2 m. The right armrest had numeric keypad buttons (number 7 and 8) fixed to it. All participants were directed to look at the screen and try to abstain from moving their body. This experiment comprised of three types of tasks (n-back tasks, discrimination/selection response tasks and word generation tasks) with three sessions each. For this study, we have used the n-back task dataset [
This dataset of the n-back test was comprised of three sessions for every subject as shown in
A short beep of 250 ms was used to signify the person about the beginning and end of every task period. A fixation cross was shown on the screen for the rest period. Every task period consisted of twenty trials, each of 2 s. In every trial, a random one-digit number was displayed on the screen for 0.5 s, followed by a fixation cross for 1.5 s. For the 0-back test, participants pressed either the number 7 button for a ‘target’ digit or number 8 button for a ‘non-target’ digit. In the case of 2- and 3-back tasks, participants were instructed to press the ‘target’ button, number 7, if presently shown digit matched the 2 or 3 preceding numbers respectively, otherwise the ‘non-target’ button 8. For each type of n-back task, a total of 180 trials were carried out (3 session
Total three sessions were conducted with 9 series each, and every series was comprised of initial 2 s of instruction about the type of task (0-, 2- and 3-back), the 40 s of the task, 1 s of “STOP” word shown and 20 s of rest. The task period had 20 trials in it, and each trial of a total of 2 s consisted of 0.5 s of digit display and 1.5 s of fixation cross display [
fNIRS and EEG signals were recorded in parallel. EEG data were acquired at the sampling frequency of 200 Hz using a multichannel BrainAmp EEG amplifier (Brain Products GmbH, Gilching, Germany). According to the international 10–5 system, thirty electrodes were attached to a flexible fabric cap (EASYCAP GmbH, Herrsching am Ammersee, Germany) as shown in
fNIRS data was recorded at the sampling frequency of 10Hz with a NIRScout (NIRx Medizintechnik GmbH, Berlin, Germany). Sixteen sources and sixteen detectors were attached at frontal (sixteen channels around AFz, AF3, AF4, AF7 and AF8), parietal (four channels each around P3 and P4), motor (four channels each around C3 and C4), and occipital (four channels around POz) areas. An adjoining source-detector pair sets up an fNIRS channel. A configuration of a total of 36 channels was formed. The fNIRS channels were configured according to the international 10–5 system around AFpz, AFp3, AFp4, AFp7, AFp8, AF1, AF2, AF7, AF8, AF5h, AF6h, AFFz, AFF3h, AFF4h, AFF5, AFF6, FCC3, FCC4, C3h, C4h, C5h, C6h, CCP3, CCP4, CPP3, CPP4, P3h, P4h, P5h, P6h, PPOz, PPO3, PPO4, PO1, PO2, and POOz as shown in
Twelve frontal channels (
Before using the data for any technique, we have preprocessed the signals to get the best possible results. For EEG data, since the fundamental frequencies for this data were lying in the alpha (α) (8–13 Hz) and theta (θ) (4–8 Hz) bands, so, to remove the noise and to remain within the interested frequency bands, a bandpass filter (5
In this research, Hilbert transform (HT) has been used to calculate the imaginary component of EEG signals along with their phases and magnitudes. Polar plot construction for each trial of each series of all EEG signals is then achieved to detect activity.
For an EEG signal
Then the analytical signal corresponding to
where
and
Outcomes of HT are used to construct the polar plots of EEG signal trials to indicate activity. Mean values are calculated, for both (
The vector-phase analysis is a technique that can be used to detect the hemodynamic response by using just the two components,
The magnitude and phase of a vector
The eight phases of this vector plane are as shown in
A threshold circle is drawn based on the maximum value of the rest period in a signal. If the trajectory of
In this study, a modified form of vector-based phase analysis has been proposed. For this study, we have used just
So, the radius of the threshold circle can be calculated as
A vector-phase diagram based on both EEG and fNIRS activity detection has been proposed in this design. So, for that purpose, we draw a circle for each trial activity completion in EEG signal. As the activity can be detected earlier in EEG signal than fNIRS signal [
If any trail is detected through phase plot of EEG signal or modified VPA, then it is considered the presence of activity. The flowchart for the proposed methodology is shown in
For this novel technique, we have used two gamma function to construct the ideal trajectory of
where
The
where
There were 20 trials for each series for this experiment, so we have convolved 20 impulses with ideal
In this novel methodology, data from selected channels of subject one was initially filtered to retain signals only in the 0.1–15 Hz frequency range. Then data from all channels were averaged out to construct one average signal. After that, HT is used first to calculate the imaginary component of the average signal using
For the 1
Next, we have implemented the same scheme for all the trials of series 1 as shown in
For every subject, fNIRS signals of all selected channels are preprocessed and then averaged to get an average signal. As mentioned in the previous section, the conventional VPA plots were constructed, for series 1 of 1
For depicting the channels’ activation, brain maps have been constructed. For this purpose, five brain maps of each series (0-, 2- and 3-back tasks) for two subjects have been created, as shown in
Five maps are constructed for all three types of series (0-, 2- and 3-back tasks), as shown in
fNIRS channel No. | EEG channel selected corresponding to fNIRS channel |
---|---|
1(AF7) | 2(AFF5h) |
2(AFF5) | 2(AFF5h) |
3(AFp7) | 1(Fp1) |
4(AF5h) | 2(AFF5h) |
5(AFp3) | 1(Fp1) |
6(AFF3h) | 4(F1) |
7(AF1) | 3(AFz) |
8(AFFz) | 3(AFz) |
9(AFpz) | 3(AFz) |
10(AF2) | 3(AFz) |
11(AFp4) | 19(F2) |
20(AFF4h) | 17(Fp2) |
21(AF6h) | 18(AFF6h) |
22(AFF6) | 18(AFF6h) |
23(AFp8) | 19(F2) |
24(AF8) | 18(AFF6h) |
After using this novel classifier for all the 3 sessions for all subjects, the classification accuracy for every series has been calculated. For each subject’s average signal, all types of tasks (0-, 2- and 3-back) were performed nine times each. The overall accuracy for this novel classifier,
Subjects | Average accuracies 0-back (%) | Average |
Average |
Overall average accuracy (%) | SVM average accuracies (%) | CNN average accuracies (%) | DNN average accuracies (%) |
---|---|---|---|---|---|---|---|
Subject 1 | 92.78 | 98.89 | 97.22 | 82 | 92 | 91 | |
Subject 2 | 86.67 | 99.44 | 98.89 | 80 | 73 | 70 | |
Subject 3 | 83.33 | 95.56 | 96.11 | 76 | 84 | 82 | |
Subject 4 | 71.11 | 73.89 | 80.56 | 76 | 88 | 86 | |
Subject 5 | 90.56 | 92.78 | 92.78 | 87 | 85 | 83 | |
Subject 6 | 85.56 | 95.56 | 88.89 | 71 | 84 | 81 | |
Subject 7 | 87.78 | 98.89 | 100 | 80 | 98 | 96 | |
Subject 8 | 95 | 97.78 | 99.44 | 84 | 94 | 92 | |
Subject 9 | 97.78 | 97.22 | 98.33 | 80 | 86 | 85 | |
Subject 10 | 78.33 | 60 | 56.67 | 91 | 92 | 89 | |
Subject 11 | 90 | 83.33 | 85 | 78 | 90 | 89 | |
Subject 12 | 86.11 | 95 | 87.78 | 91 | 92 | 89 | |
Subject 13 | 94.44 | 95.56 | 96.67 | 67 | 79 | 77 | |
Subject 14 | 83.33 | 97.78 | 98.89 | 82 | 94 | 93 | |
Subject 15 | 94.44 | 95.56 | 96.67 | 87 | 86 | 83 | |
Subject 16 | 97.22 | 99.44 | 97.78 | 93 | 96 | 95 | |
Subject 17 | 97.22 | 90.56 | 92.78 | 78 | 86 | 83 | |
Subject 18 | 85 | 85.56 | 83.89 | 84 | 86 | 85 | |
Subject 19 | 98.33 | 96.11 | 99.44 | 84 | 86 | 84 | |
Subject 20 | 91.11 | 93.33 | 92.22 | 91 | 94 | 93 | |
Subject 21 | 98.33 | 99.44 | 99.44 | 80 | 94 | 91 | |
Subject 22 | 86.11 | 95 | 96.11 | 87 | 94 | 92 | |
Subject 23 | 92.22 | 98.33 | 97.78 | 78 | 92 | 89 | |
Subject 24 | 83.89 | 81.67 | 77.78 | 93 | 92 | 90 | |
Subject 25 | 99.44 | 98.33 | 98.89 | 84 | 92 | 89 | |
Subject 26 | 91.67 | 81.11 | 81.11 | 89 | 92 | 90 | |
Using this novel methodology, we have achieved a relatively higher average classification accuracy than other reported techniques used for this dataset and VPA with dual-threshold circles. As it can be seen from
Many researches have been carried out up till now to improve the classification accuracy using hybrid BCI [
One of the advantages of this proposed classifier is that it uses VPA to channel fNIRS signals. After rejecting the inactive channels, we are averaging the selected channels’ signals for each subject. Therefore, inactive channels do not reduce signal activation, hence improving the performance, making it more accurate to detect the activity in hemodynamic response.
Another advantage of this methodology is that it uses HT in a different way to construct phase plots of EEG signal trials to indicate the occurrence of activity, which is an easy and feasible method. Detection of activity in EEG separately further enhances the performance of our classifier by increasing the average classification accuracy.
Another benefit of this classifier is that it does not require any training like other conventional machine learning and deep learning classifiers because it is a trajectory-based approach with EEG trials-based multiple circles.
For this research, a considerably larger dataset [
In this study, channel activation has also been highlighted using brain maps constructed in a relatively different way than other conventional ways like t-score [
A limitation in this research is that activity in a time span is considered as detected if its occurrence is indicated in either EEG signal or multimodal VPA trajectory. A false positive detection can result in some false detection of activity. To further improve the classifier, research can be carried out to overcome this shortcoming. In our proposed methodology, simple preprocessing techniques have been used, such as low pass, bandpass and high pass filters. The presence of artifacts is still possible in the signals and can affect the resting state circle of the vector-phase diagram. So, to further improve the performance of this technique, advanced preprocessing techniques and artifact rejection algorithms are desirable. Moreover, in this research, a comparison between gender-based accuracy has not been conducted, so this investigation can also be carried out to indicate whether the accuracy gets affected by gender or not.
In this study, a novel methodology has been proposed for enhancing average classification accuracy using hybrid BCI (EEG-fNIRS). For this research, a hybrid (EEG-fNIRS) dataset for n-back tasks, collected at Technische Universität Berlin was used. Hilbert transform was used to construct phase plots for activity detection in EEG trials. A modified multimodal VPA was designed with multiple threshold circles, drawn at the completion time of each trial activity in EEG signals, using
This research work was supported by National University of Sciences and Technology, Pakistan.