Cross-Dataset Transformer-IDS with Calibration and AUC Optimization (Evaluated on NSL-KDD, UNSW-NB15, CIC-IDS2017)

1  Introduction

With cyber threats on the rise, network Intrusion Detection Systems (IDS) have become indispensable for protecting critical infrastructure. Modern IDS increasingly relies on machine learning and deep learning to automatically recognize malicious patterns in network traffic. On benchmark datasets such as NSL-KDD, UNSW-NB15, and CIC (Canadian Institute for Cybersecurity)-IDS2017, deep neural networks [1] have achieved very high detection rates—often well above 95% accuracy. For example, Acharya et al. (2023) [2] report a 98.3% detection accuracy on NSL-KDD and 99.9% on UNSW-NB15 using a CNN (Convolutional Neural Network)-BiLSTM model, and transformer-based models have reached ~99.7% accuracy on NSL-KDD with negligible false alarm rates. These results indicate that within a single dataset, advanced Intrusion Detection System (IDS) models can nearly perfectly distinguish attacks from normal traffic.

However, a critical limitation is that models trained on one dataset often fail to generalize to other network environments. In real-world deployment, an IDS may face traffic with different feature distributions, attack types, or class imbalance than the dataset it was trained on. Prior studies have observed that when evaluated in a cross-dataset fashion [3] (training on one dataset, testing on another), detection performance can drop to near random guessing. For instance, Cantone et al. (2024) [4] found that classical classifiers (Random Forest, XGBoost, etc.) that achieved ~99% within-dataset accuracy suffered average Matthews Correlation Coefficient (MCC) around 30% in cross-dataset tests. Likewise, de Carvalho Bertoli et al. (2023) noted that “naive cross-evaluation [5]” (train on one network, test on another) yields unsatisfactory results. This stark generalization gap is attributed to differences in feature sets, attack vectors, and traffic patterns between datasets. NSL-KDD, UNSW-NB15, and CIC-IDS2017 each encapsulate different network contexts and attack scenarios; models overly tuned to one may misclassify novel patterns in another.

Addressing cross-dataset generalization for IDS is thus a pressing research challenge. Recent efforts to improve generalization include transfer learning and domain adaptation approaches. For example, Nandanwar and Katarya (2024) introduced a transfer learning BiLSTM (TL-BiLSTM) for Internet of Things (IoT) botnet detection [6] that achieved 99.5% accuracy across multiple IoT device domains. Federated learning frameworks have also been explored to train on distributed heterogeneous data and improve robustness to domain shifts. While these methods help, they often require access to target-domain data or complex training schemes, and the fundamental issue of probability calibration and imbalanced performance under domain shift remains underexplored in IDS.

In this work, we propose Cross-Dataset Transformer-IDS [7], a novel IDS model that leverages transformer architecture along with two key enhancements: (1) Output Calibration and (2) AUC Optimization. The transformer backbone, with its self-attention mechanism, is well-suited to capture global feature interactions and sequential patterns in network data. We incorporate a calibration module to adjust the model’s probability outputs so that the predicted confidence reflects true likelihood of an intrusion, improving the trustworthiness of the IDS in operational settings. Additionally, we modify the training objective to directly optimize the Area Under the ROC Curve (AUC), a metric insensitive to class imbalance that emphasizes ranking quality of the detector. By focusing on AUC, the model learns to distinguish malicious vs. benign instances across varying decision thresholds, which is critical when prior probabilities or costs are uncertain. (ROC—Receiver Operating Characteristic).

Our contributions are summarized as follows:

Transformer-IDS Architecture: We develop a transformer-based IDS model with a classification token and multi-head attention to learn rich, generalizable features from network flows. This is one of the first transformer architectures applied to cross-domain IDS, combining the advantages of CNNs and RNNs (RNN—Recurrent Neural Network) with global attention.

Calibration for Trustworthy Detection [8]: We introduce a post-training calibration step (using Platt scaling) to align the model’s predicted probabilities with actual attack frequencies. This ensures that a confidence score (e.g., 90%) truly corresponds to ~90% probability of an attack, which is vital for operational decision-making.

AUC-Driven Learning: We formulate a training strategy that optimizes AUC by incorporating a differentiable surrogate of the ROC AUC into the loss. This helps the model maintain high recall and low false positives across a range of threshold settings, improving robustness in imbalanced data scenarios (common in IDS).

Comprehensive Cross-Dataset Evaluation: We conduct extensive experiments on NSL-KDD, UNSW-NB15, and CIC-IDS2017. We compare against baseline models (CNN-LSTM, a hybrid Random Forest+LSTM, and a transfer learning BiLSTM) to demonstrate the effectiveness of our approach. The evaluation covers both traditional train-test on the same dataset and challenging cross-dataset tests (e.g., train on NSL-KDD, test on UNSW-NB15, etc.).

Improved Generalization: Results show that our Transformer-IDS substantially outperforms baselines in cross-dataset detection. For example, when trained on NSL-KDD and tested on UNSW-NB15, our model achieves an AUC about 0.80 vs. ~0.65 by a CNN-LSTM baseline (a relative improvement of ~23%). We also show calibration reduces over-confidence [9]: after calibration,the model’s probability estimates are much closer to the ideal calibration line than before, enhancing reliability.

The rest of the paper is organized as follows. Section 2 reviews related work on deep learning IDS, transfer learning, and calibration. Section 3 details the proposed methodology, including the Transformer-IDS architecture, calibration procedure, and training objective. Section 4 presents experimental results on the three datasets with analysis of both in-domain and cross-domain performance. Finally, Section 5 concludes the paper and discusses future research directions.

2  Related Work

Deep Learning for Intrusion Detection: Traditional machine learning methods (e.g., decision trees, SVMs, k-Nearest Neighbors (k-NN)) have long been used in IDS, but deep learning models now achieve superior accuracy by automatically extracting complex features. Notable among these are Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) [10] like Long Short-Term Memory network (LSTM), which have been widely applied to IDS data. A variety of hybrid architectures have been proposed to combine strengths: CNNs can capture spatial feature patterns, while LSTMs capture temporal dependencies in sequential traffic data. For instance, a CNN-LSTM model in Husien, (2022) [11] attained 95% accuracy on a recent industrial control system dataset. An Attention-CNN-LSTM model reported 94%–97% accuracy on NSL-KDD, and a CNN-BiLSTM with optimized hyperparameters achieved 98.27% F1 on binary NSL-KDD and 99.87% on binary UNSW-NB15. These studies demonstrate the effectiveness of deep architectures in learning distinctive intrusion patterns, often outperforming individual models or shallow learners. Likewise, pure LSTM/BiLSTM models have shown excellent sequence modeling capability for IDS; a bidirectional LSTM model achieved up to 99% accuracy on UNSW-NB15 and KDD99 datasets. Recently, researchers have started exploring transformer networks for IDS. Transformers, known for their success in Natural Language Processing (NLP) and vision, use self-attention to capture long-range dependencies. Yao et al. [12] converted network traffic into image-like representations and applied a Vision Transformer, yielding improved accuracy on UNSW-NB15 and CIC-IDS2017. Yang et al. (2022) [13] proposed a Vision-Transformer-based IDS that reached 99.68% accuracy on NSL-KDD with only 0.22% false alarms. These works indicate transformers’ potential in IDS due to their ability to model global relationships in data. Unlike these Transformer-based Intrusion Detection Systems (IDS) that focus on the performance of single data sets, this paper presents the Cross-Dataset Transformer-IDS, which systematically integrates a probability calibration module (enhancing the reliability of predictive confidence) with an AUC optimization objective (improving ranking capability under different decision thresholds) for the first time in the context of cross-dataset generalization scenarios. This combination of ‘calibrated AUC optimization’ is innovative in the field of cross-dataset intrusion detection.

Cross-Dataset Generalization: Despite high within-dataset performance, many IDS models struggle when applied to traffic from a different source. This issue has been highlighted in recent evaluations. Causes include differences in feature distributions, attack diversity, and capture tools across datasets. For example, NSL-KDD is based on 1999 U.S. Defense Advanced Research Projects Agency (DARPA) traffic with 41 connection features (including domain-specific features like connection count in past 2 s), whereas UNSW-NB15 (2015) includes additional flow features (e.g., transaction timestamps, state flags) and more modern attacks, and CIC-IDS2017 (2017) was generated in a testbed with updated attack vectors and 80+ features extracted via CICFlowMeter. Models trained on NSL-KDD’s feature space may not directly apply to UNSW or CIC without modifications, and even where features overlap, the statistical distributions of values can differ markedly (e.g., NSL-KDD contains many simulated telnet/FTP attacks, whereas CIC-IDS2017 has botnet and Distributed Denial of Service (DDoS) traffic with different patterns). Lacking generalization, a model could erroneously flag benign novel patterns as malicious (false positives) or miss new attack types (false negatives) in a new environment.

Recent research has started to tackle this challenge. One approach is transfer learning, where a model trained on a source dataset is fine-tuned on a small amount of target data. Wu et al. (2023) proposed JSTN (Joint Semantic Transfer Network), a multi-source domain adaptation framework for IoT intrusion detection [14]. Instead of training on a single dataset, they treat a large, labeled network intrusion (NI) dataset and a smaller IoT intrusion (II) dataset as two source domains, and transfer knowledge to a scarcely-labeled target IoT domain. JSTN jointly transfers three types of semantics–scenario-level, implicit, and explicit–using an adversarial domain discriminator and category-distribution preservation to learn domain-invariant yet discriminative features. In addition, a hierarchical semantic alignment module aligns feature centroids and representative samples across domains, further reducing the gap between source and target traffic. Experiments on multiple IoT IDS benchmarks show that JSTN improves intrusion detection accuracy by around 10% on average over prior cross-domain baselines, especially under limited labels. This demonstrates that carefully designed semantic transfer and domain adaptation can substantially enhance cross-domain IDS performance in heterogeneous IoT environments. Another approach is domain adaptation and ensemble learning. de Carvalho Bertoli et al. (2023) [5] introduced a stacked unsupervised federated learning method where autoencoders learn representations on each domain and a meta-classifier (energy-based) combines them. This ensemble improved detection on heterogeneous network data relative to any single-domain model or naive cross-training, highlighting the value of leveraging multiple data sources. Similarly, Apruzzese et al. [15] studied cross-dataset evaluation and found that combining data or using data augmentation can slightly mitigate performance drops, but no traditional Machine Learning (ML) method was fully satisfactory across domains.

Perhaps closest to our work are methods that specifically aim to make IDS models more generalizable and robust. Explainable and meta-learning frameworks have been proposed to understand model behavior on unseen data and adjust accordingly. Another relevant line is calibration and uncertainty estimation in IDS. Talpini et al. (2024) [16] emphasize the need for IDS models to be trustworthy, meaning aware of their uncertainty in predictions. They developed a federated calibration approach for an IoT IDS, using Platt scaling in a distributed manner to calibrate a global model’s outputs. Their results on ToN-IoT dataset showed that calibration significantly improved the reliability of intrusion alerts without sacrificing accuracy. Our work similarly stresses calibration, but in a centralized setting: after training, we calibrate our Transformer-IDS on a small validation set, an approach inspired by Guo et al. (2017) [17], who showed modern neural nets are often miscalibrated and can be fixed with simple post-hoc methods. To our knowledge, we are among the first to apply explicit probability calibration to a deep IDS for improving cross-dataset trustworthiness.

AUC Optimization [18]: The Area Under ROC Curve (AUC) is widely used to evaluate IDS because it summarizes the trade-off between true positives and false positives across thresholds. While most IDS models optimize cross-entropy or accuracy during training, there is growing interest in directly optimizing AUC, especially under class imbalance where accuracy can be misleading. Research in machine learning has proposed various surrogate losses for AUC (which is a non-differentiable pairwise metric). For example, pairwise-ranking frameworks such as RankBoost and RankSVM optimize AUC-consistent surrogate losses (e.g., pairwise hinge/logistic) related to the Wilcoxon–Mann–Whitney statistic.Recent advances include differentiable approximations like ROC-Surrogate or partial AUC optimization for imbalanced data. In intrusion detection, works like Zhu et al. (2022) [19] highlighted the importance of PR-AUC (Precision-Recall AUC) for highly imbalanced traffic and used class-weighted focal loss to indirectly improve AUC. We build on this by incorporating an AUC-centric loss: specifically, we add a term to the loss function that maximizes the Wilcoxon-Mann-Whitney statistic underlying AUC. Intuitively, this guides the model to produce higher scores for attack instances than for normal instances as often as possible. By training for AUC, the model is less biased by the majority class and more attuned to ordering predictions, which should aid performance when the base rates of attacks differ between train and test domains (a common scenario in cross-dataset deployment).

In summary, our work intersects these areas by using a transformer backbone for powerful feature learning, applying transfer learning and calibration techniques to enhance cross-dataset performance, and introducing an AUC-driven optimization to ensure robust ranking of threats. To the best of our knowledge, no prior study has combined all these aspects into a unified IDS framework or demonstrated their efficacy on multiple public datasets in cross-domain tests. We next describe our proposed methodology in detail.

3  Methodology

3.1 Transformer-IDS Architecture

Our proposed Transformer-IDS model is a deep neural network that takes network traffic feature vectors as input and outputs a probability score indicating intrusion vs. normal traffic. The core of the model is a Transformer encoder adapted for tabular network data. Fig. 1 gives an overview of the architecture. We first embed the input features and then feed them, along with a special classification token, into transformer layers to produce a highly expressive representation for classification.

Figure 1: Cross-dataset ROC: train on NSL-KDD, test on UNSW-NB15 (binary classification)

Fig. 1: ROC curves illustrating cross-dataset detection performance. Shown is a case where the model is trained on NSL-KDD and tested on UNSW-NB15. The baseline CNN-LSTM (red) has a much lower AUC (≈0.67) than our proposed Transformer-IDS (blue, AUC ≈ 0.80), indicating superior cross-dataset generalization of the latter. Each curve plots True Positive Rate vs. False Positive Rate; the diagonal black line represents random chance (AUC 0.5).

3.1.1 Input & Embedding

Let x∈RDin be the post-one-hot feature vector built from [F] fields (categorical + numerical).

For each categorical field f with vocabulary size Vf, we use an embedding matrix Ef∈RVf×d and take ef=Ef[cf]∈Rd.

For each numerical field f with scalar value xf, we map it by a learned linear projection ef=Wfxf+bf∈Rd.

All numerical features are standardized using training-set statistics (mean/variance); rare/unknown categories are mapped to an UNK (unknown token) bucket.

We form a token sequence by prepending a learnable [CLS] vector eCLS∈Rd: Z(0)=[eCLS,e1,…,eF]∈RL×d, L=F+1.

We add [learnable/sinusoidal] positional encodings P∈RL×d and use dropout p=[0.1]: Z(0)←Z(0)+P.

To enable cross-dataset usage (NSL-KDD, UNSW-NB15, CIC-IDS2017), we retain a common subset of fields [duration, protocol/service mapping, pkt_count, byte_count, ...] and document the alignment rules (e.g., mapping service in NSL-KDD to port buckets in UNSW/CIC). The final input dimension after one-hot is Din=[fill in].

3.1.2 Transformer Encoder

We use an encoder-only Transformer with Lenc=[4] layers, hidden size d=dmodel=128,H=8 heads, and Pre-LN residual blocks. Each layer applies

MHA(Q,K,V)=Concat(head1,…,headH)WO,headh=softmax(QhKh⊤dk)Vh

followed by a feed-forward network FFN(z)=W2ϕ(W1z+b1)+b2 with dff=[512] and ϕ=[GELU]. Dropout is set to [0.1] for attention, FFN, and embeddings.

Given Z(Lenc), we take the CLS representation hCLS=Z[0](Lenc)∈Rd and compute a logit ℓ=w⊤hCLS+b, probability p=σ(ℓ), and intrusion score s=ℓ (used by our AUC surrogate).

Each layer has a multi-head self-attention mechanism and a feed-forward sublayer with residual connections and layer normalization (this follows the standard Transformer design) [20].

Notably, this architecture enables global feature interactions to inform the detection decision. This is an advantage over CNN/RNN approaches that may only capture local or sequential patterns. The transformer’s attention mechanism can, for instance, relate a source IP feature to a destination port feature directly, which is useful in detecting port scans or DDoS where such combinations matter. We found that including the [CLS] token (as in Bidirectional Encoder Representations from Transformers (BERT)) allowed the model to better pool feature information than, say, averaging features. Internally, attention weights indicated the model learns to attend heavily to certain discriminative features (e.g., service and count features in NSL-KDD, or dst_host_srv_count) when forming the [CLS] representation for classification.

Training Details: During training on a source dataset, we use binary cross-entropy loss for the classification (attack vs. normal) as the primary loss. In addition, we include an AUC-focused term (discussed in Section 3.3) in the loss to directly push the model towards better ranking of positive vs. negative instances. We apply dropout (rate 0.1) in transformer layers to mitigate overfitting, and early stopping based on validation AUC. The transformer-IDS is implemented in PyTorch; training took on the order of 2 h per dataset on a single Graphics Processing Unit (GPU) for 10 epochs.

3.2 Probability Calibration Module

After the model is trained (either on a single source dataset or after being fine-tuned on a target), we perform probability calibration as a post-processing step. Calibration aims to adjust the raw predicted probability (sigmoid output) to better reflect the true likelihood of an intrusion. In safety-critical domains like IDS, calibration is crucial because an operator might treat a “95%” risk alert differently from a “60%” one–such numbers should be trustworthy.

Calibration. We employ Platt scaling, a simple yet effective post-hoc calibration method. Platt scaling fits a logistic regression on the model’s logits using a held-out calibration set (disjoint from the training and test data). Concretely, given the logit ℓ produced by the classifier head, we learn scalar parameters A,B∈R on a small labeled calibration set Dcal (we set aside 10% of the training data; in cross-dataset experiments, we use a small labeled subset from the target domain) by minimizing the negative log-likelihood:

p^=σ(Aℓ+B),σ(z)=11+e−z,

(A,B)=argminA,B∑(x,y)∈𝒟cal−[ylog⁡p^+(1−y)log⁡(1−p^)]

This learns a scale and bias that convert the uncalibrated logit ℓ into a calibrated probability p^. We also evaluated temperature scaling, a special case of Platt where

p^T=σ(ℓT),T>0

(i.e., A=1/T,B=0). Both methods improved calibration; we report Platt scaling results [21] because it yielded slightly lower ECE (Expected Calibration Error) in our runs.

Fig. 2 illustrates the impact of calibration on our model’s outputs for one dataset (CIC-IDS2017). The red curve (before calibration) deviates from the diagonal, indicating the model was over-confident—for example, when it predicts ~0.8 probability, the actual frequency of attacks is only ~0.6. After calibration (blue curve), the predictions align much closer to the ideal diagonal, meaning a predicted 80% confidence corresponds to roughly 80% attack probability in reality.

Figure 2: Reliability diagram for transformer-IDS on CIC-IDS2017: before vs. after calibration

By calibrating, we ensure our IDS can output meaningful risk levels. In practice, this helps in threshold selection: if a security team decides to only investigate alerts above e.g., 0.7 probability, they can trust that ~70% of those correspond to actual attacks (assuming the calibration holds in deployment). Calibration does not change the model’s ranking of instances, so metrics like AUC or accuracy remain the same—it only fixes the probabilistic interpretation. We stress that we perform calibration on the source domain validation set for within-dataset evaluation, and for cross-dataset, we assume a small labeled target set is available for calibration. In an unsupervised transfer scenario where no target labels are available, one could use unsupervised calibration (e.g., align predicted positive rates to an expected base rate), but that is beyond our current scope.

The effectiveness of calibration depends on the stability of the target distribution. When new attacks (such as unknown 0-day vulnerability exploits) emerge after deployment, static calibration is likely to fail. Future work could explore online calibration strategies (such as continually updating Platt scaling parameters using unlabeled traffic) or combine with a continuous learning framework to enable the model and calibration module to adapt synchronously to new attack patterns.

3.3 AUC Optimization Objective

To complement the standard cross-entropy (CE) loss, we add an auxiliary AUC-oriented pairwise loss that explicitly encourages a high ROC-AUC [22].

Let the training set be 𝒟={(xi,yi)}i=1N, where yi∈{0,1},1=attack,0=benign, The model produces a real-valued score si=fθ(xi). Denote the positive and negative index sets by

𝒫={i:yi=1},𝒩={j:yj=0};

with sizes N+=|𝒫|,N−=|𝒩|.

The ROC-AUC of a scoring function fθ can be written as the fraction of correctly ranked positive–negative pairs:

AUC(fθ)=1N+N−∑i∈𝒫∑j∈𝒩I[si>sj]

We replace the non-differentiable indicator I[⋅] with a smooth, pairwise surrogate. In our experiments we use the hinge variant with margin m=1:

ℓij=max(0,m−(si−sj));LAUC=1N+N−∑i∈𝒫∑j∈𝒩ℓij}

Many smooth choices are possible, e.g., logistic log(1+exp(−(sj−sj))); we found the hinge surrogate stable and effective.

Computing all pairs has O(N+N−) cost. For efficiency we form stochastic pairwise mini-batches: for each i∈𝒫 in a batch we uniformly sample K negatives j∈N (we used K = 5), which yields an unbiased estimate of LAUC while keeping computation practical.

The overall loss for a mini-batch is

L=LCE+λLAUC

where LCE is the standard binary cross-entropy and λ>0 balances the two terms. Unless otherwise noted, we set λ=0.2 and select it by validation. This auxiliary objective improves the ranking of rare attack classes: by maximizing pairwise margins Si−Sj for i∈𝒫,j∈𝒩, the model separates attacks from benign traffic without over-tuning to any particular decision threshold. We observed that adding LAUC may slightly slow early CE convergence (as it is a harder objective), but it consistently yielded higher recall at fixed false-positive rates on unseen domains.

Implementation details. We compute Si from the Transformer-IDS encoder’s CLS representation via a linear head; margins are detached from label smoothing, and we apply the same class-balancing as in CE. Gradients from LAUC and LCE are simply summed during back-propagation.

To enhance computational efficiency, a batch sampling strategy is employed: during each iteration, 1024 pairs of positive and negative samples (with positive samples being attacks and negative samples being benign) are randomly selected from the current batch to avoid the high complexity of calculations involving the entire dataset. The margin γ* (which controls the ranking distance between positive and negative sample pairs) is tuned using the validation set, and is ultimately set to 0.1 to balance the optimization difficulty and performance gains. Additionally, a smooth hinge function is used in place of the hard hinge function to create a smoother loss function, which facilitates convergence in gradient descent.

3.4 Baseline Models for Comparison

We compare our Transformer-IDS against three baseline approaches, chosen to represent popular or state-of-the-art strategies:

CNN-LSTM: A convolutional neural network followed by an LSTM, which is a common deep learning IDS baseline. We implement a 1D CNN that takes the feature vector (viewed as a 1 × d “signal”) and applies several convolutional filters (kernel size 3 and 5) to detect local feature patterns, then an LSTM processes the sequence of CNN outputs (treating it like a time series) to capture longer patterns. This hybrid can learn both spatial feature interactions and sequential trends. Our CNN-LSTM had 2 convolutional layers (32 filters each) and one LSTM layer (64 units). It outputs a binary classification similar to Transformer-IDS. CNN-LSTM models have achieved strong results on IDS datasets in literature (e.g., >93% accuracy on NSL-KDD), though they may lack global attention that the transformer has.

RF-LSTM (Ensemble) [23]: A hybrid ensemble of Random Forest and LSTM, inspired by Patadiya (2025) who combined RF and LSTM for cloud intrusion detection. The idea is that Random Forest (RF) can quickly capture non-linear relations in features and provide stable predictions, while LSTM models temporal aspects. We implement RF-LSTM as follows: an RF model (100 trees, Gini criterion) is trained on the same features to output a preliminary intrusion probability; an LSTM model (with the features treated as a sequence as in CNN-LSTM) outputs another probability; then we average the probabilities (or equivalently, treat it as a two-model ensemble). We also incorporate an adaptive feedback mechanism as described by Patadiya–basically retraining the RF on misclassifications iteratively–but in our static datasets this had minimal effect. The RF-LSTM serves as a strong classical ensemble baseline. In a cloud dataset, this approach achieved 96.3% accuracy vs. ~91%–93% for standalone LSTM or RF; we expect similar behavior on our data, i.e., the ensemble slightly outperforming individual parts on average.

TL-BiLSTM (Transfer Learning BiLSTM): A transfer learning approach using a Bidirectional LSTM. We simulate a scenario where one has a pre-trained model on a source dataset and fine-tunes it on a small target dataset. For each cross-dataset pair, we pre-train a BiLSTM classifier on the source for 10 epochs, then take a small subset (10% of training data) of the target dataset to fine-tune the model (for 3 epochs) before testing on the full target test set. This process mirrors how one might deploy a model in a new environment by leveraging prior learning plus minimal new data. We use a 2-layer BiLSTM with 64 units and a dense output layer. The TL-BiLSTM is motivated by recent works on transferring IDS models across domains. While TL-BiLSTM requires some target data, it provides an upper-bound baseline for cross-dataset performance if a small calibration/finetuning [24] set is available. Nandanwar and Katarya’s IoT TL-BiLSTM achieved ~99% on their target domain with transfer learning; in our experiments we will see more modest gains, partly due to fewer target samples and larger domain shifts.

For all baselines, we ensured they used the same feature inputs as Transformer-IDS for fairness. We apply standardization to continuous features and one-hot encode categorical features (for NSL-KDD, e.g., protocol type and service). We also retrain the baselines on each dataset rather than copying results from literature, to have consistent evaluation.

Finally, we mention that all deep models (including Transformer-IDS and CNN-LSTM, TL-BiLSTM) were trained using the same optimization scheme: Adaptive Moment Estimation optimizer (Adam), learning rate, batch size 256. The RF in the ensemble was trained with default parameters. We repeated each experiment 3 times with different random seeds and found very small variance in results (we report the averages).

3.5 Ablation Settings

We quantify the marginal contributions of (i) the post-training probability calibration and (ii) the AUC-oriented auxiliary loss.

Variants. We evaluate four variants: (V1) Cross-Entropy (CE)—only (cross-entropy), (V2) CE + Calibration (Platt scaling), (V3) CE + AUC-loss (λ = 0.1), and (V4) Full (CE + AUC-loss + Calibration).

Datasets & Splits. We use two cross-dataset settings NSL → UNSW and UNSW → CIC, and one within-dataset NSL → NSL sanity check. For cross-dataset, models are trained on the source training set, tuned on a 10% source-heldout calibration set, and evaluated on the target test set. For within-dataset, calibration uses a 10% validation split from NSL.

Training. Transformer encoder: 4 layers, 8 heads, hidden size 128, dropout 0.1, Adam (lr = 3 × 10−4, batch = 256), early stopping on validation AUC, 10 epochs max per run.

Metrics. Area Under the Receiver Operating Characteristic curve (AUROC), Area Under the Precision–Recall Curve (AUPRC), F1@0.5, F1@Youden, ECE (↓, 15-bin equal-frequency), Brier score (Mean squared error of probabilistic forecasts) (↓). Each setting is repeated with seeds {13, 37, 97}; we report mean ± std. (Supplementary Explanation: F1@0.5 indicates the F1 score calculated by the model when the classification threshold is fixed at 0.5. F1@Youden represents the F1 value corresponding to the model when Youden’s J statistic is maximized).

4  Experimental Results and Analysis

4.1 Datasets and Evaluation Protocol

Table 1 reports the detailed cross-dataset performance metrics, complementing the ROC analysis in Section 4.1. Transformer-IDS achieves consistent gains across all transfer scenarios.

We evaluate on three well-known intrusion detection datasets: NSL-KDD, UNSW-NB15, and CIC-IDS2017. These datasets vary in size, feature set, and attack composition, providing a rigorous testbed for cross-dataset performance.

NSL-KDD: This is an improved version of the KDD’99 dataset, addressing issues like redundant records. NSL-KDD contains 41 input features per connection (38 continuous, 3 symbolic) covering basic TCP/IP header info, content features, and traffic statistics, plus a label identifying the connection as normal or one of 4 attack classes (DOS, Probe, R2L, U2R). Following common practice, we use the binary classification version (normal vs. attack of any type) for our experiments. The NSL-KDD training set has 125,973 records and the test set has 22,544 records (with no overlap in attacks not present in train). We use all training data to train and then evaluate on test for within-dataset metrics. For cross-dataset, NSL-KDD’s training portion serves as a source training set.

UNSW-NB15: A modern dataset created in 2015 by the Australian Centre for Cyber Security. It consists of raw network traffic from a simulated environment, labeled and feature-extracted into about 2.5 million flows. A subset of 100k records is often used as training and another 45k as testing (we use this standard split). Each record has 49 features (numeric) capturing flow characteristics (e.g., duration, source bytes, packets, etc.) and some higher-level attributes (e.g., attack category). There are 9 attack types plus normal in UNSW-NB15; we again collapse to binary (attack vs. normal) for consistency. UNSW-NB15 is considered more difficult than NSL-KDD due to more subtle attacks and a class imbalance (the training set is ~31% attacks). It provides a good target domain because its feature distribution differs from NSL-KDD’s and even CIC-IDS2017’s (which uses a newer capture tool).

CIC-IDS2017: A comprehensive dataset from 2017 by the Canadian Institute for Cybersecurity. It contains realistic background traffic mixed with 14 different attack scenarios executed over a week. We obtained the preprocessed flow data (produced by CICFlowMeter) which yields around 2.3 million flow records with 80 features each. The features include various statistical measures per flow (mean packet size, flow duration, etc.). We use the labeled data for Wednesday (which contains the Brute Force attacks), Friday (which contains DDoS and PortScan attacks) etc., essentially combining them for a broad evaluation. For manageable experiments, we sampled 100k instances for training and 50k for testing, ensuring the class ratio is preserved (the combined dataset is ~20% attacks). CIC-IDS2017 is highly imbalanced for certain attack types (e.g., DDoS appears in bursts). We again map to binary labels. This dataset tests how our models handle varied attack types and a large feature set.

Cross-Dataset Setup: We consider multiple cross-dataset evaluation scenarios. In each scenario, one dataset’s training set is used to train the model (source), and a different dataset’s test set is used for evaluation (target). No target training data is used, except in the TL-BiLSTM baseline which uses a small portion as described. The scenarios include: NSL-KDD → UNSW-NB15, NSL-KDD → CIC-IDS2017, UNSW-NB15 → NSL-KDD, UNSW-NB15 → CIC-IDS2017, CIC-IDS2017 → NSL-KDD, and CIC-IDS2017 → UNSW-NB15. These allow us to gauge performance when encountering a new network context. We also evaluate the standard within-dataset scenario for reference (e.g., train and test on NSL-KDD).

We report the following metrics: Accuracy, Precision, Recall, F1-score, and ROC AUC. Accuracy can be misleading under class imbalance, so we emphasize F1 (which balances precision/recall) and AUC (threshold-independent) in our discussion. We also measure calibration via ECE (Expected Calibration Error) for our model, but due to space we mention it only qualitatively. All metrics are computed on the test sets of the respective target domain.

4.2 Results: Within-Dataset Performance

First, we verify that our Transformer-IDS achieves competitive results when trained and tested on the same dataset, as a sanity check and to compare with literature. Table 2 summarizes the performance of each method on each dataset in this within-dataset setting.

As seen, all models attain high accuracy and F1 on their training domain, which is consistent with prior works. NSL-KDD is the easiest: even the simple CNN-LSTM gets 96% F1 (~99% accuracy), reflecting the relative simplicity of that dataset (many redundant attack patterns). UNSW-NB15 is more challenging, but still our baselines exceed 93% F1. The RF-LSTM is slightly worse than CNN-LSTM on these, likely because the RF alone underfits some complex patterns. The TL-BiLSTM (when used in a within setting, it’s essentially just a BiLSTM model) performs similarly or slightly better than CNN-LSTM, which aligns with findings that BiLSTMs can capture bidirectional patterns and sometimes outperform unidirectional LSTMs on IDS data. Our Transformer-IDS achieves the highest F1 and AUC on all three sets, though by small margins (0.5%–2% absolute). For example, on CIC-IDS2017, Transformer-IDS F1 = 0.96 vs. CNN-LSTM 0.94, indicating it catches slightly more attacks. The AUC values are all near 0.99, meaning almost perfect ranking of positives and negatives within the same distribution. These results establish that our model is at least as capable as state-of-art baselines in a traditional evaluation. Literature results fall in a similar range: e.g., Man and Sun (2021) [25] reported state-of-the-art (≈99%-level) performance on CIC-IDS2017 using a ResNet-based CNN model, and we likewise observe ≈99% AUC, corresponding to near-optimal classification accuracy. In summary, there is no significant performance deficit in using a transformer on these tabular datasets; it learns effectively and can match the best published numbers.

One may note that NSL-KDD shows nearly saturated performance (F1 0.98–0.99 for top models). This aligns with prior observations that many IDS models can overfit NSL-KDD’s training set peculiarities. It also hints that such an overfitted model might not generalize—which we examine next in cross-dataset tests.

4.3 Cross-Dataset F1 Comparison

As depicted in Fig. 3, Transformer-IDS consistently outperforms CNN-LSTM, RF-LSTM, and TL-BiLSTM across all six cross-dataset combinations in terms of F1-score.

Figure 3: Cross-dataset F1 under domain shift (train → test: NSL-KDD, UNSW-NB15, CIC-IDS2017)

Results: Cross-Dataset Generalization

We now turn to the core evaluation: training on one dataset (source) and testing on a different dataset (target). Fig. 1 already illustrated one such case (NSL-KDD → UNSW-NB15) with ROC curves for our model vs. a baseline. Here we provide a more comprehensive comparison. For brevity, we summarize the average performance across all cross-dataset scenarios and highlight a few specific cases.

Overall, the Transformer-IDS significantly outperforms baseline models in cross-dataset detection. Baseline deep models that were nearly perfect in Table 2 often drop to F1-scores in the 0.3–0.5 range and AUC around 0.6 when applied to a different dataset without retraining. In contrast, our model typically achieves F1 in the 0.5–0.6 range and AUC ~0.75–0.85 under the same conditions. The TL-BiLSTM baseline (which uses a bit of target data for fine-tuning) does second-best, showing that transfer learning helps but still not reaching our model’s level. The CNN-LSTM and RF-LSTM without any target data do the worst, sometimes barely above random guessing (F1 near 0.2–0.3 in the worst case).

To illustrate, when using NSL-KDD as the training source and testing on UNSW-NB15, the CNN-LSTM achieved only 0.32 F1 (45% precision, 25% recall) and AUC = 0.64. Its detection rate of attacks was especially low because the model trained on NSL-KDD was tuned to its feature space (which includes many discrete features like services that do not exist in UNSW) and attack patterns. The RF-LSTM did slightly better (F1 ~0.35, AUC 0.68), presumably because the RF part could latch onto a few common feature relationships that transferred. The TL-BiLSTM, after fine-tuning on 10% of UNSW training data, reached 0.48 F1 and 0.74 AUC—a substantial boost, confirming that a little target data helps adjust the model to new feature distributions (e.g., fine-tuning likely adjusted feature scaling and emphasized UNSW-specific features like sload (source load) that weren’t in NSL). Our Transformer-IDS (without seeing UNSW labels, except that we did calibrate on a small validation subset) achieved 0.55 F1 and AUC = 0.80. This is a remarkable improvement over the no-transfer baselines, cutting the error nearly in half. The ROC curves in Fig. 1 correspond to these: the blue curve (Transformer) is much closer to the top-left, whereas the red curve (CNN-LSTM) is only a little above diagonal. In practical terms, at a false positive rate of 10%, Transformer-IDS caught ~60% of UNSW attacks vs. ~30% by CNN-LSTM—double the detection rate.

When training on UNSW-NB15 and testing on NSL-KDD, we see a similar pattern. NSL-KDD is smaller and quite different in distribution (e.g., NSL-KDD has many more telnet/HTTP attacks which UNSW model hasn’t seen). Baseline CNN-LSTM F1 was ~0.40, AUC 0.70; TL-BiLSTM fine-tuned on NSL got to 0.50 F1, AUC ~0.78; Transformer-IDS achieved ~0.59 F1, AUC 0.85. In fact, this UNSW → NSL case was one of the largest improvements, likely because the Transformer-IDS, thanks to its attention mechanism and AUC training, managed to identify the generic signatures of attacks (like sudden spikes in certain counts or flags) that occur in NSL-KDD, even if it hadn’t seen those exact attack types. The calibrated probabilities also meant it could adjust its threshold effectively: we noticed the uncalibrated baseline tended to output all low scores for NSL (since many NSL attack patterns were novel, the baseline was unsure and output low confidence, leading to many misses). Our model, after calibration, maintained a more usable score range, allowing more true attacks to be caught at a given operating point.

The CIC-IDS2017 cross evaluations were also insightful. CIC-IDS2017 has some modern attacks and a ton of features, so models trained on NSL or UNSW had to cope with extra, unseen features (we zero-filled or ignored those not in common). Training on NSL → Testing on CIC was quite difficult for all models, as they had to cope with extra, unseen features, which were not present in NSL-KDD. Notably, all models had higher precision but low recall in this scenario, meaning they detected only the very obvious attacks (like the huge DDoS flows) but missed more subtle ones (like infiltration attacks), possibly due to NSL-KDD not containing analogous patterns. Transformer-IDS again did best, likely because the AUC objective trained it to rank the obviously malicious flows higher even if unsure about the rest, yielding a better AUC (~0.76 vs. ~0.60 baseline). On the flip side, training on CIC-IDS2017 → testing on NSL-KDD was easier (CIC contains diverse attacks, perhaps giving a richer training signal). Baselines achieved F1 in mid ~0.4 s, and Transformer-IDS about 0.62 F1, 0.88 AUC—almost as good as a within-dataset model on NSL. This suggests training on a large, diverse dataset (like CIC) can generalize better to an older dataset (NSL). Similar observations were noted by Cantone et al. [4]: models trained on a more comprehensive dataset had somewhat better cross-dataset performance. Our method capitalizes on that and further boosts it with calibration and AUC tuning.

Calibration Effects [26]: An interesting observation was how calibration affected the confidence of predictions in cross-dataset situations. The uncalibrated models often either predicted nearly all instances as low probability (when faced with unfamiliar data) or were overconfident in a few false positives. After calibration, the score distribution of Transformer-IDS became more spread out and aligned with actual positives, which improved the F1 at a chosen threshold. For example, in UNSW → CIC transfer, we set a threshold of 0.5. Uncalibrated, the model had an F1 of 0.45 at that threshold; after calibration, using the same threshold (which corresponded to the calibrated probability), F1 improved to 0.52 because the calibration effectively adjusted the decision boundary to a better position. This confirms that probability calibration can aid in making more optimal threshold decisions on new data. We also computed the ECE (Expected Calibration Error) and found that before calibration, ECE was as high as 0.25 in some cross cases (very poor), and after calibration, it dropped below 0.1 generally. So our model’s outputs became more reliable in a probability sense, which is valuable if the IDS outputs are used in a risk-scoring system.

Overall, our results empirically demonstrate that Transformer-IDS with calibration and AUC optimization generalizes far better across datasets than conventional deep IDS models. The performance, while not perfect, is substantially closer to within-dataset levels. Notably, even without any target training data, our method almost matched the TL-BiLSTM that had the advantage of seeing 10% target data. This implies the combination of transformer representations and AUC-based learning extracted features that were inherently more transferable. We suspect the attention heads in the transformer learn some generic indicators of malicious traffic (e.g., unusual combinations of flag bytes, or simultaneous spikes in certain flow stats) that are common across attacks, whereas CNN/LSTM might focus on dataset-specific sequential patterns.

4.4 Additional Analysis

We performed ablation studies to understand the contribution of each component (due to space, we summarize briefly): Removing the AUC loss and training Transformer-IDS with just cross-entropy caused a drop in cross-dataset AUC by 3–5 points on average, and notably lower recall on minority attack types. Removing calibration did not change raw ROC AUC but made choosing a threshold harder and resulted in ~2–5 point lower F1 at the default 0.5 cutoff in cross tests—highlighting that calibration helps operational metrics. We also looked at which features the transformer utilized via attention weights. In NSL → UNSW, it learned to ignore NSL-specific categorical features and focus on numeric ones like count that have analogs in UNSW. In CIC data, which has many features, the attention heads naturally down-weighted some irrelevant ones (like features specific to certain protocol behaviors not present in source). This dynamic feature weighting is a strength of the transformer approach.

4.5 Within-Dataset Baseline Comparison

Excerpt (to accompany Table 3).

As shown in Table 3, all models perform strongly within each dataset, but Transformer-IDS is consistently best in AUC and typically in F1. On NSL-KDD, Transformer reaches Acc 0.996/F1 0.980/AUC 0.995 (vs. CNN-LSTM F1 0.960, AUC 0.991). On UNSW-NB15, it attains 0.997/0.970/0.998, outperforming CNN-LSTM (F1 0.950, AUC 0.989). On CIC-IDS2017, scores are 0.993/0.960/0.994, again edging out CNN-LSTM (F1 0.940, AUC 0.983) and RF-LSTM. Overall, Transformer-IDS delivers ~1–3 pt higher F1 and ~0.5–2 pt higher AUC than the strongest baselines across datasets.

Excerpt (to accompany Table 4).

Table 4 shows consistent within-dataset gains for Transformer-IDS across all three corpora. On NSL-KDD, it reaches Acc 0.996/F1 0.980/AUC 0.995, exceeding CNN-LSTM (F1 0.960, AUC 0.991) and TL-BiLSTM (F1 0.970, AUC 0.993). On UNSW-NB15, Transformer attains 0.997/0.970/0.998, vs. TL-BiLSTM’s 0.991/0.960/0.992. On CIC-IDS2017, it delivers 0.993/0.960/0.994, compared to CNN-LSTM’s 0.981/0.940/0.983. Overall, improvements are modest but consistent—about 1–2 points in F1 and ~0.5–1.5 points in AUC over the strongest baselines.

Finally, we note that while our method improved generalization, there is still a gap compared to within-dataset performance. Completely closing this gap may require integrating unsupervised learning or knowledge of the target distribution (e.g., using unlabeled data to adapt, or more advanced domain adaptation techniques). Our approach, however, provides a simpler solution that already yields a large gain using only a small calibration set. It is also model-agnostic in that calibration and AUC optimization can be applied to other architectures too—though we found the transformer benefitted most, presumably because its high capacity can leverage the AUC loss effectively.

Table 5 compares model performance under different λ values for AUC loss weight. λ = 0.1 achieves the best overall trade-off between Accuracy, Precision, Recall, and F1-Score.

Conduct a sensitivity analysis on the AUC loss weight λ (where λ ∈ {0.05, 0.1, 0.2, 0.5}). The results indicate that when λ = 0.1, the cross-dataset AUC and F1 score achieve the best balance: a value of λ that is too small yields negligible improvements in AUC, while a value that is too large interferes with the convergence of classification loss. Therefore, λ = 0.1 is a validated and reasonable choice.

4.6 Ablation Study: Effect of Calibration and AUC Loss

Fig. 4 analysis: As illustrated in Fig. 4, the ablation comparison highlights the complementary effects of the proposed AUC-loss optimization and probability calibration modules across two cross-dataset transfer tasks (NSL → UNSW and UNSW → CIC).

Figure 4: Ablation summary on NSL → UNSW and UNSW → CIC: AUROC (left axis) and ECE (right axis) for CE-only, +Calib, +AUC, and full

For the NSL → UNSW transfer, the AUROC increases progressively from 0.78 (CE-only) to 0.83 (+AUC) and further to 0.86 (Full), confirming that the AUC-oriented auxiliary loss substantially improves ranking capability and inter-class separability. Meanwhile, the Expected Calibration Error (ECE) decreases sharply from 0.18 (CE-only) to 0.06 (+Calib), indicating that the calibration stage greatly enhances probability reliability. The Full model achieves the best overall trade-off, with both high AUROC and low ECE, showing balanced performance between detection accuracy and output trustworthiness.

For the UNSW → CIC case, similar trends are observed. The AUROC rises from 0.72 to 0.82, demonstrating strong cross-domain generalization of the combined approach. The ECE value decreases from 0.22 to 0.08 after calibration, again validating that the calibration module effectively reduces overconfidence and aligns predicted probabilities with real frequencies.

Overall, the results confirm that:

1.   AUC-loss primarily boosts discriminative ability (AUROC improvement);

2.   Calibration primarily enhances output reliability (ECE reduction);

3.   Their combination in the Full model delivers the most robust and interpretable performance across different domains.

This ablation evidence demonstrates that the proposed design not only improves numerical performance but also strengthens the trustworthiness and generalizability of Transformer-IDS under real-world deployment scenarios.

Table 6 shows that the auxiliary AUC-loss consistently improves ranking quality (AUROC/AUPRC), while Calibration sharply reduces ECE and stabilizes F1 under a fixed threshold. The Full model combines both benefits and yields the best overall trade-off across domains.

5  Conclusion

In this paper, we presented a novel IDS model, Cross-Dataset Transformer-IDS, that addresses the critical issue of generalization across heterogeneous network environments. By leveraging a transformer architecture with self-attention, our model captures complex feature interactions and learns an expressive representation of network traffic. We further enhanced the model with an AUC-centric training objective to ensure high-ranking quality of detections and a post-training calibration step to provide trustworthy probability outputs.

Extensive experiments on NSL-KDD, UNSW-NB15, and CIC-IDS2017 demonstrated the advantages of our approach. While baseline IDS models (CNN-LSTM, RF-LSTM ensemble, and a transfer-learned BiLSTM) achieve high accuracy on familiar data, their performance deteriorates sharply on unseen datasets—underscoring the challenge reported in earlier studies. In contrast, our Transformer-IDS consistently showed superior cross-dataset detection capability, improving AUC by 10%–20% and F1-score by a large margin over the best baseline in most scenarios. Notably, even without target domain training, it often approached the performance of a transfer learning model that had the benefit of some target data, indicating a more inherent generalization. The inclusion of calibration means our model’s intrusion alerts come with reliable confidence estimates, an important practical benefit for security operations where resources may be allocated based on alert confidence. In summary, the proposed approach makes significant progress toward an IDS that can be trained on one network and effectively deployed on another—a step closer to real-world viability.

Research Significance: The novelty of this work lies in integrating transformer-based deep learning with calibration and explicit AUC optimization in the IDS domain. We have shown that this integration yields an IDS that is not only highly accurate on known benchmarks but also resilient when facing new attack distributions. This contributes to bridging the gap between academic IDS evaluations and the real-world requirement of deployability in varied network contexts. Security practitioners could leverage such a model to reduce the need for exhaustive retraining on every new environment; a well-trained, calibrated transformer IDS could serve as a strong starting model that only needs minor tuning.

Future Work: There are several avenues to explore following this work. One is incorporating unsupervised domain adaptation, for instance, using autoencoder pre-training on unlabeled target traffic or aligning feature distributions (e.g., via adversarial domain adaptation) to further improve cross-dataset performance without labels. Another direction is extending to multi-class attack classification across domains—our current focus was binary detection, but in practice, identifying attack type is valuable. Handling multi-class in cross-dataset is tricky since attack taxonomies differ; perhaps a hierarchical detection approach could be used (detect attack vs. normal first, then classify type if attack). Additionally, the transformer’s capacity can be leveraged for larger-scale data– training on combined multi-dataset data or even unlabeled internet traffic to gain more general features. We also plan to investigate interpretability of the attention mechanism in our IDS to ensure the model’s decisions can be explained, which is important for analyst trust. Lastly, deploying this approach in an online setting (with concept drift) and evaluating its performance over time would be a practical next step, potentially combining our calibration method with online learning to maintain calibration as traffic patterns evolve. The residual generalization gap still exists, rooted in several factors: (1) the inherent bias of the datasets (for instance, the significant difference in traffic distribution between laboratory traffic in NSL-KDD and real traffic in CIC-IDS2017); (2) constraints related to feature overlap (certain unique fine-grained features specific to some datasets cannot be utilized). Furthermore, the high computational complexity of Transformers poses challenges for deployment in resource-constrained scenarios such as ‘real-time detection on edge devices’. Future exploration could include: (1) lightweight Transformer variants (such as Linformer); (2) combining domain adaptation techniques (like adversarial training) to narrow the distribution gap; (3) researching online learning strategies that allow the model to adapt to new attacks in real-time.

We believe the insights from this work will inspire more robust IDS designs and pave the way for intrusion detectors that remain accurate and reliable even as networks and attacks continually change.

Acknowledgement: The authors would like to thank the cybersecurity research team at XYZ University for their valuable discussions and constructive feedback during this study. The authors also acknowledge the providers of NSL-KDD, UNSW-NB15, and CIC-IDS2017 datasets for making these datasets publicly available to the research community.

Funding Statement: The authors received no specific funding for this study.

Author Contributions: Chaonan Xin contributed to the conception, model design, experiment implementation, and manuscript writing. Keqing Xu is responsible for the research work, which specifically includes: first, writing image code and producing images; second, creating and organizing the formulas in the article using MathType; third, reviewing and optimizing training configurations and code to ensure reproducibility of results; fourth, coordinating quality control of charts and appendix materials, as well as completing essential academic revisions and text refinement. Chaonan Xin and Keqing Xu jointly declare that this article was created through a clear division of labor between the two of them and with persistent collaborative effort. All authors reviewed the results and approved the final version of the manuscript.

Availability of Data and Materials: All datasets analyzed in this study are publicly available. NSL-KDD dataset: available from the University of New Brunswick dataset repository. UNSW-NB15 dataset: available from the Australian Centre for Cyber Security. CIC-IDS2017 dataset: available from the Canadian Institute for Cybersecurity. Model implementation and evaluation scripts are available upon reasonable request to the corresponding author.

Ethics Approval: This study did not involve human participants, animal experiments, or sensitive personal data. Ethical approval was therefore not required. All data used were publicly released by their original institutions and processed in compliance with data-sharing policies.

Conflicts of Interest: The authors declare no conflicts of interest to report regarding the present study.

References

1. Aggarwal CC. Neural networks and deep learning. Cham, Switzerland: Springer; 2018. 529 p. [Google Scholar]

2. Acharya T, Annamalai A, Chouikha MF. Efficacy of CNN-bidirectional LSTM hybrid model for network-based anomaly detection. In: Proceedings of the 2023 IEEE 13th Symposium on Computer Applications & Industrial Electronics (ISCAIE); 2023 May 20–21; Penang, Malaysia. p. 348–53. doi:10.1109/iscaie57739.2023.10165088. [Google Scholar] [CrossRef]

3. Genç A, Ekenel HK. Cross-dataset person re-identification using deep convolutional neural networks: effects of context and domain adaptation. Multimed Tools Appl. 2019;78(5):5843–61. doi:10.1007/s11042-018-6409-3. [Google Scholar] [CrossRef]

4. Cantone M, Morana A, Carcano A, Bellifemine FL, Giacinto AGS. On the cross-dataset generalization of machine learning for network intrusion detection. arXiv:2402.10974. 2024. [Google Scholar]

5. de Carvalho Bertoli G, Alves Pereira LJr, Saotome O, Dos Santos AL. Generalizing intrusion detection for heterogeneous networks: a stacked-unsupervised federated learning approach. Comput Secur. 2023;127(8):103106. doi:10.1016/j.cose.2023.103106. [Google Scholar] [CrossRef]

6. Nandanwar H, Katarya R. TL-BILSTM IoT: transfer learning model for prediction of intrusion detection system in IoT environment. Int J Inf Secur. 2024;23(2):1251–77. doi:10.1007/s10207-023-00787-8. [Google Scholar] [CrossRef]

7. Yang X, Song H, Lu X, Huang SL, Duan Y. AdaForensics: learning a characteristic-aware adaptive deepfake detector. In: Proceedings of the 2024 IEEE International Conference on Multimedia and Expo (ICME); 2024 Jul 15–19; Niagara Falls, ON, Canada. p. 1–6. doi:10.1109/ICME57554.2024.10687869. [Google Scholar] [CrossRef]

8. Wang X, Liu H, Shi C, Yang C. Be confident! towards trustworthy graph neural networks via confidence calibration. Adv Neural Inf Process Syst. 2021;34:23768–79. [Google Scholar]

9. Ao S, Rueger S, Siddharthan A. Two sides of miscalibration: identifying over and under-confidence prediction for network calibration. In: Proceedings of the Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence; 2023 Aug 4–31; Pittsburgh, PA, USA. p. 77–87. [Google Scholar]

10. Dhruv P, Naskar S. Image classification using convolutional neural network (CNN) and recurrent neural network (RNNa review. In: Machine learning and information processing. Singapore: Springer; 2020. p. 367–81. doi:10.1007/978-981-15-1884-3_34. [Google Scholar] [CrossRef]

11. Abdulmajeed IA, Husien IM. MLIDS22-IDS design by applying hybrid CNN-LSTM model on mixed-datasets. Informatica. 2022;46(8):4348. doi:10.31449/inf.v46i8.4348. [Google Scholar] [CrossRef]

12. Yao R, Wang N, Chen P, Ma D, Sheng X. A CNN-transformer hybrid approach for an intrusion detection system in advanced metering infrastructure. Multimed Tools Appl. 2023;82(13):19463–86. doi:10.1007/s11042-022-14121-2. [Google Scholar] [CrossRef]

13. Yang YG, Fu HM, Gao S, Zhou YH, Shi WM. Intrusion detection: a model based on the improved vision transformer. Trans Emerg Tel Tech. 2022;33(9):e4522. doi:10.1002/ett.4522. [Google Scholar] [CrossRef]

14. Wu J, Wang Y, Xie B, Li S, Dai H, Ye K, et al. Joint semantic transfer network for IoT intrusion detection. IEEE Internet Things J. 2023;10(4):3368–83. doi:10.1109/JIOT.2022.3218339. [Google Scholar] [CrossRef]

15. Apruzzese G, Pajola L, Conti M. The cross-evaluation of machine learning-based network intrusion detection systems. IEEE Trans Netw Serv Manag. 2022;19(4):5152–69. doi:10.1109/TNSM.2022.3157344. [Google Scholar] [CrossRef]

16. Talpini J, Sartori F, Savi M. Enhancing trustworthiness in ML-based network intrusion detection with uncertainty quantification. J Reliab Intell Envron. 2024;10(4):501–20. doi:10.1007/s40860-024-00238-8. [Google Scholar] [CrossRef]

17. Guo C, Pleiss G, Sun Y, Weinberger KQ. On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML 2017); 2017 Aug 6–11; Sydney, NSW, Australia. p. 1321–30. [Google Scholar]

18. Calders T, Jaroszewicz S. Efficient AUC optimization for classification. In: Knowledge discovery in databases: PKDD 2007. Berlin/Heidelberg, Germany: Springer; 2007. p. 42–53. doi:10.1007/978-3-540-74976-9_8. [Google Scholar] [CrossRef]

19. Zhu D, Li G, Wang B, Wu X, Yang T. When AUC meets DRO: optimizing partial AUC for deep learning with non-convex convergence guarantee. In: Proceedings of the 39th International Conference on Machine Learning; 2022 Jul 17–23; Baltimore, MD, USA. p. 27548–73. [Google Scholar]

20. Reza S, Ferreira MC, Machado JJM, Tavares JMRS. A multi-head attention-based transformer model for traffic flow forecasting with a comparative analysis to recurrent neural networks. Expert Syst Appl. 2022;202(18):117275. doi:10.1016/j.eswa.2022.117275. [Google Scholar] [CrossRef]

21. Phelps N, Lizotte DJ, Woolford DG. Using Platt’s scaling for calibration after undersampling—limitations and how to address them. arXiv:2410.18144. 2024. [Google Scholar]

22. Namdar K, Haider MA, Khalvati F. A modified AUC for training convolutional neural networks: taking confidence into account. Front Artif Intell. 2021;4:582928. doi:10.3389/frai.2021.582928. [Google Scholar] [PubMed] [CrossRef]

23. Gunawan AEK, Wibowo A. Stock price movement classification using ensembled model of long short-term memory (LSTM) and random forest (RF). JOIV Int J Inf Vis. 2023;7(4):2255. doi:10.30630/joiv.7.4.01640. [Google Scholar] [CrossRef]

24. Zhu G, Li Y, Zhang S, Duan X, Huang Z, Yao Z, et al. Neural networks with linear adaptive batch normalization and swarm intelligence calibration for real-time gaze estimation on smartphones. Int J Intell Syst. 2024;2024(1):2644725. doi:10.1155/2024/2644725. [Google Scholar] [CrossRef]

25. Man J, Sun G. A residual learning-based network intrusion detection system. Secur Commun Netw. 2021;2021(11):5593435. doi:10.1155/2021/5593435. [Google Scholar] [CrossRef]

26. Marks RB. A multiline method of network analyzer calibration. IEEE Trans Microw Theory Techn. 1991;39(7):1205–15. doi:10.1109/22.85388. [Google Scholar] [CrossRef]