Boundary Measure Alignment via Optimal Transport for Temporal Action Detection

1  Introduction

Temporal Action Detection (TAD) aims to localize the start and end times of action instances in untrimmed videos and recognize their categories [1,2]. As a fundamental task in video understanding, TAD has broad applications in surveillance, human-computer interaction, sports analysis, and other real-world scenarios [3]. Unlike video-level action classification, TAD requires both semantic discrimination and accurate temporal boundary localization [4,5]. Although recent detectors have achieved strong performance through improved temporal modeling and Transformer-based architectures [6–9], accurate boundary localization remains challenging due to gradual action transitions, ambiguous visual changes, local noise in long videos, and annotation uncertainty [10–12]. Under strict temporal Intersection-over-Union (tIoU) thresholds, even small boundary errors can noticeably affect detection performance [12].

Existing boundary supervision commonly relies on point-wise classification, start/end confidence prediction, or local regression losses, such as cross-entropy, L1 distance, and IoU-based losses [13,14]. These objectives mainly compare predictions and targets at corresponding temporal positions. When predicted boundary responses are shifted from ground-truth boundaries, conventional local supervision does not explicitly encode how far the predicted response should move along the temporal axis [15,16].

To address this limitation, we propose Boundary Measure Alignment (BMA), a training-stage auxiliary objective for TAD. BMA represents predicted and annotated starts and ends as probability measures over the feature-level temporal domain. Predicted measures are constructed from the rising and falling edges of the foreground probability trajectory, while target measures are generated by Gaussian smoothing around annotated boundaries [1,2]. Entropic Optimal Transport (OT) is then used to align predicted and target measures, allowing the loss to explicitly model temporal boundary displacement [17,18]. Unlike OT-based assignment methods that use OT for candidate-instance matching, BMA applies OT to temporal boundary measures, where the transport cost directly reflects boundary displacement [15,19,20]. Fig. 1 illustrates the difference between conventional point-wise boundary supervision and the proposed measure-level boundary alignment.

Figure 1: Comparison between point-wise boundary supervision and measure-level alignment. Conventional methods rely on point-wise matching at local temporal positions, whereas the proposed method represents boundaries as distributions and performs global alignment through optimal transport, thereby providing more effective geometry-aware optimization signals under temporal misalignment.

BMA is used only during training and is removed during inference. Therefore, it keeps the detector architecture, model parameters, and post-processing pipeline unchanged at test time. We integrate BMA into AFSD, FCOS, and an ActionFormer-style detector. Experiments on THUMOS14 and ActivityNet-v1.3 show consistent average mAP improvements across these detectors, indicating that BMA provides useful auxiliary boundary supervision for detectors with temporal foreground or boundary response outputs.

The main contributions of this paper are summarized as follows:

(1) We introduce a boundary measure representation for TAD, where action starts and ends are separately represented as probability measures on the one-dimensional temporal domain.

(2) We propose an entropic OT-based boundary measure alignment loss that explicitly encodes temporal displacement between predicted and target boundary measures.

(3) We integrate BMA into AFSD, FCOS, and an ActionFormer-style detector. Experiments on THUMOS14 and ActivityNet-v1.3 show consistent improvements while keeping the original inference pipelines unchanged.

2  Related Work

2.1 Temporal Action Detection

Temporal Action Detection (TAD) aims to localize and classify action instances in untrimmed videos. Existing methods can be broadly grouped into proposal-based methods, anchor-free dense prediction methods, and Transformer-based detectors. Proposal-based methods first generate temporal candidates and then refine or classify them [5,21,22]. Anchor-free methods remove predefined temporal anchors and predict action categories, boundary scores, or localization offsets at dense temporal locations [23,24]. More recent Transformer-based detectors improve long-range temporal modeling through self-attention, multi-scale feature aggregation, or query-based decoding [5–8]. Although these detectors have improved temporal representation and detection accuracy, their boundary learning objectives are still commonly formulated as local classification, regression, or IoU-related losses. These local objectives provide useful supervision but do not explicitly encode the temporal displacement between shifted boundary responses and annotated boundaries.

2.2 Boundary Modeling in TAD

Accurate boundary modeling is critical for high-quality temporal localization, especially under strict tIoU thresholds. Existing methods enhance boundary localization by predicting frame-level foreground probabilities, start/end confidence curves, boundary offsets, salient boundary features, or IoU-based localization scores [3,14,25,26]. These designs make detectors more sensitive to action transitions and improve proposal refinement. However, most of them still supervise boundary responses at corresponding temporal positions. When a predicted boundary peak is shifted from the annotated boundary, point-wise losses can penalize the mismatch but do not directly describe how far the response should move along the temporal axis. This motivates us to represent action starts and ends as temporal probability measures and align them with a displacement-aware OT objective.

2.3 Distribution and Set-Based Learning

Distributional and set-based learning have been widely used to reduce the limitations of point-wise supervision. DETR-style methods formulate detection as set prediction and perform global matching between predictions and targets [16,27,28]. Other methods model localization uncertainty using discrete probability distributions over spatial coordinates [29]. These approaches show that distributional representations and global matching can provide richer supervision than independent point-wise losses. However, they mainly focus on spatial object detection, bounding-box uncertainty, or candidate-level assignment. They do not directly address the alignment of one-dimensional temporal boundary responses in TAD.

2.4 Optimal Transport in Deep Learning

Optimal Transport (OT) provides a principled framework for measuring distribution discrepancy with an explicit transport cost. It has been used for domain adaptation and feature alignment [30,31], generative modeling [32], correspondence estimation in dynamic visual scenes [33], and dynamic label assignment in object detection [15]. Entropic regularization and Sinkhorn iterations make OT differentiable and efficient for deep learning [34,35]. In detection tasks, OT is often used to match candidates with ground-truth instances. Different from these applications, BMA applies OT to predicted and target start/end boundary measures defined on the same one-dimensional temporal domain. Therefore, the transport cost has a direct temporal interpretation: it reflects how much boundary probability mass should be moved along the temporal axis. This distinction makes BMA different from general feature distribution alignment and OT-based candidate assignment.

3  Methodology

3.1 Overview

This section presents Boundary Measure Alignment (BMA), a training-stage auxiliary objective for temporal action detection. As shown in Fig. 2, BMA constructs predicted start/end measures from detector outputs and aligns them with target measures generated from annotated boundaries. The OT computation is used only during training and is removed during inference, so the original detector pipeline remains unchanged.

Figure 2: Overall pipeline of the proposed BMA module. Given an untrimmed video, temporal features are first extracted by the backbone network, and the temporal action detector generates action proposals and foreground probability trajectories. The BMA module constructs start and end boundary measures from the predicted trajectories and aligns them with target measures generated from ground-truth boundaries through entropy-regularized optimal transport. This module is used only as an auxiliary supervision branch during training and does not introduce additional optimal transport computation during inference.

BMA is not used for proposal assignment or detection-result matching. Instead, it measures the temporal displacement between predicted and target boundary measures defined on the same feature-level temporal domain. This makes the transport plan temporally interpretable and distinguishes BMA from general distribution matching or OT-based assignment. The overall training and inference pipeline of BMA is shown in Fig. 2.

3.2 Boundary Measure Representation

Given an untrimmed video, the detector extracts a temporal feature sequence of length T′. We define the feature-level temporal domain as

Ω={1,2,…,T′}(1)

For the k-th annotated action instance with original boundaries (sk,ek), the boundaries are mapped to the feature-level domain according to the feature stride, yielding (s^k,e^k). In implementation, the mapped positions are clipped to [1,T′] and rounded to the nearest feature index.

Let lt∈RC denote the classification logits at temporal location t. For a multi-label sigmoid head, the class-agnostic foreground probability is defined as

pt=maxc∈{1,…,C}σ(lt,c)(2)

For detectors with an explicit background category, it can also be defined as

pt=1−Pbg(t)(3)

The maximum operation is performed independently at each temporal location over the category dimension. The same foreground trajectory is used for start and end measure construction: positive temporal variations indicate potential starts, while negative variations indicate potential ends.

An action start usually corresponds to a rising edge of the foreground trajectory, while an action end corresponds to a falling edge. We compute the first-order temporal difference as

Δpt=pt−pt−1(4)

with Δp1=0. The predicted start and end responses are

rs(t)=ReLU (Δpt)(5)

re(t)=ReLU (−Δpt)(6)

The responses are normalized into probability measures:

μs(t)=rs(t)+ϵ∑τ=1T′(rs(τ)+ϵ)(7)

μe(t)=re(t)+ϵ∑τ=1T′(re(τ)+ϵ)(8)

Thus, μs and μe satisfy

∑t=1T′μs(t)=1(9)

∑t=1T′μe(t)=1(10)

For target measures, discrete annotated boundaries are converted into Gaussian-smoothed distributions. Given K action instances, the target start and end measures are

νs(t)=∑k=1Kexp⁡(−(t−s^k)22σb2)∑τ=1T′∑k=1Kexp⁡(−(τ−s^k)22σb2)(11)

νe(t)=∑k=1Kexp⁡(−(t−e^k)22σb2)∑τ=1T′∑k=1Kexp⁡(−(τ−e^k)22σb2)(12)

here, σb controls the smoothness of the target boundary measures. The negative sign in the exponential term ensures that each target measure peaks around the annotated boundary and decays with temporal distance.

The target measures aggregate all annotated boundaries in the video. Therefore, BMA provides class-agnostic global boundary regularization rather than instance-specific boundary matching. When multiple instances overlap or boundaries are densely located, a single foreground trajectory may produce entangled responses; this limitation is discussed in the qualitative analysis.

Fig. 3 illustrates the construction of predicted and target boundary measures.

Figure 3: Illustration of boundary measure construction. The foreground probability trajectory shows a rising trend at the action start and a falling trend at the action end. By extracting the positive and negative temporal derivatives, the start and end variation intensities are respectively characterized. After non-negative mapping and normalization, they are converted into probability measures defined over the temporal domain.

3.3 Geometry-Aware Alignment via Optimal Transport

When the predicted boundary response is shifted from the annotated boundary, point-wise losses mainly penalize values at corresponding temporal locations. If a predicted boundary peak lies around tp while the target peak lies around tg, such losses do not explicitly encode the displacement ∣tp−tg∣. OT provides a displacement-aware objective by transporting probability mass from the predicted boundary measure to the target measure.

For a predicted measure m and target measure m∗, we define the temporal cost matrix as Cij=∣ti−tj∣p. The entropic OT alignment loss is

ℒOT(m,m⋆)=minπ∈Π(m,m⋆)∑i,jπijCij+εOT∑i,jπij(log⁡πij−1)(13)

where π is the transport plan, and εOT controls the smoothness of the plan. When two measures are close to Dirac-like peaks at tp and tg, the transport cost is directly related to ∣tp−tg∣p. Thus, OT directly penalizes temporal displacement between boundary measures.

BMA does not directly optimize tIoU, which is computed from final detected segments after classification, regression, and post-processing. Instead, it reduces boundary-response displacement. Since strict tIoU thresholds are sensitive to start/end errors, this auxiliary supervision can indirectly improve boundary-sensitive detection. Fig. 4 visualizes the measure-level alignment process under temporal boundary offset.

Figure 4: Illustration of boundary measure alignment under support mismatch. When the predicted measure deviates from the target measure along the temporal axis, conventional local losses may struggle to provide effective optimization signals. In contrast, optimal transport explicitly models the temporal transport cost and moves probability mass from the predicted distribution toward the target distribution, thereby achieving geometry-aware boundary alignment.

3.4 Numerical Solution with Sinkhorn Iterations

We use Sinkhorn iterations to obtain a differentiable approximation of the entropic OT objective. Given the Gibbs kernel K=exp⁡(−C/εOT), the dual vectors are updated as

u(l+1)=a⊘(Kv(l))(14)

v(l+1)=b⊘(K⊤u(l+1))(15)

where a and b are the predicted and target measure vectors. After M iterations, the transport plan is estimated and ⟨π∗,C⟩ is used as the alignment cost. This approximation provides stable gradients while keeping training computationally feasible.

3.5 Joint Training Objective

BMA is added to the original detection objective as an auxiliary loss. The baseline detector loss is written as

ℒdet=ℒcls+λregℒreg+λauxℒaux(16)

where ℒcls, ℒreg, and ℒaux denote classification, regression, and detector-specific auxiliary losses. The BMA loss is

ℒBMA=ℒot(ms,ms∗)+ℒot(me,me∗)(17)

The final joint training objective is formulated as:

ℒ=ℒdet+λBMAℒBMA(18)

BMA does not replace the original classification, regression, or assignment objectives. It only regularizes global start/end boundary responses during training, while the original detector remains responsible for category prediction and final segment localization.

4  Experiments

We evaluate BMA on THUMOS14 and ActivityNet-v1.3. To examine detector compatibility, BMA is integrated into AFSD, FCOS, and an ActionFormer-style detector. For each detector, BMA is used only during training, while the original inference and post-processing pipeline remains unchanged. All experiments are conducted on a single NVIDIA RTX 3090 GPU.

4.1 Experimental Setup

4.1.1 Dataset

THUMOS14. THUMOS14 contains untrimmed videos from 20 action categories. Following the standard protocol, we train on the validation set and evaluate on the test set. Its dense action instances make it suitable for evaluating fine-grained temporal boundary localization.

ActivityNet-v1.3. ActivityNet-v1.3 is a large-scale benchmark with 200 action categories. We train on the training set and evaluate on the validation set. Its longer videos and diverse action categories are used to evaluate the generalization ability of the proposed BMA module.

4.1.2 Evaluation Metrics

We use mean Average Precision (mAP) as the main evaluation metric. For THUMOS14, we report mAP at tIoU thresholds of 0.3, 0.4, 0.5, 0.6, and 0.7, together with their average. For ActivityNet-v1.3, we follow the standard protocol and report mAP at tIoU thresholds from 0.5 to 0.95 with a step size of 0.05, as well as the average mAP.

4.1.3 Implementation Details

We use pre-extracted I3D features as input whenever the corresponding baseline adopts I3D features. The original losses of each detector are kept unchanged, and BMA is added as an auxiliary boundary measure alignment loss during training. During inference, the BMA branch and OT computation are disabled.

For detectors with multi-label sigmoid classification heads, such as FCOS and the ActionFormer-style detector, we construct the foreground probability trajectory by taking the maximum action probability over all classes at each temporal location. For detectors with explicit boundary confidence outputs, such as AFSD, the start and end confidence sequences are normalized as predicted boundary measures. Target measures are generated by Gaussian smoothing around annotated boundaries in the feature-level temporal domain. Unless otherwise specified, we set ϵ=10−6, σb=1.0, λBMA=0.05, εOT=0.05, use Cij=∣i−j∣, and perform 20 Sinkhorn iterations.

4.2 Main Results

The main comparison results are reported in Table 1.

The main comparison results are reported in Table 1. On THUMOS14, BMA consistently improves all three detectors. The average mAP of AFSD, FCOS, and the ActionFormer-style detector increases from 52.0%, 45.3%, and 66.8% to 53.3%, 47.1%, and 69.1%, respectively. The gains are more evident at stricter thresholds; for example, mAP@0.7 improves by 1.6%, 2.2%, and 2.9% for the three detectors.

On ActivityNet-v1.3, BMA also improves the average mAP of AFSD, FCOS, and the ActionFormer-style detector from 34.40%, 32.30%, and 35.60% to 34.83%, 32.82%, and 36.04%, respectively. The gains are smaller than on THUMOS14, and the ActionFormer-style detector does not improve at mAP@0.95. This suggests that BMA is affected by long video duration, coarser annotations, action-duration variation, and feature-level temporal resolution.

4.3 Cross-Detector and Cross-Dataset Analysis

Table 1 shows that BMA is not limited to the ActionFormer-style detector. It brings average mAP gains on AFSD, FCOS, and the ActionFormer-style detector under the same experimental setting. This indicates that BMA can provide auxiliary boundary supervision for detectors with temporal foreground or boundary response outputs.

The gains are larger on THUMOS14 than on ActivityNet-v1.3. THUMOS14 contains denser action instances, where reducing boundary displacement more directly improves tIoU. ActivityNet-v1.3 contains longer videos and coarser boundary annotations; therefore, extremely strict metrics such as mAP@0.95 are more sensitive to annotation ambiguity and feature-level discretization. Thus, BMA improves average detection performance across detectors, but its effect under extremely strict boundary matching on ActivityNet-v1.3 remains limited.

4.4 Boundary Localization Analysis

Since the objective of BMA is to improve temporal boundary alignment quality, we further evaluate its effect using boundary localization errors. For each ground-truth action instance, we select the prediction segment with the same category and the highest tIoU after non-maximum suppression as its matched prediction.

All boundary errors are computed in the feature-level temporal domain rather than on the raw video-frame index. In our THUMOS14 setting, I3D features are extracted using 16-frame clips at approximately 30 fps with a temporal stride of 4 frames. Therefore, one feature-level temporal step corresponds to 4 original video frames, i.e., approximately 0.133 s. Accordingly, an error value of e in Table 2 corresponds to 4e original frames, or 0.133e s.

Given the i-th matched prediction segment (s^i,e^i) and its corresponding ground-truth action segment (si,ei), the start boundary error and end boundary error are respectively defined as:

Es=1N∑i=1N∣s^i−si∣(19)

Ee=1N∑i=1N∣e^i−ei∣(20)

The average boundary error is defined as:

Eb=12(Es+Ee)(21)

As shown in Table 2, compared with the baseline model, introducing BMA reduces the start error from 1.3 to 0.9 feature steps and the end error from 1.1 to 0.7 feature steps. Since one feature step corresponds to 4 original frames, i.e., approximately 0.133 s, the average boundary error is reduced from 1.2 feature steps, or 0.160 s, to 0.8 feature steps, or 0.107 s. This corresponds to an average reduction of 0.4 feature steps, namely 1.6 original frames or approximately 0.053 s. These results indicate that the performance improvement comes not only from better action category discrimination, but also from more accurate temporal boundary localization. The reduction in boundary errors is consistent with the performance improvement at higher tIoU thresholds on THUMOS14.

4.5 Ablation Experiment

4.5.1 Start/End Measure Ablation

As shown in Table 3, aligning either start or end measures improves the baseline, and jointly aligning both achieves the best performance. This confirms that start and end boundaries provide complementary localization cues. Therefore, we use both start and end BMA in the final model.

Joint start-end alignment provides the most balanced boundary improvement, although a single-side alignment may occasionally yield comparable mAP due to metric variance.

4.5.2 Loss Function Comparison

Table 4 compares different boundary alignment losses.

Table 4 compares different boundary alignment losses. BCE/CE, L2, and KL losses improve the baseline, showing that boundary distribution supervision is useful. Entropic OT achieves the best mAP@0.7 and average mAP because it additionally models temporal transport cost between different boundary locations.

4.5.3 Robustness to Noisy Foreground Trajectories

Since BMA uses temporal differences of the foreground probability trajectory, we evaluate its robustness to noisy foreground responses. Gaussian perturbations are added to the trajectory used for BMA loss construction, where α controls the noise strength.

As shown in Table 5, BMA remains effective under small and moderate perturbations. When α=0.03, it still achieves 68.6% average mAP and 46.1% mAP@0.7, higher than the baseline results of 66.8% and 43.9%. With larger noise, performance decreases but remains above the baseline. This indicates that BMA tolerates moderate foreground fluctuations, while still relying on reasonably reliable foreground estimates.

4.5.4 Analysis on Dense and Overlapping Boundaries

We further evaluate the behavior of BMA under different boundary-density conditions. As shown in Table 6, BMA improves the average mAP from 68.2% to 70.8% on sparse-boundary videos and from 61.4% to 62.2% on dense-boundary videos, corresponding to absolute gains of 2.6 and 0.8 percentage points, respectively. At mAP@0.7, BMA improves sparse-boundary videos from 45.1% to 48.4%, with an absolute gain of 3.3 percentage points, while improving dense-boundary videos from 36.7% to 37.5%, with an absolute gain of 0.8 percentage points. These results indicate that BMA remains effective in dense-boundary videos, but adjacent or overlapping action boundaries may entangle foreground variations and weaken the effect of global boundary-measure alignment.

4.5.5 Fine-Grained Analysis under Different Action Durations

To further analyze the behavior of BMA under different temporal conditions, we divide action instances in THUMOS14 into short, medium, and long groups according to their temporal duration. Short actions usually have more sensitive boundary changes, while long actions are more likely to contain internal foreground fluctuations. Table 7 reports the results of the ActionFormer-style detector on different duration groups.

As shown in Table 7, BMA improves the baseline in all duration groups. For short, medium, and long actions, the average mAP increases from 63.5% to 66.4%, from 67.2% to 69.5%, and from 68.6% to 70.2%, corresponding to absolute gains of 2.9, 2.3, and 1.6 percentage points, respectively. At the stricter mAP@0.7 threshold, the gains are 3.3, 2.9, and 1.3 percentage points for short, medium, and long actions, respectively. These numerical results show that BMA is more beneficial for short and medium actions, where a small boundary shift can cause a larger tIoU change. For long actions, BMA still improves detection performance, but the gain is relatively smaller because internal foreground fluctuations may make boundary response construction less stable.

4.6 Hyperparameter Analysis

4.6.1 Effect of λBMA

The experimental results with different λBMA are shown in Table 8. A moderate BMA weight provides the best trade-off between boundary alignment and the original detection objective. Too small weights fail to provide sufficient geometric supervision, whereas overly large weights may disturb the optimization of the detector.

4.6.2 Effect of Gaussian Bandwidth σ

The experimental results with different σ are shown in Table 9. An appropriate Gaussian bandwidth is important for constructing target boundary measures. A small bandwidth makes the target distribution too sharp, while an overly large bandwidth weakens boundary localization by over-smoothing the supervision.

4.6.3 OT-Related Hyperparameter Analysis

We further analyze the influence of OT-related hyperparameters, including the number of Sinkhorn iterations, the entropic regularization coefficient, and the temporal cost function.

As shown in Table 10, 20 Sinkhorn iterations provide a good trade-off between accuracy and efficiency. The best performance is obtained with εOT=0.05 and the linear cost Cij=∣i−j∣, which are used as default settings.

4.7 Complexity and Efficiency Analysis

Since BMA is used only during training, it does not change inference-time parameters, FLOPs, or post-processing. As shown in Table 11, the BMA-enhanced detector has the same parameter count and FLOPs as the baseline, and the FPS remains nearly unchanged.

Although BMA introduces Sinkhorn iterations during training, the overhead is moderate. As shown in Table 12, the training time per epoch increases from 18.6 to 21.4 min, and GPU memory increases from 13.2 to 14.1 GB. This corresponds to a 15.1% training-time overhead, while the average mAP improves from 66.8% to 69.1%. Therefore, BMA improves boundary localization without increasing inference cost, with only moderate additional training cost.

4.8 Qualitative Analysis

Fig. 5 visualizes the effect of BMA on temporal boundary localization and training-stage measure alignment. Compared with the baseline, BMA produces segments closer to the annotated boundaries. The predicted boundary measures, target measures, and OT plans further show that BMA encourages predicted boundary mass to align with target boundary regions during training.

Figure 5: Qualitative illustration of boundary localization and training-stage measure alignment. Each example shows the ground-truth segment, baseline prediction, BMA-enhanced prediction, predicted start/end boundary measures, Gaussian-smoothed target measures, and the training-stage OT plan. Start and end measures are shown together for compact visualization. The gray curve schematically indicates the dominant transport path.

BMA may still fail when foreground responses contain strong internal fluctuations, when multiple actions are densely adjacent or overlapping, or when long-video annotations are coarse. These cases suggest that BMA is more suitable as a global boundary regularization term than as an instance-level boundary matching mechanism.

5  Conclusion

This paper proposes Boundary Measure Alignment (BMA), a training-stage auxiliary objective for temporal action detection. BMA represents predicted and annotated action starts and ends as temporal probability measures and aligns them with entropic optimal transport, thereby explicitly modeling boundary displacement. Since BMA is removed during inference, it does not change the original detector pipeline or introduce additional inference cost.

Experiments on THUMOS14 and ActivityNet-v1.3 show that BMA improves AFSD, FCOS, and an ActionFormer-style detector. The gains are more evident on THUMOS14 and under stricter tIoU thresholds, while improvements on ActivityNet-v1.3 are more moderate due to longer videos, coarser annotations, and larger action-duration variations. Robustness, ablation, efficiency, and qualitative analyses further verify the effectiveness and limitations of BMA. Future work will explore more robust foreground trajectory estimation, class-wise or instance-aware boundary measures, and broader adaptation to other TAD architectures.

Acknowledgement: Not applicable.

Funding Statement: The authors received no specific funding for this study.

Author Contributions: The authors confirm contribution to the paper as follows: Conceptualization, Tiyao Zhang; methodology, Tiyao Zhang; software, Tiyao Zhang; validation, Tiyao Zhang; formal analysis, Tiyao Zhang; investigation, Tiyao Zhang; resources, Xue Yuan; data curation, Tiyao Zhang; writing—original draft preparation, Tiyao Zhang; writing—review and editing, Xue Yuan; visualization, Tiyao Zhang; supervision, Xue Yuan; project administration, Tiyao Zhang; funding acquisition, Xue Yuan. All authors reviewed and approved the final version of the manuscript.

Availability of Data and Materials: Data available on request from the authors.

Ethics Approval: Not applicable.

Conflicts of Interest: The authors declare no conflicts of interest.

References

1. Zhao Y, Xiong Y, Wang L, Wu Z, Tang X, Lin D. Temporal action detection with structured segment networks. In: Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV); 2017 Oct 22–29; Venice, Italy. New York, NY, USA: IEEE; 2017. p. 2933–42. doi:10.1109/ICCV.2017.317. [Google Scholar] [CrossRef]

2. Lin T, Liu X, Li X, Ding E, Wen S. BMN: boundary-matching network for temporal action proposal generation. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV); 2019 Oct 27–Nov 2; Seoul, Repblic of Korea. New York, NY, USA: IEEE; 2019. p. 3888–97. doi:10.1109/iccv.2019.00399. [Google Scholar] [CrossRef]

3. Xu M, Zhao C, Rojas DS, Thabet A, Ghanem B. G-TAD: sub-graph localization for temporal action detection. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2020 Jun 13–19; Seattle, WA, USA. New York, NY, USA: IEEE; 2020. p. 10153–62. doi:10.1109/cvpr42600.2020.01017. [Google Scholar] [CrossRef]

4. Zhang CL, Wu J, Li Y. ActionFormer: localizing moments of actions with transformers. In: Computer Vision—ECCV 2022. ECCV 2022. Lecture Notes in Computer Science. Vol. 13664. Cham, Switzerland: Springer; 2022. p. 492–510. doi:10.1007/978-3-031-19772-7_29. [Google Scholar] [CrossRef]

5. Liu X, Wang Q, Hu Y, Tang X, Zhang S, Bai S, et al. End-to-end temporal action detection with transformer. IEEE Trans Image Process. 2022;31:5427–41. doi:10.1109/TIP.2022.3195321. [Google Scholar] [CrossRef]

6. Shi D, Zhong Y, Cao Q, Zhang J, Ma L, Li J, et al. ReAct: temporal action detection with relational queries. In: Computer Vision—ECCV 2022. Cham, Switzerland: Springer; 2022. p. 105–21. doi:10.1007/978-3-031-20080-9_7. [Google Scholar] [CrossRef]

7. Liberatori B, Conti A, Rota P, Wang Y, Ricci E. Test-time zero-shot temporal action localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2024 Jun 17–21; Seattle, WA, USA. New York, NY, USA: IEEE; 2024. p. 18720–29. [Google Scholar]

8. Liu Z, Ning J, Cao Y, Wei Y, Zhang Z, Lin S, et al. Video swin transformer. In: Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2022 Jun 18–24; New Orleans, LA, USA. New York, NY, USA: IEEE; 2022. p. 3192–201. doi:10.1109/CVPR52688.2022.00320. [Google Scholar] [CrossRef]

9. Gritsenko AA, Xiong X, Djolonga J, Dehghani M, Sun C, Lucic M, et al. End-to-end spatio-temporal action localisation with video transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2024 Jun 17–21; Seattle, WA, USA. New York, NY, USA: IEEE; 2024. p. 18373–83. [Google Scholar]

10. Qing Z, Su H, Gan W, Wang D, Wu W, Wang X, et al. Temporal context aggregation network for temporal action proposal refinement. In: Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2021 Jun 20–25; Nashville, TN, USA. New York, NY, USA: IEEE; 2021. p. 485–94. doi:10.1109/CVPR46437.2021.00055. [Google Scholar] [CrossRef]

11. Lu Z, Elhamifar E. FACT: frame-action cross-attention temporal modeling for efficient action segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2024 Jun 17–21; Seattle, WA, USA. New York, NY, USA: IEEE; 2024. p. 18175–85. [Google Scholar]

12. Alwassel H, Caba Heilbron F, Escorcia V, Ghanem B. Diagnosing error in temporal action detectors. In: Computer Vision—ECCV 2018. Cham, Switzerland: Springer; 2018. p. 264–80. doi:10.1007/978-3-030-01219-9_16. [Google Scholar] [CrossRef]

13. Rezatofighi H, Tsoi N, Gwak J, Sadeghian A, Reid I, Savarese S. Generalized intersection over union: a metric and a loss for bounding box regression. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2019 Jun 15–20; Long Beach, CA, USA. New York, NY, USA: IEEE; 2020. p. 658–66. doi:10.1109/CVPR.2019.00075. [Google Scholar] [CrossRef]

14. Zheng Z, Wang P, Liu W, Li J, Ye R, Ren D. Distance-IoU loss: faster and better learning for bounding box regression. Proc AAAI Conf Artif Intell. 2020;34(7):12993–3000. doi:10.1609/aaai.v34i07.6999. [Google Scholar] [CrossRef]

15. Ge Z, Liu S, Li Z, Yoshie O, Sun J. OTA: optimal transport assignment for object detection. In: Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2021 Jun 20–25; Nashville, TN, USA. New York, NY, USA: IEEE; 2021. p. 303–12. doi:10.1109/cvpr46437.2021.00037. [Google Scholar] [CrossRef]

16. Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S. End-to-end object detection with transformers. In: Proceedings of the Computer Vision—ECCV 2020. Cham, Switzerland: Springer International Publishing; 2020. p. 213–29. doi:10.1007/978-3-030-58452-8_13. [Google Scholar] [CrossRef]

17. Cuturi M. Sinkhorn distances: lightspeed computation of optimal transportation distances. arXiv:1306.0895. 2013. [Google Scholar]

18. Peyré G, Cuturi M. Computational optimal transport with applications to data sciences. Found Trends® Mach Learn. 2019;11(5–6):355–607. doi:10.1561/2200000073. [Google Scholar] [CrossRef]

19. Frogner C, Zhang C, Mobahi H, Araya M, Poggio TA. Learning with a Wasserstein loss. Adv Neural Inf Process Syst. 2018. [Google Scholar]

20. Torres LC, Pereira LM, Amini MH. A survey on optimal transport for machine learning: theory and applications. arXiv:2106.01963. 2021. [Google Scholar]

21. Tan J, Tang J, Wang L, Wu G. Relaxed transformer decoders for direct action proposal generation. In: Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 2021 Oct 10–17; Montreal, QC, Canada. New York, NY, USA: IEEE; 2022. p. 13506–15. doi:10.1109/ICCV48922.2021.01327. [Google Scholar] [CrossRef]

22. Song Y, Kim D, Cho M, Kwak S. Online temporal action localization with memory-augmented transformer. arXiv:2408.02957. 2024. [Google Scholar]

23. Lin C, Li J, Wang Y, Tai Y, Luo D, Cui Z, et al. Fast learning of temporal action proposal via dense boundary generator. Proc AAAI Conf Artif Intell. 2020;34(7):11499–506. doi:10.1609/aaai.v34i07.6815. [Google Scholar] [CrossRef]

24. Shi D, Zhong Y, Cao Q, Ma L, Lit J, Tao D. TriDet: temporal action detection with relative boundary modeling. In: Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2023 Jun 17–24; Vancouver, BC, Canada. New York, NY, USA: IEEE; 2023. p. 18857–66. doi:10.1109/CVPR52729.2023.01808. [Google Scholar] [CrossRef]

25. Zhao P, Xie L, Ju C, Zhang Y, Wang Y, Tian Q. Bottom-up temporal action localization with mutual regularization. In: Computer Vision—ECCV 2020. Cham, Switzerland: Springer; 2020. p. 539–55. doi:10.1007/978-3-030-58598-3_32. [Google Scholar] [CrossRef]

26. Yang L, Peng H, Zhang D, Fu J, Han J. Revisiting anchor mechanisms for temporal action localization. IEEE Trans Image Process. 2020;29:8535–48. doi:10.1109/TIP.2020.3016486. [Google Scholar] [CrossRef]

27. Zhu X, Su W, Lu L, Li B, Wang X, Dai J. Deformable DETR: deformable transformers for end-to-end object detection. arXiv:2010.04159. 2020. [Google Scholar]

28. Sun P, Zhang R, Jiang Y, Kong T, Xu C, Zhan W, et al. Sparse R-CNN: end-to-end object detection with learnable proposals. In: Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2021 Jun 20–25; Nashville, TN, USA. New York, NY, USA: IEEE; 2021. p. 14449–58. doi:10.1109/cvpr46437.2021.01422. [Google Scholar] [CrossRef]

29. Li X, Wang W, Wu L, Chen S, Hu X, Li J, et al. Generalized focal loss: learning qualified and distributed bounding boxes for dense object detection. arXiv:2006.04388. 2020. [Google Scholar]

30. Courty N, Flamary R, Tuia D, Rakotomamonjy A. Optimal transport for domain adaptation. IEEE Trans Pattern Anal Mach Intell. 2017;39(9):1853–65. doi:10.1109/TPAMI.2016.2615921. [Google Scholar] [CrossRef]

31. Damodaran BB, Kellenberger B, Flamary R, Tuia D, Courty N. DeepJDOT: deep joint distribution optimal transport for unsupervised domain adaptation. In: Proceedings of the Computer Vision—ECCV 2018. Cham, Switzerland: Springer; 2018. p. 467–83. doi:10.1007/978-3-030-01225-0_28. [Google Scholar] [CrossRef]

32. Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning; 2017 Aug 6–11; Sydney, Australia. p. 214–23. [Google Scholar]

33. Puy G, Boulch A, Marlet R. FLOT: scene flow on point clouds guided by optimal transport. arXiv:2007.11142. 2020. [Google Scholar]

34. Kolouri S, Park SR, Thorpe M, Slepcev D, Rohde GK. Optimal mass transport: signal processing and machine-learning applications. IEEE Signal Process Mag. 2017;34(4):43–59. doi:10.1109/msp.2017.2695801. [Google Scholar] [CrossRef]

35. Villani C. Optimal transport: old and new. Berlin/Heidelberg, Germany: Springer; 2009. 976 p. [Google Scholar]

36. Lin T, Zhao X, Su H, Wang C, Yang M. BSN: boundary sensitive network for temporal action proposal generation. In: Proceedings of the European Conference on Computer Vision (ECCV); 2018 Sep 8–14; Munich, Germany. Berlin/Heidelberg, Germany: Springer; 2018. p. 3–19. [Google Scholar]

37. Liu Q, Wang Z. Progressive boundary refinement network for temporal action detection. Proc AAAI Conf Artif Intell. 2020;34(7):11612–9. doi:10.1609/aaai.v34i07.6829. [Google Scholar] [CrossRef]

38. Zhao C, Thabet AK, Ghanem B. Video self-stitching graph network for temporal action localization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV); 2021 Oct 10–17; Montreal, QC, Canada. New York, NY, USA: IEEE; 2021. p. 13658–67. [Google Scholar]

39. Wang Q, Zhang Y, Zheng Y, Pan P. RCL: recurrent continuous localization for temporal action detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2022 Jun 18–24; New Orleans, LA, USA. New York, NY, USA: IEEE; 2022. p. 13566–75. [Google Scholar]

40. Nag S, Zhu X, Song YZ, Xiang T. Proposal-free temporal action detection via global segmentation mask learning. In: Proceedings of the European Conference on Computer Vision (ECCV); 2022 Jun 18–24; New Orleans, LA, USA. New York, NY, USA: IEEE; 2022. p. 645–62. [Google Scholar]

41. Cheng F, Bertasius G. TallFormer: temporal action localization with a long-memory transformer. In: Proceedings of the European Conference on Computer Vision (ECCV); 2022 Jun 18–24; New Orleans, LA, USA. New York, NY, USA: IEEE; 2022. p. 503–21. [Google Scholar]

42. Lin C, Xu C, Luo D, Wang Y, Tai Y, Wang C, et al. Learning salient boundary feature for anchor-free temporal action localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2021 Oct 10–17; Montreal, QC, Canada. New York, NY, USA: IEEE; 2021. p. 3320–9. [Google Scholar]

43. Tian Z, Shen C, Chen H, He T. FCOS: fully convolutional one-stage object detection. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV); 2019 October 27–Nov 2; Seoul, Republic of Korea. p. 9627–36. [Google Scholar]