MALT-Drive: Moment-Aligned Language-to-Trajectory Planning for Autonomous Driving

1  Introduction

End-to-end autonomous driving is moving from modular pipelines toward unified multimodal systems, where perception, prediction, and planning are jointly modeled and optimized within a differentiable framework [1–4]. Recent vision-language-action (VLA) models further extend this paradigm by introducing natural language into driving policy learning. Beyond sparse route commands and geometric waypoints, language enables driving systems to express scene constraints and high-level behavioral decisions more effectively [5–8].

Compared with representative driving VLA methods, MALT-Drive focuses on the explicit coupling between generated control-instruction states and trajectory-producing planning states. Table 1 summarizes this positioning across language role, representation granularity, planning-state binding, and supervision mechanism. DriveLM organizes driving reasoning through graph-structured VQA [5], DriveMLM models decision states and explanations [9], SimLingo strengthens language-action association through alignment and Action Dreaming [10], ORION conditions trajectory generation on planning tokens [11], and AutoVLA generates reasoning and physical action tokens autoregressively [8]. MALT-Drive uses the generated control instruction as an intermediate planning variable and aligns its control-instruction token states with branch-specific planning tokens for path geometry, speed regulation, and executability estimation.

Throughout this paper, control instruction denotes the short, action-bearing textual output associated with a driving-oriented VQA prompt. The teacher-forcing target is termed the reference control instruction, and the autoregressive model output is termed the generated control instruction. A canonical control-instruction class is one of the 30 entries in the controlled instruction library.

The planning value of language supervision depends on whether generated control instructions can constrain downstream motion generation in a behaviorally consistent manner. Driving instructions usually affect different factors of future motion [8,10,11]. Maneuver-oriented instructions, such as lane changing and obstacle bypassing, mainly determine future path geometry. Modulation-oriented instructions, such as slowing down, stopping, and waiting, mainly affect the speed profile and executability state. Compressing these factors into a single planning representation can entangle path semantics, speed semantics, and feasibility-related semantics within the same latent space, weakening the correspondence between language states and executable actions.

Based on this observation, this paper proposes MALT-Drive, a moment-aligned VQA-supervised language-to-trajectory planning framework for end-to-end autonomous driving. Given front-view camera inputs, ego-motion history, route priors, and a driving-oriented VQA prompt, the VLA backbone first autoregressively generates a concise control instruction within a controlled instruction space. The generated control-instruction states are then reused as control-related intermediate representations for trajectory decoding. MALT-Drive further introduces a dual-branch action decoder. The Maneuver Branch predicts future path geometry, while the Modulation Branch predicts the speed profile and executability score. This design separates path-level maneuver planning from speed-executability modulation, allowing generated control instructions to constrain the corresponding planning factors through branch-specific channels.

Moment alignment is introduced to support this branch-wise coupling. Token-level alignment emphasizes local token correspondences, global embedding alignment compresses control-instruction and planning states into one vector, and standard contrastive alignment is usually applied at the sample-embedding level. In MALT-Drive, the first-order moment summarizes dominant control semantics, while the grouped second-order sketch preserves token-relation patterns within the control-instruction and planning token sets. The resulting descriptor aligns maneuver-related control-instruction states with path-generation tokens and modulation-related control-instruction states with speed/executability tokens.

At the supervision level, standard imitation learning usually provides only one expert trajectory for each visual scene, making it difficult to observe how the planner responds to language variations. MALT-Drive constructs same-scene counterfactual control-instruction rollouts. For the same scene, multiple prompt–control-instruction pairs cover executable maneuver instructions, executable modulation instructions, and non-executable instructions that should be rejected. Each reference or counterfactual control instruction is associated with corresponding path, speed, and executability targets. This supervision explicitly reveals behavior differences induced by language variations and supports the evaluation of instruction-action consistency and unsafe-instruction rejection.

The main contributions of this paper are summarized as follows:

•   This paper proposes MALT-Drive, a VQA-supervised language-to-trajectory planning framework that treats generated control instructions within a controlled instruction space as control-related intermediate variables and explicitly binds their token states to branch-specific path, speed, and executability planning states.

•   A dual-branch planning decoder with branch-level moment alignment is designed to separately model path-level maneuvers and speed-executability modulation, and to align branch-specific planning states with the corresponding instruction semantics through compact moment descriptors.

•   Same-scene counterfactual control-instruction rollouts are introduced to provide instruction-sensitive language-action supervision and enable direct evaluation of executable action generation and unsafe-instruction rejection.

The rest of this paper is organized as follows. Section 2 reviews related work. Section 3 introduces the proposed MALT-Drive framework, including dual-branch planning, moment alignment, and counterfactual control-instruction rollouts. Section 4 reports experimental results and ablation studies. Section 5 concludes the paper.

2  Related Work

2.1 End-to-End Autonomous Driving

End-to-end autonomous driving seeks to learn planning and control policies directly from sensor observations within a unified differentiable framework. Representative methods integrate perception, prediction, and planning through joint optimization, which reduces error accumulation commonly observed in modular pipelines. TransFuser fuses multi-view sensory features for navigation-oriented planning [1], while UniAD and VAD organize perception, prediction, and planning into unified task structures [2,3]. Recent generative approaches, such as GenAD, further improve behavior modeling by formulating planning as a generative prediction problem [4]. Even with this progress, most end-to-end planners are primarily supervised by expert trajectories paired with visual scenes and route priors. Such supervision is effective for imitation but provides limited evidence that a planner can respond consistently to different high-level behavioral intents under the same scene. In scenes where multiple feasible behaviors exist, a single expert trajectory may encourage visual shortcut learning rather than intent-sensitive planning. By contrast, this paper studies how generated control instructions can be consistently associated with the latent planning states that produce path geometry, speed profiles, and executability estimates.

2.2 Language-Grounded Driving

Language has become an increasingly common component of autonomous driving to support scene understanding, explanation, instruction following, and action generation. DriveLM demonstrates the value of language supervision for driving-oriented reasoning [5]. LMDrive and EMMA condition driving policies on natural-language instructions [6,7]. Recent driving VLA models, including DriveMLM, ORION, AutoVLA, Reasoning-VLA, and OpenREAD, further explore action tokens, autoregressive generation, and unified reasoning-action modeling [8,9,11]. These studies respectively advance graph VQA, decision-state explanation, language-action learning, planning-token conditioning, and autoregressive action-token generation. MALT-Drive studies a complementary interface: branch-wise coupling between generated-control-instruction states and trajectory-producing states. Rather than introducing another global language-action objective, it uses the generated control instruction as a controlled intermediate planning variable whose control-instruction token states are aligned with path, speed, and executability planning tokens.

2.3 Instruction Consistency

Long-tail driving scenarios often call for decisions beyond ordinary trajectory imitation, such as yielding, emergency braking, obstacle bypassing, and rejecting infeasible actions. Language-action paired supervision has therefore been explored to improve behavioral consistency. SimLingo jointly optimizes closed-loop driving, vision-language understanding, and language-action alignment [10], while SteerVLA emphasizes the role of language in long-tail reasoning and high-level behavior selection [12]. Recent post-training strategies in AutoVLA and DynVLA also suggest that supervised and reinforcement fine-tuning can improve the alignment between generated language, actions, and driving rewards [8]. Still, language-action data alone does not guarantee instruction-sensitive behavior. When each scene is paired with only one instruction and one trajectory, a model may still rely on visual priors and ignore instruction changes. A more discriminative supervision signal should keep the visual context fixed while varying the instruction and the corresponding motion target. Motivated by this point, this paper introduces counterfactual control-instruction rollouts under the same scene, in which executable and non-executable instructions are paired with path, speed, and executability targets. This setting makes instruction-action consistency more identifiable and enables direct evaluation of instruction compliance and unsafe-instruction rejection.

2.4 Multimodal Alignment

Multimodal alignment has been widely studied in vision-language learning. CLIP establishes a global contrastive learning paradigm for image-text alignment [13], while EAU and ProLIP improve robustness through uncertainty-aware or probabilistic matching [14,15]. In parallel, compact high-order descriptors such as DeepBDC and moment-based representations show that second-order statistics can preserve feature interactions that are often lost by simple mean pooling [16,17]. Common alignment choices operate at different granularities: token-level alignment emphasizes local token correspondences, global embedding alignment compresses all instruction and planning states into one vector, and standard contrastive alignment usually matches sample-level embeddings. MALT-Drive targets the language-to-trajectory interface by constructing branch-level moment descriptors for control-instruction token states and planning-token states. The first-order component summarizes dominant control semantics, and the grouped second-order component preserves token-relation patterns within each branch. This provides a compact descriptor for aligning maneuver-related language states with path-generation states and modulation-related language states with speed/executability states.

3  Method

3.1 Problem Formulation

This work studies VQA-supervised control-instruction-conditioned trajectory planning for end-to-end autonomous driving. At decision time t, the planner receives a scene context

𝒳t={It−H+1:tfront,st−H+1:t,gt},(1)

where It−H+1:tfront denotes the front-camera image history over the most recent H frames, st−H+1:t denotes the ego-state history, and gt denotes the route prior. Instead of assuming an externally provided language command, the policy receives a driving-oriented VQA prompt qt, generates a control instruction, and decodes trajectory-level actions:

c^t∼pθ(ct∣𝒳t,qt),(P^t,u^t,e^t)=πθ(𝒳t,qt,c^t),(2)

where P^t∈RTp×2 is the predicted future path in the ego coordinate system, u^t∈RTu is the predicted speed sequence, and e^t∈[0,1] estimates whether the generated control instruction can be safely executed in the current traffic scene. During supervised training, the autoregressive language decoder is teacher-forced with the reference control instruction ct⋆; during inference, it consumes its own generated control-instruction tokens. In this setting, language generation is placed before action decoding and serves as an explicit intermediate control interface.

The controlled instruction space is implemented as a finite action-grounding library with 30 canonical control-instruction classes, including 10 maneuver classes, 8 modulation classes, 6 mixed maneuver-modulation classes, and 6 non-executable classes. The mixed classes are formal action-bearing classes rather than exceptions to the branch design. They are used when an instruction jointly changes path geometry and longitudinal behavior, such as lane changing with yielding, merging with deceleration, or bypassing with low speed. Thus, the dual-branch decoder decomposes action factors rather than forcing each generated control instruction into a single branch. Each class is expressed as a short natural-language sentence, such as “follow the route”, “change to the left lane” or “stop and wait”. Each class is linked to path geometry, speed profile, executability supervision, or conservative fallback behavior, and the complete library is listed in Table 2. The controlled instruction space defines the operational scope of MALT-Drive. It is designed to cover the action-bearing behaviors evaluated in Bench2Drive, including route following, turning, lane changing, merging, bypassing, speed regulation, yielding, stopping, waiting, and unsafe request rejection. During inference, a generated control instruction is used for planning only when it exactly matches or can be normalized to one canonical control-instruction class. If the generated control instruction is ambiguous or unsupported by the library, it is treated as non-executable and routed to the conservative fallback behavior before the decoded path and speed are sent to the control layer.

Different instructions constrain different factors of future motion. Maneuver-oriented instructions mainly determine path geometry, such as lane changing or bypassing, whereas modulation-oriented instructions mainly determine longitudinal behavior, such as slowing down, stopping, or accelerating. Based on this observation, MALT-Drive formulates generated-control-instruction planning as a branch-wise latent alignment problem. The Maneuver Branch generates path geometry, and the Modulation Branch predicts speed and estimates executability. Both branches are conditioned on the same multimodal context and aligned with generated control-instruction states through branch-specific planning tokens and moment descriptors.

The framework consists of three stages. First, a shared backbone fuses multimodal tokens from visual observations, ego-state history, route priors, VQA prompts, and generated control-instruction tokens. Second, two sets of planning queries extract maneuver and modulation tokens from the fused context. Third, branch-wise action heads and moment-alignment losses jointly supervise path prediction, speed regulation, and executability estimation. Fig. 1 illustrates the planner architecture, including the VLA backbone, branch-specific planning tokens, moment descriptors, action heads, and alignment losses.

Figure 1: Planner architecture of MALT-Drive. Front images, ego-state/route-prior tokens, and the VQA prompt are encoded by the LoRA-adapted VLA backbone. Self-attention exposes generated-control-instruction states to the Moment-Aligned Planner, where maneuver, control-instruction, and modulation tokens are projected into branch-specific latent spaces. The Maneuver Branch aligns maneuver and control-instruction moments for path prediction, while the Modulation Branch aligns modulation and control-instruction moments for speed prediction and executability estimation. Solid arrows denote forward/inference flow, the snowflake symbol marks frozen parameters, the flame symbol marks trainable LoRA parameters, and bidirectional pale arrows denote alignment losses.

3.2 Branch-Specific Planning

The dual-branch decoder follows a hierarchical view of human driving decisions. In a typical driving process, the driver first determines a tactical path-level behavior, such as keeping the lane, changing lanes, merging, or bypassing an obstacle. The driver then regulates longitudinal motion through speed keeping, deceleration, braking, stopping, waiting, or resuming. A feasibility judgment is also involved in this process to prevent unsafe or traffic-rule-violating actions from being executed. MALT-Drive maps this decision hierarchy into the decoder design. The Maneuver Branch corresponds to tactical path selection and predicts future path geometry, while the Modulation Branch corresponds to longitudinal regulation and executability estimation.

Mixed instructions activate both branches through the same generated control-instruction states. For example, “change to the left lane and slow down to yield” induces a lateral path update in the Maneuver Branch and a reduced speed/yielding profile in the Modulation Branch.

The backbone uses a token-level multimodal fusion paradigm. The visual encoder maps the images into visual tokens Vt∈RNv×d, while the ego-state history and route prior are embedded as structured state tokens St∈RNs×d. The VQA prompt is represented by prompt tokens Qt∈RNq×d, and the control instruction is represented by contextualized control-instruction token states Lt∈RNl×d. These token groups are concatenated and fed into a multimodal Transformer [5,10].

Ht=fθ([Vt;St;Qt;Lt])∈RNh×d(3)

where [⋅;⋅] denotes token concatenation and d is the hidden dimension.

Rather than decoding all future motion from a single pooled context vector, MALT-Drive introduces two sets of learnable planning queries, Uman∈RKman×d and Umod∈RKmod×d. They attend to the fused context and produce branch-specific planning tokens:

Mt=Dman(Uman,Ht)∈RKman×d,Rt=Dmod(Umod,Ht)∈RKmod×d(4)

where Dman and Dmod are lightweight Transformer query decoders with independent parameters. The Maneuver Branch tokens Mt are used for path prediction, whereas the Modulation Branch tokens Rt are used for speed prediction and executability estimation. In practice, Kman=Tp and Kmod=Tu, so that token indices correspond to the prediction horizon. This design allows the two branches to extract different types of control evidence from the same generated control instruction and scene context: maneuver tokens focus on route geometry and lateral deviation, while modulation tokens focus on speed, temporal behavior, and safety feasibility.

Maneuver branch. The Maneuver Branch predicts local path offsets in ego coordinates. For each future step τ∈{1,…,Tp}, the corresponding maneuver token is mapped to a displacement vector, and the final trajectory is obtained by cumulative integration.

Δp^t,τ=WpMt,τ,p^t,τ=∑j=1τΔp^t,j,P^t=[p^t,1,…,p^t,Tp]⊤(5)

where Wp∈R2×d. This local-offset formulation preserves path continuity and makes instruction-induced geometric changes easier to represent. For example, a generated lane-change or obstacle-bypass instruction can affect a coherent sequence of maneuver tokens rather than weakly perturbing a single global trajectory embedding.

Modulation branch. The Modulation Branch predicts the longitudinal speed curve and the executability score. Each modulation token is mapped to a non-negative speed value, while the pooled branch state estimates whether the instruction is safely executable:

u^t,τ=softplus(wu⊤Rt,τ),e^t=sigmoid(we⊤mean(Rt)).(6)

The predicted speed sequence u^t describes longitudinal behaviors such as maintaining speed, braking, stopping, waiting, or resuming motion. The executability score e^t provides an explicit rejection signal that helps reduce the risk of unsafe or infeasible generated control instructions being directly converted into vehicle motion.

The executability head is attached to the Modulation Branch because unsafe-instruction rejection is mainly determined by temporal motion feasibility. Red-light compliance, stop-line holding, gap acceptance, car-following, and pedestrian yielding are all reflected in the predicted speed evolution and waiting behavior. The pooled modulation state therefore provides a compact summary for deciding whether the generated control instruction can be safely executed. A separate classifier would introduce an additional feasibility representation that is less directly tied to the decoded speed profile. The rollout checker is used to construct executability labels for counterfactual supervision, while the learned sigmoid head provides a lightweight inference-time rejection signal.

3.3 Branch-Wise Moment Alignment

The dual-branch decoder separates maneuver and modulation states, but this separation alone does not define how control-instruction token states should be coupled with trajectory-producing planning states. MALT-Drive therefore introduces branch-wise moment alignment at the token-set descriptor level. Token-level alignment emphasizes local token correspondences, global embedding alignment compresses instruction and planning states into one vector, and standard contrastive alignment is usually applied at the sample-embedding level. MALT-Drive instead computes branch-level descriptors for instruction-token states and planning-token states, so the Maneuver Branch is aligned with path-generation tokens and the Modulation Branch is aligned with speed/executability tokens.

For branch b∈{man,mod}, the projected language and planning tokens are defined as

Ltb=ΠLb(Lt),Atb={ΠAman(Mt),b=man,ΠAmod(Rt),b=mod.(7)

where ΠLb and ΠAb are learnable projection layers that map language and planning tokens into the same branch-specific latent space. They are applied token-wise and output d-dimensional branch tokens, so Ltb∈RNl×d and Atb∈RKb×d, where Kb=Kman for b=man and Kb=Kmod for b=mod. Thus, Ltman and Atman form the language-planning pair for path geometry, while Ltmod and Atmod form the language-planning pair for speed regulation and executability. The following moment descriptor is computed on these paired token sets rather than on a single global pooled feature.

Moment descriptor. For each branch b, the descriptor operator is applied to either the projected language tokens Ltb or the projected planning tokens Atb. Thus, X∈{Ltb,Atb}, where n=Nl for control-instruction token states and n=Kb for branch planning tokens. Let 1∈Rn denote an all-one column vector. For a token matrix X∈Rn×d, both first-order and grouped second-order statistics are computed. The first-order component is

μ(X)=1n1⊤X,X=[X(1),…,X(G)],X(g)∈Rn×dg(8)

where dg=d/G and G is the number of channel groups, with d divisible by G. For each group, the mean is removed and a covariance-like relation sketch is computed, assuming n>1:

X¯(g)=X(g)−1μ(X(g)),C(X(g))=1n−1(X¯(g))⊤X¯(g).(9)

Since C(X(g)) is symmetric, only its upper-triangular entries are retained. The grouped second-order descriptor is then written as

ρ(X)=Wρconcatg=1Gvec△(C(X(g)))(10)

where vec△(⋅) denotes upper-triangular vectorization and Wρ projects the concatenated sketch into a compact descriptor space. The final moment descriptor combines the two components:

z(X)=Norm(Wz[μ(X);ρ(X)])(11)

where Norm(⋅) denotes ℓ2 normalization, and z(⋅) is the moment-descriptor operator that maps a projected token set to a unit-norm vector in a compact alignment space. In this descriptor, the first-order term summarizes the dominant control semantics of a branch-specific token set, while the grouped second-order term preserves token-relation patterns that are often lost by simple pooling. This descriptor-level design is suitable for the language-to-trajectory interface considered here because it aligns branch-level token sets rather than isolated token pairs or a single global state.

Computationally, the grouped second-order sketch avoids forming a full d×d covariance matrix. For a token set X∈Rn×d, mean pooling costs O(nd). The grouped covariance term computes G matrices of size dg×dg, where dg=d/G, giving a second-order cost of O(Gndg2)=O(nd2/G). The number of retained upper-triangular entries is Gdg(dg+1)/2. Thus, increasing G reduces the quadratic group dimension, while very small dg may weaken relation modeling. The default setting uses G=8 to balance latency and alignment quality.

The moment descriptor gives branch-specific language-planning alignment a concrete action meaning. For example, for the instruction “change to the left lane”, the Maneuver Branch links the “left lane” cue to path tokens that generate lateral displacement, while the Modulation Branch maintains the nominal speed when no braking or yielding cue appears. The first-order moment captures the dominant control semantics, and the grouped second-order sketch preserves relations between instruction cues and action-producing planning tokens.

Alignment objective. In the mini-batch objective, the decision-time subscript t is omitted and i,j index training tuples. For a mini-batch of size B, the positive pair in branch b is the matched language-planning descriptor pair from the same training tuple, i.e., (z(Lib),z(Aib)). Mismatched pairs (z(Lib),z(Ajb)) with j≠i and (z(Lkb),z(Aib)) with k≠i serve as sample-level in-batch negatives. This contrastive definition of negative pairs is independent of the executability label. The similarity between the language descriptor of sample i and the planning descriptor of sample j in branch b is

sijb=z(Lib)Tz(Ajb).(12)

Since both descriptors are ℓ2-normalized, sijb is equivalent to cosine similarity. The diagonal entries siib correspond to positives, whereas the off-diagonal entries are used as negatives in the corresponding row-wise or column-wise softmax. Based on this similarity matrix, the language-to-planning and planning-to-language matching probabilities are

pAj∣Lib=exp⁡(sijb/τ)∑k=1Bexp⁡(sikb/τ),pLi∣Ajb=exp⁡(sijb/τ)∑k=1Bexp⁡(skjb/τ)(13)

where τ is the temperature. The branch-wise alignment loss is defined as

ℒalignb=−12B∑i=1B[log⁡pAi∣Lib+log⁡pLi∣Aib](14)

This symmetric form encourages each generated control instruction to retrieve its corresponding branch-specific planning state and, conversely, encourages each planning state to remain identifiable in the language space. In the maneuver branch, this constraint directly couples instruction semantics with the path-producing tokens Mt; in the modulation branch, it couples instruction semantics with the speed- and executability-producing tokens Rt.

To reduce collapse between the two branches, a weak branch-separation regularizer is further introduced. For compact notation, let zL,ib=z(Lib) and zA,ib=z(Aib):

ℒsep=1B∑i=1B[cos2⁡(zL,iman,zL,imod)+cos2⁡(zA,iman,zA,imod)](15)

The separation term is applied as a weak regularizer. It encourages the maneuver and modulation descriptors to specialize in different control evidence while keeping both branches conditioned on the same generated control-instruction tokens. Mixed instructions are represented by paired labels across the two branches. For example, “change to the left lane and slow down to yield” provides a left-lane-change target for the Maneuver Branch and a deceleration/yield target for the Modulation Branch. The regularizer separates branch channels; it does not suppress shared instruction conditioning.

3.4 Counterfactual Control-Instruction Supervision

Imitation learning usually provides only one expert trajectory for a scene, making it difficult to determine whether the model follows language instructions in the intended way. To make this dependency observable, MALT-Drive constructs counterfactual control-instruction rollouts. The symbol ct⋆(m) denotes the reference control instruction used as the teacher-forcing target during training, whereas c^t denotes the generated control instruction autoregressively produced by the model during inference. For each scene context 𝒳t, a VQA prompt–control-instruction set is constructed:

𝒟tvqa={(qt(m),ct⋆(m))}m=1Ntvqa(16)

where Ntvqa denotes the number of VQA prompt–control-instruction pairs attached to the same scene. Each control instruction is associated with a path target Pt⋆(m)∈RTp×2, a speed target ut⋆(m)∈RTu, and an executability label et⋆(m)∈{0,1}. The reference control instruction is paired with the expert path and speed, whereas additional counterfactual control instructions are generated according to local topology and interaction-target constraints. For all counterfactual tuples, the scene context 𝒳t remains fixed; only the prompt–control-instruction pair and its corresponding motion-executability target vary.

Fig. 2 summarizes the counterfactual rollout mechanism. It shows how the same scene and VQA prompt are paired with reference, alternative executable, and non-executable candidate control instructions, how rollout evaluation assigns executable/non-executable labels, and how the training objective backpropagates through the trainable planner components while remaining distinct from the data-forward flow used to generate candidate supervision.

Figure 2: Counterfactual control-instruction rollout and training supervision. Given the same scene and VQA prompt, the VQA planner instantiates reference, alternative executable, and non-executable candidate control instructions. Candidate path-speed rollouts are checked for trajectory feasibility, collision/clearance, drivable-area containment, and traffic-rule consistency, producing executability labels for the training target. The executability head is supervised by these labels, and the dashed blue path indicates the training-time backpropagation flow, separated from the solid data-forward flow.

VQA library: Table 2 lists the complete canonical control-instruction library used in MALT-Drive. The library contains 30 canonical classes. Maneuver classes mainly change future path geometry. Modulation classes mainly change the longitudinal speed profile or waiting behavior. Mixed classes jointly affect path and speed targets. Non-executable classes describe unsafe or rule-violating requests and are supervised with a conservative fallback target and an executability label of zero. The coverage of the library is checked using Bench2Drive metadata and expert trajectories, including route direction, lane availability, traffic-light state, obstacle layout, leading-object distance, conflict zones, and drivable-area constraints. These cues activate canonical classes such as lane changing, stopping and waiting, obstacle bypassing, or lane borrowing. During inference, generated control instructions that match or can be normalized to a canonical class are passed to the executability check, whereas ambiguous or unsupported generated control instructions are treated as out of scope and routed directly to the conservative fallback behavior.

Counterfactual rollout: For each scene state, one reference control instruction consistent with the expert behavior is retained, and additional maneuver, modulation, and non-executable variants are instantiated. Maneuver counterfactuals are generated according to local topology and obstacle conditions. A lane-change instruction is activated when the adjacent lane is reachable and the transition corridor is free. An obstacle-bypass or lane-borrowing instruction is activated when a parked vehicle, construction area, or blocked lane affects the nominal route. Modulation counterfactuals are generated according to longitudinal constraints. Speed keeping or acceleration is used when the front gap, speed limit, and route curvature allow forward motion. Deceleration, stopping, or waiting is used near red lights, stop lines, crossing pedestrians, slow leading vehicles, or yield-required gaps. Non-executable counterfactuals are generated for unsafe-instruction rejection, including red-light running, collision-seeking motion, blocked-lane traversal, and non-drivable-area entry. For these samples, the control-instruction text is preserved, the executability label is set to et⋆(m)=0, and the motion target is replaced by a conservative fallback trajectory.

A requested instruction is first interpreted by its intended maneuver and longitudinal effect, and the candidate rollout is then checked against the current scene constraints. If the intended action violates traffic rules, route topology, drivable-area boundaries, or short-horizon collision constraints, the instruction is labeled as non-executable. For example, an instruction such as “overtake the front vehicle” is rejected at a red light when the rollout would cross the stop line or enter the conflict area before passage is allowed; the fallback target stops or waits before the stop line. In these non-executable tuples, the original control-instruction text is retained, et⋆(m)=0, and the path/speed target is replaced by the conservative fallback target.

All counterfactual tuples are constructed using the same deterministic procedure summarized in Algorithm 1. For each scene, the reference control instruction is retained, and additional canonical control instructions are activated according to lane topology, obstacle layout, longitudinal constraints, and traffic-rule states. Maneuver classes modify the route-consistent path, modulation classes modify the speed profile, and mixed classes modify both targets. Each candidate is then evaluated through a 4.0s rollout with a 0.5s interval. A candidate that satisfies all safety and topology checks receives its candidate path-speed target and an executability label of one. A failed candidate retains its requested control-instruction text but is assigned the conservative fallback target and an executability label of zero. The same rules and thresholds are used for all counterfactual samples.

The fallback target follows the valid route-aligned lane and applies a monotonically decreasing speed profile that stops the ego vehicle at the nearest safe stopping position within the prediction horizon.

3.5 Training and Inference

All losses below are written for one training tuple and then averaged over the mini-batch. To keep notation compact, the counterfactual index m is omitted unless necessary. The VQA-based control-instruction generator is trained with the autoregressive negative log-likelihood. For a reference control instruction ct⋆={wt,ℓ⋆}ℓ=1Nc, where wt,ℓ⋆ denotes the ℓ-th control-instruction token,

ℒvqa=−1Nc∑ℓ=1Nclog⁡pθ(wt,ℓ⋆∣wt,<ℓ⋆,𝒳t,qt)(17)

During SFT, the control-instruction token states used by the planning decoder are obtained from teacher-forced reference control instructions, while during inference they are obtained from the generated control instruction. This train-inference difference is an exposure-bias source for the language-to-trajectory interface: an incorrect or unstable generated control instruction can shift the control-instruction token states received by the path, speed, and executability heads. Conditioned on the control-instruction token states, the Maneuver Branch is supervised by path regression, the Modulation Branch is supervised by speed regression, and the executability head is supervised by binary cross-entropy:

ℒpath=1Tp∑τ=1Tpℓsl1(p^t,τ,pt,τ⋆),ℒspeed=1Tu∑τ=1Tuℓsl1(u^t,τ,ut,τ⋆),ℒexe=−et⋆log⁡e^t−(1−et⋆)log⁡(1−e^t)(18)

where ℓsl1 denotes the smooth-ℓ1 loss. For non-executable tuples, Pt⋆ and ut⋆ correspond to the fallback target, while et⋆=0 trains the executability head to reject the requested instruction. The full supervised objective is

ℒSFT=λcℒvqa+λpℒpath+λuℒspeed+λeℒexe+λa(ℒalignman+ℒalignmod)+λsℒsep(19)

This objective is applied to both nominal and counterfactual tuples; therefore, multiple samples may share the same scene but have different control instructions and different motion-executability targets. Training begins by warm-starting the VQA generator and planning heads using nominal expert prompt–control-instruction pairs. Counterfactual control-instruction rollouts are then introduced so that the same scene is associated with multiple instruction-conditioned outcomes. Finally, non-executable samples are upweighted to improve executability calibration and reduce unsafe-instruction acceptance. Throughout these SFT stages, the alignment losses use the matched language-planning descriptors from the same tuple as positives, including counterfactual tuples derived from the same scene.

After supervised fine-tuning, the policy can be further refined by reinforcement fine-tuning [18]. For each scenario prompt, a group of Gr candidate outputs oi=(c^t,i,P^t,i,u^t,i,e^t,i) is sampled from the current policy. Each output receives a scalar reward and a group-relative advantage:

ri=rdrive,i+ηalignralign,i+ηfmtrfmt,i−ηunsaferunsafe,i,Ai=ri−mean({rj}j=1Gr)std({rj}j=1Gr)+ϵstd(20)

where rdrive summarizes route progress, collision avoidance, comfort, and traffic-rule compliance; ralign measures consistency between the generated control instruction and decoded action; rfmt preserves the canonical control-instruction format; and runsafe penalizes acceptance of non-executable instructions. The KL-regularized optimization objective is

𝒥RFT(θ)=1Gr∑i=1Grmin(ρiAi,clip(ρi,1−ϵ,1+ϵ)Ai)−βDKL(πθ‖πSFT)(21)

where

ρi=exp⁡[log⁡πθ(oi∣𝒳t,qt)−log⁡πθold(oi∣𝒳t,qt)](22)

πθ(oi∣𝒳t,qt) denotes the policy probability or density of the sampled output, and πSFT is the frozen supervised reference policy. The KL term prevents the refined policy from drifting away from the controlled instruction space and the branch-wise alignment learned during supervised fine-tuning. During RFT, the optimization uses sampled outputs from the current policy, including the generated control instruction, path sequence, speed sequence, and executability score. This exposes the planner to inference-style control-instruction token states after SFT and serves as a partial mitigation for the teacher-forcing mismatch. The remaining gap is measured by the control-instruction-source diagnostic in Section 4.4.

During inference, the model first autoregressively generates the control instruction and then decodes the path, speed sequence, and executability score from the generated control-instruction states. No teacher forcing is used at inference:

c^t∼pθ(ct∣𝒳t,qt),(P^t,u^t,e^t)=πθ(𝒳t,qt,c^t)(23)

where γ is the executability threshold used to convert the predicted executability score into an execute-or-reject decision. The canonical-class check is applied before the final control command is issued. If a generated control instruction cannot be exactly matched or normalized to the 30-class library, it is marked as unsupported and routed to the conservative fallback branch. For supported classes, the predicted executability score then determines whether the decoded motion can be executed. If e^t≥γ, the planner executes the generated-control-instruction-conditioned path and speed profile. If e^t<γ, the decoded pair (P^t,u^t) is replaced by the fallback pair (Ptfb,utfb), while the generated control instruction is treated as rejected rather than executed. This two-stage rule preserves sensitivity to supported valid instructions while reducing the likelihood that unsupported or unsafe generated control instructions directly influence the control loop.

4  Experiments

4.1 Experimental Settings

Dataset. Closed-loop driving performance is evaluated on Bench2Drive, a CARLA-based benchmark designed to assess multiple abilities in end-to-end autonomous driving covering dense urban interaction, adverse weather, and long-tail traffic events [19]. Following the official protocol, each method is evaluated on the complete set of 220 routes. To further evaluate cross-benchmark generalization beyond the CARLA-based setting, an additional experiment is conducted on the NAVSIM benchmark [20]. NAVSIM is built from real-world driving logs from nuScenes and provides a different planning evaluation setting with different traffic distributions, sensor characteristics, and behavior patterns. Following the official NAVSIM protocol, the reported metrics include no at-fault collision (NC), drivable-area compliance (DAC), ego progress (EP), time-to-collision margin (TTC), comfort (Comf.), and the Planning Driving Metric Score (PDMS).

VQA Generation. The VQA supervision follows the driving-oriented VQA design of DriveLM [5]. Using simulator metadata, route priors, and expert trajectories [21], this QA taxonomy is instantiated on Bench2Drive anchor frames while keeping the prompts short and planning-oriented. To make the language-action correspondence easier to identify, counterfactual examples are further constructed from the Dreamer data in SimLingo [10]. For each scene, the nominal VQA prompt–control-instruction pair is replaced with alternative control instructions that specify a different maneuver or speed regulation, the corresponding candidate path and speed targets are generated, and an executability label is assigned through rollout-based checks for collisions, drivable-area violations, and traffic-rule compliance.

Evaluation Metrics. MALT-Drive is evaluated along three axes: Closed-loop Driving, VQA Generation, and Instruction-Action Consistency.

Closed-loop Driving: Closed-loop driving performance follows the official Bench2Drive protocol. The reported metrics include Driving Score (DS), Success Rate (SR), and the success rates of five ability groups: Merging, Overtaking, Emergency Brake, Give Way, and Traffic Sign [19]. DS summarizes route completion under infraction penalties, while SR measures the percentage of successfully completed routes over the 220-route closed-loop benchmark. The ability-group success rates further evaluate scenario-specific driving competence under interactive and rule-constrained traffic conditions.

VQA Generation: The autoregressive VQA-based control-instruction generator is evaluated with GPT Score and SPICE, following the driving VQA evaluation protocols used in DriveLM and SimLingo [5,10]. GPT Score evaluates semantic correctness by providing a language-model evaluator with the question, the reference control instruction, and the generated control instruction, and asking it to assign a score from 0 to 100. SPICE parses the generated and reference control instructions into semantic proposition graphs and computes the tuple-level F-score [22]. These metrics assess whether the generated control instruction preserves the intended driving semantics.

Instruction-Action Consistency: Instruction-Action Consistency F1 (IAC-F1) is reported as a trajectory-grounded instruction-action consistency metric that evaluates whether the planned motion realizes the intended control semantics of each reference control instruction. Following decision-planning consistency evaluation in VLM-based driving [23], the reference control instruction and executability annotation define the instruction-side label, while the predicted path, speed, and executability score define the action-side label through deterministic kinematic rules. Concretely, signed lateral displacement assigns maneuver labels, speed trend and terminal speed assign speed-regulation labels, and e^<γ assigns the non-executable label. Let 𝒴=𝒴man∪𝒴spd∪𝒴exe denote the structured label space for maneuver, speed-regulation, and executability classes. For each class y∈𝒴, precision and recall are computed as

Precy=TPyTPy+FPy,Recy=TPyTPy+FNy,(24)

where TPy denotes matched instruction-side and action-side assignments to class y, FPy denotes action-side assignments to class y without the corresponding instruction-side label, and FNy denotes instruction-side assignments to class y not realized by the action-side label. IAC-F1 is the macro-F1 over all structured control classes:

IAC-F1=1|𝒴|∑y∈𝒴2PrecyRecyPrecy+Recy.(25)

To make the metric implementation explicit, fixed kinematic thresholds are used for action-label assignment. Let Δy^Tp denote the signed terminal lateral displacement of the predicted path, Δu^=u^Tu−u^1 denote the speed change over the prediction horizon, and u^Tu denote the terminal speed. The maneuver label is assigned as left-change if Δy^Tp>δy, right-change if Δy^Tp<−δy, and route-following otherwise. The speed-regulation label is assigned as stopping/waiting if u^Tu<vstop, acceleration if Δu^>δu, deceleration if Δu^<−δu, and speed-keeping otherwise. The non-executable label is assigned when e^<γ. In all experiments, δy=0.6m, δu=0.5m/s, vstop=0.3m/s, and γ=0.5 are fixed after validation and kept unchanged for all evaluated variants.

Implementation Details. The input consists of front-camera RGB images, ego-state history, an assigned global route, and a driving-oriented VQA prompt. Four frames are sampled at 2Hz from the Bench2Drive annotations. Each front image is resized to 1024×512 and split into two 512×512 horizontal tiles. Each tile is then resized to 448×448 and encoded by the InternVL2 visual encoder. The ego-state vector contains velocity, acceleration, and heading. The route prior is obtained from the assigned CARLA global route, transformed into the ego coordinate system, and projected into route tokens through a two-layer MLP.

VLA Backbone: The VLA backbone is instantiated as InternVL2-1B from the Mini-InternVL family [24], which consists of InternViT-300M-448px as the visual encoder and Qwen2-0.5B-Instruct as the language model. The language model is adapted with LoRA inserted into all linear layers, with rank r=32, scaling factor α=64, and dropout 0.1. The model first autoregressively generates a control instruction constrained by the canonical control-instruction library. The hidden states of the generated control-instruction tokens are then reused by the dual-branch action decoder, allowing the generated control instruction to serve as the intermediate control interface for trajectory decoding.

Action Targets: The action representation follows a disentangled path-and-speed design. The Maneuver Branch predicts Tp=8 geometric path points in the ego coordinate system. The Modulation Branch predicts Tu=8 speed targets at future time offsets {0.5,1.0,…,4.0}s, obtained from the expert trajectory and speed annotations. Both branches use 8 learnable planning queries, and the grouped moment descriptor uses G=8 channel groups.

Supervised Fine-Tuning: SFT contains three stages and lasts 12 epochs. In the first stage, the model is trained on nominal expert VQA-action tuples for 3 epochs. In the second stage, it is trained on a balanced 1:1 mixture of nominal and counterfactual VQA tuples for 7 epochs. In the third stage, non-executable VQA tuples are upweighted to 40% of each mini-batch for 2 epochs to improve unsafe-instruction rejection. Training uses AdamW with weight decay 0.1, one-cycle cosine scheduling, and a linear warm-up over the first 4% of iterations. The learning rate is 3×10−5 for backbone adaptation parameters and 2×10−4 for newly initialized modules. Mixed precision and gradient clipping at 1.0 are used. The supervised loss weights are fixed as λc=1.0, λp=1.0, λu=0.5, λe=0.5, λa=0.2, and λs=0.05. Models are trained on 8 NVIDIA A100-80GB GPUs with a global batch size of 64. The default 3/7/2 schedule and the 40% non-executable ratio are used in all experiments.

Reinforcement Fine-Tuning: After SFT, reinforcement fine-tuning is performed for one epoch from the best SFT checkpoint, with the SFT model retained as the frozen KL reference policy. For each VQA prompt, the policy samples grouped candidate outputs consisting of a generated control instruction, a path sequence, a speed sequence, and an executability score. The reward combines route progress, collision avoidance, traffic-rule compliance, comfort, control-instruction-format validity, instruction-action consistency, and unsafe-acceptance penalty. The clipped policy objective uses ϵ=0.2 and KL coefficient β=0.02. The executability threshold is fixed to γ=0.5 after validation and used unchanged in all closed-loop and counterfactual VQA evaluations.

4.2 Main Results

The Bench2Drive comparison is organized according to the sensing and planning formulation of the evaluated methods. The non-privileged baselines include conventional end-to-end planners, such as TCP/TCP-traj, UniAD, and VAD [2,3,25–27]; recent generation or policy-refinement planners, including DriveTransformer and DiffAD [28–33]; and vision-language-action planners, including Driveπ0, DriveMoE, and SimLingo [8,10,11,34–36]. All non-privileged entries are evaluated under the 220-route Bench2Drive closed-loop protocol without map-level privileged planning input. Think2Drive and PDM-Lite are listed separately as privileged references because they use map-level or expert-planner information unavailable to sensor-only closed-loop planners [21,37].

The results in Table 3 provide benchmark-level context under the reported settings. The evaluated methods differ in sensor inputs, VLM/LLM backbones, action representations, data construction procedures, and training recipes. These results consequently do not support direct component-level attribution across methods. To make the comparison transparent, the table reports the backbone, input configuration, training/action supervision, and language supervision of each method.

Table 3 reports the closed-loop performance and fine-grained ability scores on Bench2Drive. Among the listed methods, SimLingo provides the closest reference because both methods use InternVL2-1B and one-camera input on Bench2Drive. Under their respective data construction and training pipelines, SimLingo reports DS/SR values of 85.07/67.27, while MALT-Drive records 86.20/70.91.

AutoVLA and ORION provide additional reference points for post-trained and planning-token-based VLA planners. AutoVLA uses a Qwen2.5-VL-3B backbone and physical action tokens, while ORION uses six-camera input, Vicuna-7B, and B2D+Chat-B2D supervision. Their results are included to show the performance range of current VLA driving methods under different configurations.

At the ability level, MALT-Drive records a mean score of 68.84%, with scores of 58.75% for Merging and 93.33% for Emergency Brake. The model records 50.00% for Give Way and 82.11% for Traffic Sign, where no clear advantage over SimLingo is observed. These results indicate that short-gap yielding and fine-grained traffic-sign grounding remain challenging under the current setting. In Give Way, the remaining errors are related to short-gap yielding: the generated control instruction can indicate a reasonable yielding intent, while the Modulation Branch may still produce an overly conservative speed profile or miss a short executable gap. For Traffic Sign, the failure diagnosis suggests that the remaining errors are mainly associated with partially occluded traffic signs and perception-route ambiguity, where a sign or priority cue may be assigned to an adjacent lane or side road.

Table 4 provides additional reference points on the NAVSIM benchmark. MALT-Drive records a PDMS of 91.7 with camera-only input. It also records an ego-progress score of 89.8 while maintaining competitive safety and comfort metrics. These results provide cross-benchmark evidence under the reported method-specific settings. Differences from other methods should not be interpreted as controlled component-level improvements.

4.3 Ablation Studies

All controlled ablation and sensitivity experiments use the same backbone, sensor input, prediction horizon, and evaluation protocol. The main component, descriptor, auxiliary-objective, hyperparameter, and SFT-protocol results are reported as mean ± standard deviation over three runs. These repeated runs evaluate trend stability within the tested configurations and ranges.

Counterfactual Supervision, Branch Alignment, and RFT: Table 5 directly evaluates the three main training components. Removing both counterfactual supervision and branch-wise alignment reduces DS and IAC-F1 to 83.06±0.05 and 67.28±0.10, while the full model reaches 86.17±0.06 and 79.22±0.07. With counterfactual supervision but without alignment, GPT and SPICE increase to 79.15±0.10 and 74.08±0.09, while IAC-F1 remains at 68.72±0.11. This result shows that improved instruction generation alone does not ensure action consistency. The SFT-only full model reaches 78.51±0.06 IAC-F1, and RFT further increases it to 79.22±0.07. Table 6 also shows improvements from 69.2/68.5/66.9 to 80.4/79.6/78.8 for lane changing, stopping/waiting, and obstacle avoidance. These controlled trends support the role of counterfactual supervision in exposing instruction-dependent outcomes and the role of branch-wise alignment in connecting instruction semantics with path and speed/executability states.

Moment Descriptor: Table 7 shows that the combined μ+ρ descriptor reaches 86.17 ± 0.06 DS and 79.22 ± 0.07 IAC-F1, compared with 85.36 ± 0.06/76.88 ± 0.09 for μ alone and 85.03 ± 0.07/77.62 ± 0.08 for ρ alone. The results support the complementary contributions of first-order control summaries and second-order token-relation information. In Table 8, IAC-F1 remains between 78.2 and 79.2 for G∈{2,4,8,16,32}. The selected G=8 setting gives the highest DS and IAC-F1 with a 3.6% latency increase over mean pooling. This result supports local stability within the evaluated grouping range.

Auxiliary Objectives: Table 9 evaluates the two auxiliary objectives after fixing the full descriptor and branch-wise alignment. Removing ℒexe reduces DS/SR to 84.72/68.48 and IAC-F1 to 75.98, corresponding to drops of 1.45 DS, 2.43 SR, and 3.24 IAC-F1 from the full model. This shows that explicit executability supervision is important for matching non-executable instructions with conservative fallback behavior. Removing ℒsep produces a smaller but consistent drop, with DS/SR of 85.39/69.55 and IAC-F1 of 77.86, supporting its role as a regularizer that prevents maneuver and modulation representations from collapsing into a single shared channel.

Hyperparameter Sensitivity: Table 10 shows that moderate changes around the default configuration preserve similar performance. Across the tested nonzero alignment weights, grouping settings, descriptor dimensions, thresholds, and nonzero counterfactual ratios, DS remains between 85.24 and 85.97, and IAC-F1 remains between 77.16 and 78.93. Setting λa=0 causes the largest IAC-F1 reduction, directly supporting the language-planning alignment objective. Increasing the executability threshold reduces unsafe acceptance while lowering SR, and increasing the counterfactual ratio produces a similar safety-progress trade-off. These results support local stability around the selected operating point within the evaluated ranges.

SFT Protocol Sensitivity: Table 11 keeps the architecture, supervised losses, evaluation setting, and 12-epoch budget fixed. Removing the late non-executable phase increases unsafe acceptance from 4.1±0.2% to 6.1±0.3%. Increasing the non-executable ratio or extending this phase further reduces unsafe acceptance, with small reductions in DS or SR. Moving the non-executable phase before balanced counterfactual training also lowers IAC-F1. These controlled results support the selected 3/7/2 schedule with a 40% non-executable ratio as a balance between driving performance and unsafe-instruction rejection.

Executability-Threshold Analysis: Finally, Table 12 reports the validation-set calibration used to select the executability threshold γ, which is then fixed for all reported experiments. Executable instructions are treated as the positive class: precision measures the reliability of accepted instructions, recall measures the retention of executable instructions, and unsafe-acceptance rate measures how often unsafe instructions are incorrectly accepted. Lower thresholds retain more executable instructions but accept more unsafe ones, whereas higher thresholds reduce unsafe acceptance at the cost of recall. The selected γ=0.50 gives a balanced operating point, with 86.8% precision, 86.1% recall, and a 3.7% unsafe-acceptance rate.

4.4 Analysis and Discussion

Instruction-Conditioned Decoding. Table 13 examines the effect of control-instruction token sources on trajectory decoding. The reference-control-instruction setting feeds the decoder with the ground-truth reference control instruction ct⋆ and acts as an upper-bound diagnostic for control-instruction token quality. The generated-control-instruction setting corresponds to normal inference. On all routes, generated control instructions reduce DS from 87.05 to 86.20 and IAC-F1 from 81.5 to 79.2, giving a gap of 0.85 DS and 2.3 IAC-F1. This confirms that the teacher-forcing mismatch introduces a measurable degradation.

The complex subset is further evaluated. The gap between reference and generated control instructions increases to 1.33 DS and 3.5 IAC-F1, indicating that the gap becomes larger when generated control instructions need to encode finer interaction, traffic-rule, and conflict-avoidance semantics. Removing control-instruction tokens causes a larger drop, and randomly mismatching control-instruction tokens reduces IAC-F1 to 44.9. These results show that generated control-instruction states remain useful for planning, while exposure bias remains a limitation under complex or shifted traffic distributions.

Controlled-Space Coverage. The generated-control-instruction coverage of the controlled instruction space is further quantified on the validation counterfactual set. A generated control instruction is counted as covered if it exactly matches or can be normalized to one of the 30 canonical control-instruction classes. Otherwise, it is counted as unsupported or ambiguous and is handled by the fallback rule. On all validation routes, 3.8% of generated control instructions are unsupported or ambiguous. On the complex subset, this rate increases to 6.4%. This trend is consistent with Table 13, where generated control instructions show a larger gap on complex scenes. The fallback behavior is also related to the executability-threshold analysis in Table 12, where the selected threshold gives 86.8% precision, 86.1% recall, and a 3.7% unsafe-acceptance rate. This diagnostic is limited to the current controlled setting and does not claim robustness to unrestricted open-ended language.

Sensitivity of IAC-F1. The sensitivity of IAC-F1 to the thresholds used for action-label assignment is further evaluated. Table 14 varies one threshold at a time while keeping the others at their default values. The results show that IAC-F1 remains stable under moderate threshold changes. For example, when the lateral-displacement threshold varies from 0.4 to 0.8 m, IAC-F1 changes only from 79.0 to 78.6. When the speed-change threshold varies from 0.3 to 0.7 m/s, IAC-F1 remains within 78.8–79.2. Similar stability is observed for the stop-speed and executability thresholds. These results indicate that the main conclusion of instruction-action consistency is not sensitive to a specific threshold choice.

Failure-Source Diagnosis. Failures are further diagnosed according to where the error appears in the language-to-trajectory pipeline. A language-generation error means that the generated control instruction gives an incorrect driving intent, such as describing a partially occluded traffic-sign scene as a keep-speed case. A trajectory-planning error means that the generated control instruction is reasonable, but the decoded path and speed profile do not realize it. Table 15 shows that most remaining failures are planning-side errors. For example, in Give Way cases, the model often generates a correct yielding instruction, but the Modulation Branch may miss a short executable gap.

Qualitative Visualization. Fig. 3 shows representative instruction-conditioned planning cases. In the right-turn, red-light, and car-following scenes, the generated control instructions closely match the reference control intent while the decoded trajectories follow the corresponding maneuver and speed requirements.

Figure 3: Qualitative visualization of MALT-Drive on Bench2Drive. (a) Right turn with gradual deceleration. (b) Lane keeping and stopping at a red light. (c) Lane keeping and steady-speed car following. In each panel, the generated control instruction is compared with the canonical control instruction and the decoded trajectory.

Fig. 4 adds two diagnostic cases for instruction rejection and safety fallback. In the green-light intersection case, the generated control instruction asks the ego vehicle to accelerate and turn left, while straight-through traffic occupies the conflict area. The decoded trajectory waits near the current lane, showing that the modulation and executability states route the output to a conservative fallback. In the stationary-front-vehicle case, the generated control instruction asks the ego vehicle to overtake, while a left-front oncoming vehicle makes the lateral maneuver unsafe. The decoded trajectory remains in lane and rejects the non-executable instruction. These cases make the counterfactual control-instruction supervision visible at inference time and complement the executability-threshold analysis in Table 12.

Figure 4: Diagnostic visualization of counterfactual and safety-fallback cases. (a) Rejected acceleration-and-left-turn instruction: straight-through traffic occupies the conflict area, so the fallback trajectory waits in the current lane. (b) Rejected overtaking instruction: an oncoming vehicle blocks the lateral maneuver, so the fallback trajectory remains behind the stationary vehicle.

Runtime Reference. Table 16 reports closed-loop performance with source-reported latency for recent VLA-based driving methods. The latency values are collected from different papers and hardware settings, so they are used only as runtime context. The latency of MALT-Drive is measured on a single NVIDIA A100 80GB GPU with batch size 1, using the same input resolution, four-frame setting, VQA generation procedure, and 8-step path-speed prediction horizon as in the Bench2Drive evaluation. This measurement reports the practical runtime of the MALT-Drive implementation under a fixed setting.

Mixed-Instruction Diagnosis. Because the controlled instruction library includes coupled lateral-longitudinal instructions, mixed maneuver-modulation cases are further diagnosed. The mixed subset contains instructions whose targets jointly affect path geometry and speed or yielding behavior. Table 17 shows that mixed instructions benefit from weak branch specialization. Removing ℒsep weakens both maneuver and modulation F1, while an overly strong separation weight reduces mixed-instruction consistency. The default weak setting gives the best mixed IAC-F1, indicating that the regularizer supports channel specialization without harming coupled lateral-longitudinal instructions.

5  Conclusion

This paper has presented MALT-Drive, a VQA-supervised language-to-trajectory planning framework for autonomous driving. The core idea is to treat the generated control instruction as an intermediate control variable and explicitly align its token states with the latent planning states that produce motion. By separating maneuver-related and modulation-related planning factors, the framework provides a structured way to connect the generated control instruction with path geometry, speed regulation, and executability estimation.

Several limitations remain. The current formulation relies on a controlled instruction space and is evaluated mainly in a simulator-based closed-loop benchmark. Unsupported or ambiguous generated control instructions are routed to a conservative fallback behavior, but this mechanism only reduces the risk of directly executing out-of-library instructions and does not establish robustness to unrestricted language. In addition, part of the counterfactual supervision is instantiated from SimLingo Dreamer data, which introduces a data-construction dependency when comparing with SimLingo on Bench2Drive. Repeated-run results cover controlled ablation and sensitivity settings but do not establish formal statistical significance or robustness under unrestricted hyperparameter, language, or domain shifts. Robustness to open-ended language, richer traffic interactions, independent counterfactual sources, route-disjoint evaluation, and real-world sensor noise requires further study.

Acknowledgement: Not applicable.

Funding Statement: This work has been supported by the National Natural Science Foundation of China (Grants 52225212 and 52472433), the Jiangsu Province Frontier Technology Research and Development Plan (BF2025082).

Author Contributions: The authors confirm contribution to the paper as follows: study conception and design: Ziheng Lu, Yingfeng Cai, Hai Wang and Long Chen; data collection and experiment implementation: Ziheng Lu and Wei Dong; analysis and interpretation of results: Ziheng Lu, Wei Dong and Yingfeng Cai; draft manuscript preparation: Ziheng Lu; supervision and manuscript revision: Yingfeng Cai, Hai Wang and Long Chen. All authors reviewed and approved the final version of the manuscript.

Availability of Data and Materials: The Bench2Drive and NAVSIM benchmark data used in this study are publicly available from their official repositories.

Ethics Approval: Not applicable.

Conflicts of Interest: The authors declare no conflicts of interest.

References

1. Chitta K, Prakash A, Jaeger B, Yu Z, Renz K, Geiger A. TransFuser: imitation with transformer-based sensor fusion for autonomous driving. IEEE Trans Pattern Anal Mach Intell. 2023;45(11):12878–95. doi:10.1109/tpami.2022.3200245. [Google Scholar] [PubMed] [CrossRef]

2. Hu Y, Yang J, Chen L, Li K, Sima C, Zhu X, et al. Planning-oriented autonomous driving. In: Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2023 Jun 17–24; Vancouver, BC, Canada. p. 17853–62. doi:10.1109/cvpr52729.2023.01712. [Google Scholar] [CrossRef]

3. Jiang B, Chen S, Xu Q, Liao B, Chen J, Zhou H, et al. VAD: vectorized scene representation for efficient autonomous driving. In: Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); 2023 Oct 1–6; Paris, France. p. 8306–16. doi:10.1109/iccv51070.2023.00766. [Google Scholar] [CrossRef]

4. Zheng W, Song R, Guo X, Zhang C, Chen L. GenAD: generative end-to-end autonomous driving. In: Computer vision—ECCV 2024. Cham, Switzerland: Springer Nature; 2025. p. 87–104. doi:10.1007/978-3-031-73650-6_6. [Google Scholar] [CrossRef]

5. Sima C, Renz K, Chitta K, Chen L, Zhang H, Xie C, et al. DriveLM: driving with graph visual question answering. In: Computer vision—ECCV 2024. Cham, Switzerland: Springer Nature; 2025. p. 256–74. doi:10.1007/978-3-031-72943-0_15. [Google Scholar] [CrossRef]

6. Shao H, Hu Y, Wang L, Song G, Waslander SL, Liu Y, et al. LMDrive: closed-loop end-to-end driving with large language models. In: Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2024 Jun 16–22; Seattle, WA, USA. p. 15120–30. doi:10.1109/cvpr52733.2024.01432. [Google Scholar] [CrossRef]

7. Hwang JJ, Xu R, Lin H, Hung WC, Ji J, Choi K, et al. EMMA: end-to-end multimodal model for autonomous driving. Transactions on machine learning research. 2025 [cited 2026 Jan 1]. Available from: https://openreview.net/forum?id=kH3t5lmOU8. [Google Scholar]

8. Zhou Z, Cai T, Zhao S, Zhang Y, Huang Z, Zhou B, et al. AutoVLA: a vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning. Adv Neural Inf Process Syst. 2026;38:27920–56. [Google Scholar]

9. Cui E, Wang W, Li Z, Xie J, Zou H, Deng H, et al. DriveMLM: aligning multi-modal large language models with behavioral planning states for autonomous driving. Visual Intell. 2025;3(1):22. doi:10.1007/s44267-025-00095-w. [Google Scholar] [CrossRef]

10. Renz K, Chen L, Arani E, Sinavski O. SimLingo: vision-only closed-loop autonomous driving with language-action alignment. In: Proceedings of the 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2025 Jun 10–17. Nashville, TN, USA. p. 11993–2003. doi:10.1109/cvpr52734.2025.01120. [Google Scholar] [CrossRef]

11. Fu H, Zhang D, Zhao Z, Cui J, Liang D, Zhang C, et al. Orion: a holistic end-to-end autonomous driving framework by vision-language instructed action generation. In: Proceedings of the 2025 IEEE/CVF International Conference on Computer Vision (ICCV); 2025 Oct 19–25. Honolulu, HI, USA. p. 24823–34. doi:10.1109/iccv51701.2025.02302. [Google Scholar] [CrossRef]

12. Gao T, Tan C, Glossop C, Gao T, Sun J, Stachowicz K, et al. SteerVLA: steering vision-language-action models in long-tail driving scenarios. arXiv:2602.08440. 2026. [Google Scholar]

13. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, et al. Learning transferable visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning. Vol. 139 of Proceedings of Machine Learning Research; 2021 Jul 18–24; Virtual. p. 8748–63. [Google Scholar]

14. Gao Z, Jiang X, Xu X, Shen F, Li Y, Shen HT. Embracing unimodal aleatoric uncertainty for robust multimodal fusion. In: Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2024 Jun 16–22; Seattle, WA, USA. p. 26866–75. doi:10.1109/cvpr52733.2024.02538. [Google Scholar] [CrossRef]

15. Chun S, Kim W, Park S, Yun S. Probabilistic language-image pre-training. In: Proceedings of the International Conference on Learning Representations; 2025 Apr 24–28; Singapore. [Google Scholar]

16. Xie J, Long F, Lv J, Wang Q, Li P. Joint distribution matters: deep brownian distance covariance for few-shot classification. In: Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2022 Jun 18–24; New Orleans, LA, USA. p. 7962–71. doi:10.1109/cvpr52688.2022.00781. [Google Scholar] [CrossRef]

17. Gao M, Wang Q, Lin Z, Zhu P, Hu Q, Zhou J. Tuning pre-trained model via moment probing. In: Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); 2023 Oct 1–6; Paris, France. p. 11769–79. doi:10.1109/iccv51070.2023.01084. [Google Scholar] [CrossRef]

18. Shao Z, Wang P, Zhu Q, Xu R, Song J, Bi X, et al. DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300. 2024. [Google Scholar]

19. Jia X, Yang Z, Li Q, Zhang Z, Yan J. Bench2Drive: towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. Adv Neural Inf Process Syst. 2024;37:819–44. doi:10.52202/079017-0025. [Google Scholar] [CrossRef]

20. Dauner D, Hallgarten M, Li T, Weng X, Huang Z, Yang Z, et al. NAVSIM: data-driven non-reactive autonomous vehicle simulation and benchmarking. Adv Neural Inf Process Syst. 2024;37:28706–19. doi:10.52202/079017-0902. [Google Scholar] [CrossRef]

21. Beißwenger J. PDM-Lite: a rule-based planner for CARLA Leaderboard 2.0. Tübingen, Germany: University of Tübingen; 2024. [Google Scholar]

22. Anderson P, Fernando B, Johnson M, Gould S. SPICE: semantic propositional image caption evaluation. In: Computer vision—ECCV 2016. Cham, Switzerland: Springer; 2016. p. 382–98. doi:10.1007/978-3-319-46454-1_24. [Google Scholar] [CrossRef]

23. Jiang B, Chen S, Liao B, Zhang X, Yin W, Zhang Q, et al. Senna: bridging large vision-language models and end-to-end autonomous driving. arXiv:2410.22313. 2024. [Google Scholar]

24. Gao Z, Chen Z, Cui E, Ren Y, Wang W, Zhu J, et al. Mini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance. Visual Intell. 2024;2(1):32. doi:10.1007/s44267-024-00067-6. [Google Scholar] [CrossRef]

25. Wu P, Jia X, Chen L, Yan J, Li H, Qiao Y. Trajectory-guided control prediction for end-to-end autonomous driving: a simple yet strong baseline. Adv Neural Inf Process Syst. 2022;35:6119–32. doi:10.52202/068431-0443. [Google Scholar] [CrossRef]

26. Jia X, Wu P, Chen L, Xie J, He C, Yan J, et al. Think twice before driving: towards scalable decoders for end-to-end autonomous driving. In: Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2023 Jun 17–24; Vancouver, BC, Canada. p. 21983–94. doi:10.1109/cvpr52729.2023.02105. [Google Scholar] [CrossRef]

27. Jia X, Gao Y, Chen L, Yan J, Liu PL, Li H. DriveAdapter: breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In: Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); 2023 Oct 1–6; Paris, France. p. 7919–29. doi:10.1109/iccv51070.2023.00731. [Google Scholar] [CrossRef]

28. Jia X, You J, Zhang Z, Yan J. DriveTransformer: unified transformer for scalable end-to-end autonomous driving. In: Proceedings of the International Conference on Learning Representations; 2025 Apr 24–28. Singapore. [Google Scholar]

29. Wang T, Zhang C, Qu X, Li K, Liu W, Huang C. DiffAD: a unified diffusion modeling approach for autonomous driving. arXiv:2503.12170. 2025. [Google Scholar]

30. Shang S, Chen Y, Wang Y, Li Y, Zhang Z. DriveDPO: policy learning via safety DPO for end-to-end autonomous driving. Adv Neural Inf Process Syst. 2026;38:81565–85. [Google Scholar]

31. Feng L, Gao Y, Zablocki E, Li Q, Li W, Liu S, et al. RAP: 3D rasterization augmented end-to-end planning. In: Proceedings of the International Conference on Learning Representations; 2026 Apr 23–27. Rio de Janeiro, Brazil. [Google Scholar]

32. Yang Z, Jia X, Li Q, Yang X, Yao M, Yan J. Raw2Drive: reinforcement learning with aligned world models for end-to-end autonomous driving (in CARLA v2). Adv Neural Inf Process Syst. 2026;38:134122–47. [Google Scholar]

33. Jaeger B, Chitta K, Geiger A. Hidden biases of end-to-end driving models. In: Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); 2023 Oct 1–6; Paris, France. p. 8206–15. doi:10.1109/iccv51070.2023.00757. [Google Scholar] [CrossRef]

34. Yang Z, Chai Y, Jia X, Li Q, Shao Y, Zhu X, et al. DriveMoE: mixture-of-experts for vision-language-action model in end-to-end autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2026 Jun 3–7; Denver, CO, USA. p. 10678–88. [Google Scholar]

35. Li Y, Xiong K, Guo X, Li F, Yan S, Xu G, et al. ReCogDrive: a reinforced cognitive framework for end-to-end autonomous driving. In: Proceedings of the International Conference on Learning Representations; 2026 Apr 23–27. Rio de Janeiro, Brazil. [Google Scholar]

36. Fu H, Zhang D, Zhao Z, Cui J, Xie H, Wang B, et al. MindDrive: a vision-language-action model for autonomous driving via online reinforcement learning. arXiv:2512.13636. 2025. [Google Scholar]

37. Li Q, Jia X, Wang S, Yan J. Think2Drive: efficient reinforcement learning by thinking with latent world model for autonomous driving (in CARLA-V2). In: Computer vision—ECCV 2024. Cham, Switzerland: Springer Nature; 2025. p. 142–58. doi:10.1007/978-3-031-72995-9_9. [Google Scholar] [CrossRef]

38. Chen Y, Wang Y, Zhang Z. DrivingGPT: unifying driving world modeling and planning with multi-modal autoregressive transformers. In: Proceedings of the 2025 IEEE/CVF International Conference on Computer Vision (ICCV); 2025 Oct 19–25. Honolulu, HI, USA. p. 26890–900. doi:10.1109/iccv51701.2025.02496. [Google Scholar] [CrossRef]

39. Shi C, Shi S, Sheng K, Zhang B, Jiang L. DriveX: omni scene modeling for learning generalizable world knowledge in autonomous driving. In: Proceedings of the 2025 IEEE/CVF International Conference on Computer Vision (ICCV); 2025 Oct 19–25; Honolulu, HI, USA. p. 28599–609. doi:10.1109/iccv51701.2025.02656. [Google Scholar] [CrossRef]

40. Liao B, Chen S, Yin H, Jiang B, Wang C, Yan S, et al. DiffusionDrive: truncated diffusion model for end-to-end autonomous driving. In: Proceedings of the 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2025 Jun 10–17; Nashville, TN, USA. p. 12037–47. doi:10.1109/cvpr52734.2025.01124. [Google Scholar] [CrossRef]

41. Liu Q, Xu H, Li J, Sun B, Hao Z, She D, et al. Uni-World VLA: interleaved world modeling and planning for autonomous driving. arXiv:2603.27287. 2026. [Google Scholar]

42. Li K, Li Z, Lan S, Xie Y, Zhang Z, Liu J, et al. Hydra-MDP++: advancing end-to-end driving via expert-guided hydra-distillation. arXiv:2503.12820. 2025. [Google Scholar]