LSTM-Enhanced Deep Reinforcement Learning for Active Motion Compensation of Surgical Robots with Known Target Position

1  Introduction

Robot-assisted minimally invasive surgery has been widely used because of its advantages in reducing surgical trauma, postoperative pain, and hospitalization time [1]. However, for procedures involving dynamic organs such as the heart, continuous physiological motion from heartbeats and respirations changes the position of the surgical target, which greatly increases the difficulty of precise manipulation. To improve surgical accuracy and reduce the burden on surgeons, active motion compensation has become an important topic in surgical robotics [2].

Existing studies on active motion compensation mainly focus on two aspects: target motion measurement and motion control. For motion measurement, various sensing modalities have been explored, including ultrasound [3,4], position sensors [5], stereo endoscopy [6–8], accelerometers [9], and laser scanning [10]. Since physiological motion is partially predictable, prediction mechanisms have also been introduced to improve tracking robustness. For example, Richa et al. [11,12] investigated prediction methods based on Kalman filtering and time-varying Fourier series. Stoyanov et al. [13] developed stereo reconstruction and scene-flow approaches for recovering the deforming surgical field, and Yang et al. [14,15] reported robust three-dimensional tracking and prediction methods for soft-tissue and beating-heart motion. Recently, Zhang et al. [16] proposed a motion prediction method based on Long Short-Term Memory (LSTM), using three-dimensional trajectories extracted from stereo endoscopic videos.

In terms of control, traditional active motion compensation methods usually rely on direct error feedback, predictive control or adaptive filtering. Ginhoux et al. [17] proposed predictive control schemes to compensate for heartbeat and respiration, while Franke et al. [18,19] studied adaptive filtering methods for future cardiac motion estimation and control. Although these methods have shown effectiveness, most of them regard motion estimation, prediction and control as separate modules. Such modular pipelines may accumulate errors and usually rely on hand-crafted models or manually adjusted controllers, which limits their adaptability in complex dynamic environments.

In recent years, deep reinforcement learning (DRL) has shown great potential in continuous control problems. Among the representative approaches, the Deep Deterministic Policy Gradient (DDPG) [20] is able to achieve determinism policy learning in the continuous action space, while the Twin Delayed Deep Deterministic Policy Gradient (TD3) [21] improves training stability and reduces value overvaluation. These algorithms have been successfully applied to robot operation and control tasks [22,23]. In surgical motion compensation, Shin et al. [24] explored learning-based predictive control methods and demonstrated the feasibility of reinforcement learning in this field. Compared with traditional control methods, DRL provides a data-driven way to optimize the control strategy in nonlinear dynamic systems without clearly deriving control laws.

However, using DRL-based control for surgical motion compensation presents domain-specific challenges. In a practical surgical robotic system, sensing delays are unavoidable due to image acquisition, stereo reconstruction, and data transmission. These delays mean that the current observation no longer reflects the true system state, thus violating the Markov assumption on which standard DRL controllers are based. Although LSTM has shown strong capabilities in sequence modeling and motion prediction [25], there remains a lack of research on integrating it with DRL to achieve active motion compensation with delay perception. This gap is particularly relevant to surgical robotics because target motions consist of irregular and quasi-periodic cardiac and respiratory components, which require temporal reasoning. Therefore, the core question is not whether LSTM and DRL can be combined in general, but rather whether temporal modeling is a necessary and sufficient architectural addition to enable robust compensation under sensing constraints specific to robotic surgery.

To solve these problems, this study investigates an active motion compensation framework of surgical robots with the assumption that the target position is known. The task is modeled as a Markov decision process, in which the controller receives the position of the surgical instrument and the target point, and outputs continuous control forces along the three Cartesian axes. An LSTM module is incorporated into the actor-critic architecture in order to use historical information to infer the movement trend, so that the method can be more robust under the delayed and noisy observations. Based on this framework, DDPG and TD3 are extended with LSTM and used to train delay-aware controllers. Their performance is systematically compared with that of fully connected controllers and a conventional PD controller. Our study focuses on the control stage of motion compensation under the assumption that an upstream sensing or localization module provides the target position. This deliberate separation allows for a more thorough analysis of the controller’s ability to manage delay and noise, free from interference by confounding factors in the perception pipeline. Integration with online target localization is beyond the scope of this study and will be addressed in future work.

The main contributions of this work are as follows. First, we demonstrate that standard, fully connected DRL controllers are ineffective at tracking physiological motion under a typical sensing delay of 40 ms. We identify the violation of the Markov property as the underlying cause of this failure. This finding was obtained using real in vivo and phantom cardiac motion data. Second, we show that augmenting the actor-critic architecture with an LSTM module restores robust tracking by exploiting the temporal regularity of heartbeats and respiratory motions. This eliminates the need for a separate, handcrafted prediction model. Third, through a systematic comparison of four controllers and a PD baseline, we establish quantitative benchmarks and derive a practical design principle: temporal encoders are a necessary component for learning-based surgical controllers operating under delay.

2  Method

2.1 Problem Formulation

This study adopts a learning-based control framework that combines reinforcement learning (RL) [26] with an LSTM network. The LSTM module is incorporated to capture temporal dependencies in the system and to mitigate the effect of delays through its predictive capability. Meanwhile, RL is employed to learn an optimal control policy through continuous interaction with the environment, enabling accurate tracking of the target point.

In order to evaluate the effectiveness of the proposed method, four algorithms, including DDPG, TD3 and their LSTM-enhanced variants, are studied. In addition, the performance of the learned controllers is compared with that of a conventional PD controller to provide a comprehensive evaluation.

The motion compensation task is formulated as a Markov Decision Process (MDP) [27], in which the agent selects actions according to the current state, receives rewards from the environment, and then transitions to the next state. Separate control policies are trained using DDPG, TD3, and their LSTM-enhanced variants, respectively, and are gradually optimized through interaction with the environment until convergence.

In order to ensure a controlled simulation environment, the surgical instrument is modeled as a rigid body with three translational degrees of freedom along the X, Y and Z axes, and its orientation is assumed to remain unchanged. In practice, the movement of the instrument is affected by various dynamic factors, including gravity, damping and stiffness. Therefore, a dynamic model is built to capture these effects.

Specifically, each degree of freedom is modeled as an independent one-dimensional second-order dynamic system. The Z-axis model considers gravity, damping and stiffness, while the X- and Y-axes mainly consider damping and stiffness. The corresponding dynamic equations are given by the standard mass-spring-damper formulation. Fig. 1 illustrates the overall framework of the proposed method.

Figure 1: Overall framework of the proposed LSTM-enhanced reinforcement learning motion compensation system.

2.2 MDP Formulation for Active Motion Compensation

To realize learning-based active motion compensation, the task is formulated as an MDP. The controller is regarded as an agent that selects an action at according to the current state st. The environment then returns a reward rt and transitions to the next state st+1. Through continuous interaction with the environment, the control policy is gradually optimized to achieve stable and accurate target tracking. This study adopts deep reinforcement learning algorithms, specifically DDPG and TD3, to train the controller by iteratively updating the policy and value networks using environmental feedback.

2.2.1 State Space and Action Space

The design of state space plays a crucial role in reinforcement learning as it directly affects the efficiency and performance of policy learning. In this study, the system state is represented as a flat vector to ensure interpretability and computational efficiency.

For the active motion compensation task, the control decision depends on the spatial relationship between the surgical instrument and the target point. Therefore, the state is defined as a six-dimensional vector that includes the Cartesian coordinates of the surgical instrument Pagent=(xa,ya,za) and the target point of interest Ppoi=(xp,yp,zp). The state is therefore represented as St=[xa,ya,za,xp,yp,zp].

In the considered motion compensation task, the control objective is achieved by applying forces to the surgical instrument along its degrees of freedom. Accordingly, the action space is defined as the control forces applied along the three axes at=[fx,fy,fz], where fx, fy, and fz denote the forces applied along the X, Y, and Z directions, respectively.

Since gravity affects the motion along the Z-axis, gravitational compensation is applied in advance in the simulation environment. As a result, the control inputs correspond to the net forces after gravity compensation.

2.2.2 Reward Function

The design of the reward function aims to guide the controller to achieve precise, efficient and safe motion compensation. Its goal is to minimize the distance between the surgical instrument and the target position and avoid unsafe interactions while ensuring the stability of system behavior. For this reason, a compound reward function is defined:

rt=rtime+rtarget+rmove+rposition+rout+rtouch(1)

At each time step, a small constant penalty term rtime=−0.005 is applied to promote faster convergence. Due to measurement noise and discrete sampling, achieving exactly zero distances is impractical. A tolerance region dt≤d1 is therefore introduced, within which the agent receives a positive reward rtarget=5 to provide effective incentive feedback.

To explicitly encourage movement toward the target, a distance-based reward term is introduced

rmove={Δddt,Δd>02Δddt,Δd<0(2)

where Δd=dt−1−dt. This term normalizes the reward by the current distance dt, providing adaptive scaling of the feedback signal. When approaching the target, the controller receives a stronger guidance signal; when moving away, it is penalized more heavily. An asymmetric scaling factor is used for negative Δd to suppress oscillations near the target, preventing the controller from repeatedly overshooting.

Additionally, a location-dependent shaping term is introduced to guide the agent based on the distance to the target:

rposition={(d2−dt)k1,d1<dt<d2(d3−dt)k2,d2≤dt<d3(3)

This term provides positive feedback when the agent is near the target and applied an increasing penalty as the agent moves farther away, promoting gradual convergence.

To enforce the workspace constraint, when dt≥d3, a penalty rout=−5 is applied and the episode is terminated. This ensures that the agent operates within a physically feasible workspace and prevents divergence during training. Finally, to ensure safe operation, if the surgical instrument touches the soft tissue, a large penalty rtouch=−10 is applied and episode is terminated immediately. Such interactions are strictly prohibited in surgical scenarios, and this penalty mechanism enforces safety-critical behaviors during policy learning.

Note that the MDP formulation above implicitly assumes that the observation reflects the true system state. When sensing delay and measurement noise are introduced, the observation at time step t no longer equals the true system state, and the problem becomes a partially observable Markov decision process (POMDP). In this framework, the LSTM module acts as a recurrent encoder that compresses the history of observations into a hidden state, which serves as a learned belief state. The subsequent actor-critic networks operate on this belief state, effectively reducing the POMDP to a belief MDP that can be solved by standard policy gradient methods.

2.3 Controller Network Architecture

The controller is built in the Actor-Critic framework [28], in which the policy network (actor) is responsible for generating continuous control actions according to the current state, while the value network (critic) evaluates the quality of the state-action pair to guide policy optimization. This framework is especially suitable for the motion compensation task, which involves continuous control in three-dimensional space.

In the baseline design, both the policy network and the value network are realized by a fully connected (FC) architecture. The policy network takes the six-dimensional state vector as input, representing the Cartesian positions of the surgical instrument and the target point, and outputs a three-dimensional action corresponding to the control force on each degree of freedom. The network contains two hidden layers with 200 and 100 neurons, respectively, followed by an output layer with a Tanh activation function to bound the action value. The value network adopts a similar structure, but takes both state and action as input, forming a nine-dimensional input vector and outputting a scalar Q-value.

To address the limitation of relying only on instantaneous observation, an extended architecture integrating LSTM is introduced. In practical scenarios, the system is affected by sensing delay and measurement noise, where the current state may not fully represent the true system state. By integrating an LSTM layer into the network, temporal dependencies can be captured, enabling the controller to use historical information for more stable and accurate decision-making.

Specifically, the LSTM module is inserted after the second fully connected layer in both the policy and value networks. The resulting architecture first extracts spatial features through FC layers and then models temporal correlations via the LSTM, before generating the final action or value output. This design enhances the robustness of the controller and improves its ability to handle delayed and noisy observations. The architecture of the proposed LSTM-enhanced Actor–Critic network is illustrated in Fig. 2. It consists of an Actor network that generates control actions based on the current state and a Critic network that evaluates the corresponding state–action pairs. Both networks incorporate fully connected layers for feature extraction and an LSTM layer for temporal modeling, enabling the framework to capture time-dependent dynamics under delayed and noisy conditions.

Figure 2: Architecture of the proposed LSTM-enhanced Actor–Critic network.

2.4 Dynamic Modeling of the Controlled Objects

To approximate the motion behavior of the surgical instrument in the simulation environment, a simplified dynamic model is constructed. In this study, the surgical instrument is modeled as a rigid body with mass and three degrees of freedom in Cartesian space, allowing motion along the X-, Y-, and Z-axes.

During motion compensation, the instrument is subject to external control forces as well as intrinsic physical effects such as inertia, damping, and stiffness. A second-order dynamic model is therefore adopted to describe the relationship between the applied forces and the resulting motion.

For the Z-axis, where gravitational effects must be considered, the dynamic equation is defined as:

fz−fg=Mp¨z(t)+Bp˙z(t)+Kpz(t)(4)

Similarly, the dynamics along the X- and Y-axes can be expressed as:

fx=Mp¨x(t)+Bp˙x(t)+Kpx(t)(5)

fy=Mp¨y(t)+Bp˙y(t)+Kpy(t)(6)

In Eqs. (4)–(6), fx, fy, and fz denote the control forces applied along the X-, Y-, and Z-axes, respectively, while fg represents the gravitational force acting along the Z-axis, M denotes the mass of the surgical instrument, B is the damping coefficient, and K is the stiffness coefficient. The terms px(t), py(t), and pz(t) represent the position of the surgical instrument in the Cartesian coordinates, while p˙x(t), p˙y(t), and p˙z(t) denote the corresponding velocities, and p¨x(t), p¨y(t), and p¨z(t) denote the accelerations along each axis.

As shown in Eqs. (4)–(6), the motion of the surgical instrument is governed by a unified second-order dynamic system across all three axes. This formulation establishes a physically interpretable mapping between the control inputs and the system responses, so that the reinforcement learning agent can interact with a physically grounded and controlled environment.

2.5 LSTM-Enhanced Reinforcement Learning Framework

To achieve stable and precise motion compensation in a dynamic environment, this paper proposes a delay-aware reinforcement learning framework. The method builds on deterministic policy gradient algorithms (DDPG and TD3) and integrates temporal sequence modeling to handle sensing delay and partial observability. The framework consists of three main components: a basic Actor–Critic learning mechanism, a stability enhancement strategy, and an LSTM-based temporal modeling module.

2.5.1 Base Actor-Critic Learning

The controller is built under the Actor-Critic framework for continuous control. In this framework, the Actor network generates control actions according to the current state, while the Critic network evaluates the corresponding state-action pair to guide the optimization of policy.

To improve training efficiency and stability, an experience replay mechanism is employed, where transition samples (st,at,rt,st+1) are stored and randomly sampled during training. In addition, target networks are introduced to stabilize the learning process by slowly tracking the parameters of the online networks.

The Critic network is optimized by minimizing the temporal-difference (TD) error:

Loss=1N∑i=1N(yi−Q(si,ai))2(7)

where the target value is computed using the target networks:

yi=ri+γQ′(si+1,μ′(si+1))(8)

In Eq. (8), Q and Q′ denote the action-value functions of the online and target Critic networks, respectively, μ and μ′ represent the deterministic policies of the Actor and target Actor networks, γ∈(0,1) is the discount factor, and N denotes the mini-batch size.

The Actor network is updated by maximizing the expected Q-value through policy gradients, enabling the controller to gradually improve its action selection strategy. The training framework of the DDPG algorithm is illustrated in Fig. 3.

Figure 3: Training framework of the DDPG algorithm.

2.5.2 Stability Enhancement via TD3

Although the Actor–Critic framework is effective for continuous control, it is prone to overestimation bias in value learning. To address this issue, the TD3 algorithm is adopted as a stability enhancement strategy.

TD3 introduces three key improvements: (1) twin Critic networks to reduce overestimation by taking the minimum predicted value, (2) delayed policy updates to stabilize training, and (3) target policy smoothing by adding noise to the target action. These modifications improve convergence stability during training.

With these improvements, the target value is modified as:

yi=ri+γminj=1,2Qj′(si+1,a~i+1)(9)

where Q1′ and Q2′ denote two target Critic networks, and a~i+1 represents the target action with clipped noise added to the target policy output. Fig. 4 illustrates the training process of the TD3 algorithm. TD3 adopts twin Critic networks to mitigate overestimation bias by taking the minimum predicted Q-value, compared with the standard Actor–Critic framework. Target policy smoothing and delayed policy update mechanisms are also introduced to improve training stability and robustness.

Figure 4: Training framework of the TD3 algorithm.

2.5.3 Temporal Modeling with LSTM

Although the above framework provides a solid foundation for continuous control, it is based on the assumption that the current state can fully represent the environment. However, in the actual motion compensation scenario, the system is often affected by the sensing delay and measurement noise, resulting in partial observability of the system. With delayed and noisy observations, the current state alone may not accurately reflect the true system dynamics, and temporal information becomes crucial for effective decision-making.

To address this limitation, we introduce an LSTM-based temporal modeling mechanism in the Actor-Critic framework. By integrating sequence information, the controller can capture temporal dependencies and infer implicit system dynamics from historical observation data.

In this setting, the replay buffer stores complete episode trajectories. During training, N trajectories are randomly selected from the buffer. For each trajectory, a starting index is chosen at random and a sequence of S consecutive steps is extracted. Before processing each sequence, the LSTM hidden states of all networks, including the current Actor and Critic(s), as well as their target counterparts, are reset to zero. This reset prevents gradient flow across unrelated trajectory segments and ensures that the hidden states evolve solely from the observations within the current sequence. This design mimics the cold-start condition that the controller encounters during deployment while maintaining the stability of off-policy learning. The TD3-specific stabilizing mechanisms (clipped double Q-learning, delayed policy updates, and target policy smoothing) remain in effect to mitigate overestimation bias in the sequence-based setting. Empirically, the training process remained stable across all LSTM-enhanced experiments.

Fig. 5 illustrates the training frameworks of the proposed LSTM-enhanced DDPG and TD3 algorithms. Both methods incorporate sequence-based sampling and LSTM modules to capture temporal dependencies. Compared with DDPG, TD3 further improves stability through twin Critic networks, delayed policy updates, and target policy smoothing. The LSTM extension is consistently applied to both Actor and Critic networks in each framework. To further clarify the training procedures, the LSTM-enhanced DDPG and TD3 algorithms are summarized in Algorithms 1 and 2, respectively.

Figure 5: Training frameworks of the LSTM-enhanced DDPG and TD3 algorithms.

3  Results and Analysis

3.1 Unity-Based Simulation Environment Setup

The simulation environment is built using the Unity ML-Agents framework. The control process of the surgical instrument is implemented through customized scripts that define state observations, control actions, and reward signals. The CollectObservations() function is used to obtain the displacement information of surgical instruments and target points, while OnActionReceived() maps the agent output to control command. The reward function is implemented through AddReward(), and collision events is detected by OnCollisionEnter().

The surgical instrument is modeled as rigid bodies with mass, and its movement is driven by external forces via rBody.AddForce(). The movement of the target point is controlled by updating the transform.localPosition variable at each time step. Specifically, the update is executed in the FixedUpdate() function, so that the target point moves along a predefined trajectory, which is generated from the motion signals described in Section 3.2.1.

In this study, the target point on the soft tissue is treated as a fixed tracking point, and the soft tissue deformation is not explicitly modeled. The surgical instrument is represented as a cylinder, which is a commonly used representation in the simulation environment. Both the instrument and the target point move along X, Y, and Z axes, corresponding to the three translational degrees of freedom.

The goal of motion compensation is to maintain a consistent relative position between the surgical instrument and the moving target. The instrument is driven by the control forces, while the target follows the predefined movement patterns. Fig. 6 shows the typical configurations in different states.

Figure 6: Representative configurations of the surgical instrument and the target point under different states.

3.2 Implementation Details and Evaluation Metrics

3.2.1 Training Data and Motion Modeling

In the active motion compensation task, the target movement is mainly driven by the combined action of cardiac and respiratory dynamics. To construct a controllable and interpretable simulation scenario, these two components are modeled as sinusoidal signals with different frequencies and amplitudes. Table 1 summarizes the specific parameters used in simulation. As a reference movement for the compensation task, a quasi-period trajectory is generated through combining these two components. Fig. 7 shows the movement generated along the Z-axis, in which each component and the combined signal can be clearly observed.

Figure 7: Simulated motion signals composed of respiratory (red) and cardiac (blue) components, and their combined motion (green).

In addition to simulated motion signals, two real motion datasets are also used for verification: Phantom dataset and in vivo dataset. Both datasets come from the publicly available Hamlyn Centre Laparoscopic/Endoscopic Video Dataset released by Imperial College London, which has been widely used in surgical vision research. The Phantom dataset provides controlled motion patterns generated by a pneumatically driven silicone heart model [29], while the in vivo dataset was collected during a total endoscopic coronary artery bypass (TECAB) procedure using the Da Vinci robot, reflecting more real surgical conditions, including tissue deformation and complex imaging artifacts [13,30]. Each dataset contains 750 frames captured at a 25 Hz sampling rate, of which the first 600 frames are used for training and the remaining 150 frames are used for testing.

Each frame contains the three-dimensional positions of the target point. Fig. 8 shows the trajectories of the two datasets on the X, Y and Z axes.

Figure 8: Motion trajectories of the target point in the Phantom (a) and in vivo (b) datasets along the X, Y, and Z axes.

To better reflect realistic operating conditions, a sensing delay of 40 ms is introduced in the simulation environment. This is equivalent to a delay of one time step, i.e., the observation at the current time step is taken from the previous frame. In addition, to simulate the measurement noise in a real sensing system, zero-mean Gaussian noise with a standard deviation of 0.1 mm is added independently to each axis.

3.2.2 Evaluation Metric and Testing Protocol

The performance of the controller is evaluated based on the tracking accuracy between the surgical instrument and the target point. The quantitative metric adopted is the root mean square error (RMSE) of the Euclidean distance.

For simulated signals, a random starting point within one motion cycle is selected, and the tracking performance over the next 100 time steps is evaluated. For the in vivo and Phantom datasets, training is performed on the first 600 frames, while testing is conducted on the remaining 150 frames. During testing, a starting point is randomly selected from frames 601 to 650, and the tracking performance over the subsequent 100 steps is measured.

To reduce the influence of randomness in both controller initialization and starting point selection, multiple controllers are trained independently. For each controller, multiple starting points are randomly sampled. Since the tracking and training curves were consistent across different initializations and starting points, representative examples are shown in Figs. 9–20 for clarity. All RMSE values reported in subsequent tables are computed using Eq. (10):

RMSE=1NCNsNt∑i=1Nc∑j=1Ns∑k=1Nt∥Pi,j,kpred−Pi,j,kgt∥2(10)

where NC is the number of trained controllers (set to 10), Ns is the number of randomly selected starting points (set to 20), Nt is the prediction horizon (100 steps), Ppred and Pgt denote the predicted and ground-truth positions, respectively.

Figure 9: Training performance of DDPG on the in vivo dataset: (a) cumulative reward per episode; (b) average reward per step.

Figure 10: Training performance of TD3 on the in vivo dataset: (a) cumulative reward per episode; (b) average reward per step.

Figure 11: Tracking performance of DDPG on the in vivo dataset: (a) trajectory tracking; (b) tracking error.

Figure 12: Tracking performance of TD3 on the in vivo dataset: (a) trajectory tracking; (b) tracking error.

Figure 13: Training performance of the DDPG-LSTM controller on the in vivo dataset: (a) cumulative reward per episode; (b) average reward per step.

Figure 14: Motion compensation performance of the DDPG-LSTM controller on the in vivo dataset: (a) trajectory tracking along the X, Y, and Z axes; (b) tracking error.

Figure 15: Motion compensation performance of the DDPG-LSTM controller on the Phantom dataset: (a) trajectory tracking along the X, Y, and Z axes; (b) tracking error.

Figure 16: Training performance of the TD3-LSTM controller on the in vivo dataset: (a) cumulative reward per episode; (b) average reward per step.

Figure 17: Motion compensation performance of the TD3-LSTM controller on the in vivo dataset: (a) trajectory tracking along the X, Y, and Z axes; (b) tracking error.

Figure 18: Motion compensation performance of the TD3-LSTM controller on the Phantom dataset: (a) trajectory tracking along the X, Y, and Z axes; (b) tracking error.

Figure 19: Motion compensation performance of the PD controller on the Phantom dataset: (a) trajectory tracking along the X, Y, and Z axes; (b) tracking error.

Figure 20: Motion compensation performance of the PD controller on the in vivo dataset: (a) trajectory tracking along the X, Y, and Z axes; (b) tracking error.

3.2.3 Training Environment

All experiments are conducted using the Unity ML-Agents framework. The implementation relies on script-based interaction between the environment and the learning agent. The system configuration is summarized in Table 2.

3.3 Performance of the FC-Based Controller

The FC-based controller is implemented using a fully-connected neural network as the policy network. To evaluate its performance under realistic conditions, a sensing delay of 40 ms is introduced into the observed state. In addition, zero-mean Gaussian noise is added to the position measurements to assess the robustness of the controller. During training, the parameters of the reward function are set as d1=0.3 mm, d2=2.0 mm, d3=10.0 mm, k1=0.5, and k2=0.15.

The training performance of the DDPG-based controller on the in vivo dataset is shown in Fig. 9, where the left subplot presents the cumulative reward per episode, and the right subplot shows the average reward per step. The model is trained for 1500 episodes. As observed, after approximately 200 episodes, the cumulative reward exhibits a gradual increase; however, it remains below zero throughout training. The final cumulative reward stabilizes in the range of approximately −100 to 0, while the average step reward converges to about −0.5. These values indicate that the controller failed to achieve effective motion compensation, given the reward function described above.

Fig. 10 shows the corresponding results obtained using the TD3 algorithm. Compared with DDPG, TD3 exhibits better learning behavior. After about 500 rounds, the cumulative rewards began to increase, with some episodes achieving positive rewards. The final cumulative reward stabilizes in the range of approximately −50 to 20, and the average reward per step converges between 0 and 0.5. However, the reward values still indicate suboptimal performance, meaning that the FC-based TD3 controller remains insufficient for precise motion compensation.

The tracking performance of both controllers is further evaluated, as shown in Figs. 11 and 12. In both cases, the controller is able to follow the general trend of the target trajectory; however, significant errors are observed, particularly near turning points. Moreover, noticeable lag is present in the response, and the tracking error is predominantly within the range of 1–3 mm. These results demonstrate that neither DDPG nor TD3 can provide precise motion compensation when implemented with a purely feedforward FC architecture.

Similar observations are obtained when training on the Phantom dataset and simulated signals; therefore, those results are omitted for brevity. The main reason for this limitation lies in the delayed observation of the system state. Because of the 40 ms sensing delay, the available state information at the current time step corresponds to the previous time step. Since the FC-based controller lacks temporal modeling capability, the current system dynamics cannot be inferred from historical information. Consequently, the controller cannot effectively compensate for the delay, leading to persistent tracking errors.

These results indicate that under delay and noise, the traditional DDPG and TD3 algorithms with feedforward FC networks remain inadequate for the motion compensation tasks.

3.4 Performance of the FC-LSTM Controller

The FC-LSTM controller combines fully connected layers with an LSTM layer to achieve temporal modeling. A sensing delay of 40 ms is introduced in the observed state, and Gaussian noise is added to evaluate robustness under realistic conditions, consistent with the settings of the FC-based controller. The reward function parameters are identical to those described above.

3.4.1 Performance of DDPG-LSTM

Fig. 13 shows the training process of the DDPG-LSTM controller on the in vivo data set, presenting the cumulative reward per episode and the average reward per step. A total of 800 episodes were conducted. In the first 100 episodes, the agent mainly interacts with the environment to accumulate experience, after which policy learning begins.

As training progressed, the cumulative reward value began to rise around the 500th episode and reached the peak of approximately 2100 around the 700th episode. Simultaneously, the average reward per step increased to about 3.8. These values indicate that the controller successfully learned effective motion compensation strategies.

The tracking performance on the test set is shown in Fig. 14. The controller captures the overall trend of the target trajectory, but deviations are observed in regions with rapid motion changes. The tracking error is mainly distributed in the range of 0–1.25 mm, with an occasional peak reaches 1.75 mm. The final RMSE is 0.6889 mm, which is higher than the results attained from the simulated and Phantom datasets, reflecting the increased complexity and variability of in vivo motion.

Fig. 15 shows the performance of the DDPG-LSTM controller on the Phantom dataset. The tracking results indicate that the controller can follow the target trajectory with relatively small deviation. Most errors are concentrated in the range of 0–0.8 mm, with an occasional peak ups to 1.4 mm. The final RMSE is 0.4720 mm, which is lower than that of the in vivo dataset, as the motion pattern is more regular.

3.4.2 Performance of TD3-LSTM

Fig. 16 presents the training results of the TD3-LSTM controller on the in vivo dataset. Compared with DDPG-LSTM, the training process of TD3-LSTM is more stable. The cumulative reward began to rise around the 500th episode, reached approximately 2600 around the 1300th episode, and then stabilized. The average reward per step converges to roughly 4.2.

This improvement can be attributed to the design of TD3, which incorporates twin Critic networks and delayed policy updates, thereby reducing overestimation bias and stabilizing the learning process.

The corresponding tracking performance is shown in Fig. 17. Compared with DDPG-LSTM, the controller achieves more precise prediction. The tracking error is mainly distributed in the range of 0.2–1.0 mm, with an occasional peak ups to 1.6 mm. The RMSE is 0.6136 mm, which is 0.0753 mm lower than that of DDPG-LSTM.

Fig. 18 shows the tracking performance of TD3-LSTM on the Phantom dataset. The controller accurately follows the trajectory, with error mainly concentrated in the range of 0–0.6 mm, and only occasional deviations reaching 1.0 mm. The final RMSE is 0.3823 mm, which is 0.0897 mm lower than that of DDPG-LSTM, indicating improved accuracy and robustness.

3.4.3 Quantitative Comparison

Table 3 summarizes the quantitative comparison of motion compensation performance across the three datasets. Overall, the FC-LSTM controllers significantly outperform the FC-based controllers, demonstrating that temporal modeling is essential for handling delayed observations. By integrating historical information, the LSTM layer allows the controller to infer the current system state more accurately, thereby reducing tracking error.

Differences in tracking accuracy can also be observed across datasets. Simulated signals and Phantom data yield lower errors due to their relatively regular and predictable motion patterns, while the in vivo dataset presents greater challenges because of its inherent variability and noise.

In addition, TD3-LSTM consistently outperforms DDPG-LSTM in terms of both RMSE and training stability. This improvement is mainly attributed to the twin Critic networks and delayed policy updates in TD3, which reduce value overestimation bias and enhance stability. The reduced policy update frequency, however, leads to longer training times.

These results suggest that conventional reinforcement learning methods based on the feedforward network are insufficient for motion compensation under delayed observation. In contrast, the proposed FC-LSTM controller introduces temporal modeling, which effectively addresses this limitation. Among the evaluated methods, TD3-LSTM achieves the best overall performance, striking a good balance between accuracy and stability.

3.5 Comparison with Conventional and Learning-Based Methods

To further evaluate the effectiveness of the proposed FC-LSTM controller, its performance is compared with a traditional direct error feedback control method. In this study, given its simple structure and widespread use in real-time control applications, the proportional–derivative (PD) controller is selected as a representative benchmark. The integral item is omitted, and an independent PD controller is designed for each degree of freedom of the surgical instrument, given the requirements of the motion compensation task for fast response and stability.

Eqs. (11) and (12) present the control error and its temporal difference:

e=pa−pt(11)

e˙=enow−elast(12)

where pa and pt represent the positions of the surgical instruments and the target points, respectively, and enow and elast represent the current and previous errors, respectively. The control input is given by:

f=kpe+kde˙(13)

where kp is the proportional gain and kd is the derivative gain.

To ensure a fair comparison, the same experimental conditions used for the learning-based method are adopted. Fig. 19 shows the motion compensation results of the PD controller on the Phantom dataset. The controller can roughly follow the motion trend, but obvious deviations occur at the rapid changes in the trajectory. The tracking error is mainly distributed in the range of 0.5–1.5 mm, with an RMSE of 0.9610 mm.

The performance on the in vivo dataset is presented in Fig. 20. Compared with the Phantom dataset, the motion trajectory is less regular, and the tracking accuracy is further reduced. Although the controller can still follow the general motion pattern, the error remains significant, with most values falling within the range of 0–1.0 mm. The corresponding RMSE is 0.8353 mm.

A quantitative comparison between the learning-based controllers (DDPG-LSTM and TD3-LSTM) and the PD controller is summarized in Table 4. The PD controller consistently yields larger errors across both datasets. For the Phantom dataset, the RMSE of the PD controller is 0.5787 mm higher than that of the TD3-LSTM controller, and for the in vivo dataset, the difference is 0.2217 mm.

These results indicate that the FC-LSTM controller significantly outperforms the conventional PD controller in motion compensation tasks under delayed and noisy observations. The improvement can be attributed to the ability of the LSTM module to capture temporal dependencies and implicitly compensate for sensing delays, which cannot be handled by memoryless controllers such as PD. Additionally, through the reinforcement learning framework, the controller learns adaptive control strategies during interaction with the environment, further enhancing the robustness of the system under complex dynamic conditions.

Overall, the comparison demonstrates that combining temporal modeling with reinforcement learning provides an effective solution for motion compensation in delayed systems, particularly when dealing with irregular and partially observable motion patterns.

4  Discussion

The experimental results demonstrate that temporal modeling is crucial for active motion compensation under delayed observations. Standard DDPG and TD3 controllers with fully connected architectures fail to achieve satisfactory tracking because a single delayed observation cannot accurately reflect the current system state, preventing the controller from inferring motion trends or compensating for the delay. In contrast, LSTM-enhanced controllers significantly improve performance across all datasets by modeling temporal dependencies and implicitly inferring system dynamics from historical observations. The LSTM module is not a generic add-on but a targeted design choice motivated by the strong temporal correlations inherent in heartbeat and respiration. This allows the controller to compensate for delays without an explicit prediction module, offering architectural simplicity that is attractive for surgical applications.

Between the two LSTM-enhanced variants, TD3-LSTM consistently outperforms DDPG-LSTM in both accuracy and stability, owing to its twin Critic structure and delayed policy updates that reduce overestimation bias. The performance ranking across datasets, simulated signals achieving the lowest error, followed by Phantom, then in vivo, indicates that the proposed framework benefits from regularity in the target motion. Highly irregular in vivo motion weakens the advantage of temporal memory, leading to larger residual errors. Nevertheless, the sub-millimeter RMSE achieved on in vivo data is a promising indicator of robustness under clinically realistic conditions.

The comparison with the PD controller further confirms the value of the learning-based approach. Unlike direct error feedback with manually tuned gains, the proposed controller learns adaptive strategies through interaction with the environment, making it more robust to coexisting delay, measurement noise, and the nonlinear, quasi-periodic tissue motion.

Several limitations should be acknowledged. First, all experiments were conducted in a Unity-based simulation environment. Although the motion data are derived from real in vivo and phantom recordings, using a physical robotic platform introduces additional challenges, including hardware integration, real-time sensing pipelines, and safety constraints, which are not addressed here. Within this simulation, the surgical instrument is modeled as a rigid body with three translational degrees of freedom and decoupled second-order dynamics, a classical and widely used modeling approach in robotics and control. The soft tissue is treated as a fixed tracking point without explicit deformation. These modeling choices were made to focus the experimental setting on the delay compensation problem. However, they do not capture coupled robot-tissue dynamics or multi-axis interactions. Validation on a physical robot and incorporating of more realistic physical interactions are therefore important directions for future work. Second, the target position is assumed to be known, whereas in practice localizing the target is challenging due to tissue deformation, visual occlusion, and tracking errors. This work focuses on the control stage and does not address the perception stage. Integrating the proposed controller with a vision-based localization module remains a key area for future research. Third, the sensing delay is fixed at 40 ms, and the measurement noise is modeled as zero-mean Gaussian. Real surgical systems may involve more diverse and time-varying delay sources, as well as non-Gaussian sensing errors arising from occlusions, tissue artifacts, and tracking failures. Systematic delay sensitivity analysis and more realistic noise model are important directions for future work. Finally, although LSTM improves robustness under delay, its performance may be constrained by the limited training data (750 frames per dataset), which may restrict the model’s ability to capture longer temporal dependencies. This is partly because in vivo recordings are short by nature, and the other datasets were matched to this length for fair comparison. Future work with larger and more diverse datasets, along with more expressive temporal models, could help reduce the remaining tracking errors on highly irregular in vivo motion.

5  Conclusion

This study proposes an LSTM-enhanced deep reinforcement learning framework for active motion compensate of surgical robots with a known target position. The compensation task was formulated as a continuous control problem, and four controllers, DDPG, TD3, DDPG-LSTM and TD3-LSTM, were evaluated under sensing delay and measurement noise. The results show that the standard fully connected DRL controllers cannot effectively compensate under delayed observations, whereas the introduction of LSTM significantly improves tracking accuracy by incorporating temporal information. Among the evaluated methods, TD3-LSTM achieves the best overall performance and exhibits stronger robustness than DDPG-LSTM. Additionally, the proposed LSTM-enhanced controller outperforms the conventional PD controller. These results provide a clear design recommendation for the surgical robotics community: when sensing delays are present, purely memoryless reinforcement learning controllers are fundamentally limited, and temporal encoders such as LSTM should be considered a necessary component for delay-aware surgical control. Although this study is currently limited to a simulation environment with the assumption of a known target position, it establishes a practical foundation for future research on more complex, clinically deployable motion compensation systems.

Acknowledgement: Not applicable.

Funding Statement: Supported by Chengdu Science and Technology Program [2026-YF08-00034-GX].

Author Contributions: The authors confirm contribution to the paper as follows: study conception and design: Bo Yang, Wenfeng Zheng; data collection: Wei Wei, Qi Hu; analysis and interpretation of results: Shujuan Li, Legend Zhang, Junmin Lyu; draft manuscript preparation: Wei Wei, Bo Yang, Qi Hu. All authors reviewed and approved the final version of the manuscript.

Availability of Data and Materials: Data descriptions and references are provided in Section 3.2.1.

Ethics Approval: Not applicable.

Conflicts of Interest: The authors declare no conflicts of interest.

References

1. Haidegger T, Speidel S, Stoyanov D, Satava RM. Robot-assisted minimally invasive surgery—surgical robotics in the data age. Proc IEEE. 2022;110(7):835–46. doi:10.1109/JPROC.2022.3180350. [Google Scholar] [CrossRef]

2. Ciuti G, Webster RJ III, Kwok KW, Menciassi A. Robotic surgery. Nat Rev Bioeng. 2025;3(7):565–78. doi:10.1038/s44222-025-00294-6. [Google Scholar] [CrossRef]

3. Pavone M, Seeliger B, Teodorico E, Goglia M, Taliento C, Bizzarri N, et al. Ultrasound-guided robotic surgical procedures: a systematic review. Surg Endosc. 2024;38(5):2359–70. doi:10.1007/s00464-024-10772-4. [Google Scholar] [CrossRef]

4. Hidalgo EM, Wright L, Isaksson M, Lambert G, Marwick TH. Current applications of robot-assisted ultrasound examination. JACC Cardiovasc Imaging. 2023;16(2):239–47. doi:10.1016/j.jcmg.2022.07.018. [Google Scholar] [CrossRef]

5. Dinesh S, Sahu UK, Sahu D, Dash SK, Yadav UK. Review on sensors and components used in robotic surgery: recent advances and new challenges. IEEE Access. 2023;11(1):140722–39. doi:10.1109/ACCESS.2023.3339555. [Google Scholar] [CrossRef]

6. Ma X, Song C, Qian L, Liu W, Chiu PW, Li Z. Augmented reality-assisted autonomous view adjustment of a 6-DOF robotic stereo flexible endoscope. IEEE Trans Med Robot Bionics. 2022;4(2):356–67. doi:10.1109/TMRB.2022.3155254. [Google Scholar] [CrossRef]

7. Wang Y, Long Y, Fan SH, Dou Q. Neural rendering for stereo 3D reconstruction of deformable tissues in robotic surgery. In: Medical Image Computing and Computer Assisted Intervention—MICCAI 2022. Cham, Switzerland: Springer Nature; 2022. p. 431–41. doi:10.1007/978-3-031-16449-1_41. [Google Scholar] [CrossRef]

8. Yang B, Xu S, Yin L, Liu C, Zheng W. Disparity estimation of stereo-endoscopic images using deep generative network. ICT Express. 2025;11(1):74–9. doi:10.1016/j.icte.2024.09.017. [Google Scholar] [CrossRef]

9. Procházka A, Martynek D, Vitujová M, Janáková D, Charvátová H, Vyšata O. Mobile accelerometer applications in core muscle rehabilitation and pre-operative assessment. Sensors. 2024;24(22):7330. doi:10.3390/s24227330. [Google Scholar] [CrossRef]

10. Lee HC, Pacheco NE, Fichera L, Russo S. When the end effector is a laser: a review of robotics in laser surgery. Adv Intell Syst. 2022;4(10):2200130. doi:10.1002/aisy.202200130. [Google Scholar] [CrossRef]

11. Richa R, Bó APL, Poignet P. Beating heart motion prediction for robust visual tracking. In: 2010 IEEE International Conference on Robotics and Automation; 2010 May 3–7; Anchorage, AK, USA. p. 4579–84. doi:10.1109/ROBOT.2010.5509894. [Google Scholar] [CrossRef]

12. Richa R, Bo APL, Poignet P. Motion prediction for tracking the beating heart. In: 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2009 Aug 20–25; Vancouver, BC, Canada. p. 3261–4. doi:10.1109/IEMBS.2008.4649900. [Google Scholar] [CrossRef]

13. Stoyanov D, Scarzanella MV, Pratt P, Yang GZ. Real-time stereo reconstruction in robotically assisted minimally invasive surgery. In: Medical Image Computing and Computer-Assisted Intervention—MICCAI 2010. Berlin/Heidelberg, Germany: Springer; 2010. p. 275–82. doi:10.1007/978-3-642-15705-9_34. [Google Scholar] [CrossRef]

14. Yang B, Wong WK, Liu C, Poignet P. 3D soft-tissue tracking using spatial-color joint probability distribution and thin-plate spline model. Pattern Recognit. 2014;47(9):2962–73. doi:10.1016/j.patcog.2014.03.020. [Google Scholar] [CrossRef]

15. Yang B, Liu C, Zheng W, Liu S. Motion prediction via online instantaneous frequency estimation for vision-based beating heart tracking. Inf Fusion. 2017;35(10):58–67. doi:10.1016/j.inffus.2016.09.004. [Google Scholar] [CrossRef]

16. Zhang W, Yao G, Yang B, Zheng W, Liu C. Motion prediction of beating heart using spatio-temporal LSTM. IEEE Signal Process Lett. 2022;29:787–91. doi:10.1109/LSP.2022.3154317. [Google Scholar] [CrossRef]

17. Ginhoux R, Gangloff J, de Mathelin M, Soler L, Sanchez MMA, Marescaux J. Active filtering of physiological motion in robotized surgery using predictive control. IEEE Trans Robot. 2005;21(1):67–79. doi:10.1109/TRO.2004.833812. [Google Scholar] [CrossRef]

18. Franke TJ, Bebek O, Cavusoglu MC. Improved prediction of heart motion using an adaptive filter for robot assisted beating heart surgery. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems; 2007 Oct 29–Nov 2; San Diego, CA, USA. p. 509–15. doi:10.1109/IROS.2007.4399461. [Google Scholar] [CrossRef]

19. Franke TJ, Bebek O, Cavusoglu MC. Prediction of heartbeat motion with a generalized adaptive filter. In: 2008 IEEE International Conference on Robotics and Automation; 2008 May 19–23; Pasadena, CA, USA. p. 2916–21. doi:10.1109/ROBOT.2008.4543652. [Google Scholar] [CrossRef]

20. Sumiea EH, Abdulkadir SJ, Alhussian HS, Al-Selwi SM, Alqushaibi A, Ragab MG, et al. Deep deterministic policy gradient algorithm: a systematic review. Heliyon. 2024;10(9):e30697. doi:10.1016/j.heliyon.2024.e30697. [Google Scholar] [CrossRef]

21. Fujimoto S, van Hoof H, Meger D. Addressing function approximation error in actor-critic methods. In: International Conference on Machine Learning. Cambridge, MA, USA: PMLR; 2018. p. 1587–96. [Google Scholar]

22. Gu S, Holly E, Lillicrap T, Levine S. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: 2017 IEEE International Conference on Robotics and Automation (ICRA); 2017 May 29–Jun 3; Singapore, Singapore. p. 3389–96. doi:10.1109/ICRA.2017.7989385. [Google Scholar] [CrossRef]

23. Hua J, Zeng L, Li G, Ju Z. Learning for a robot: deep reinforcement learning, imitation learning, transfer learning. Sensors. 2021;21(4):1278. doi:10.3390/s21041278. [Google Scholar] [CrossRef]

24. Shin C, Ferguson PW, Pedram SA, Ma J, Dutson EP, Rosen J. Autonomous tissue manipulation via surgical robot using learning based model predictive control. In: 2019 International Conference on Robotics and Automation (ICRA); 2019 May 20–24; Montreal, QC, Canada. p. 3875–81. doi:10.1109/icra.2019.8794159. [Google Scholar] [CrossRef]

25. Behera B, Orlando MF, Anand RS. Enhancing surgeon feedback via LSTM driven prediction of tissue puncturing events. In: Bridging the Gap between mind and machine: exploring the future of Human-AI-Neurotechnology integration. Cham, Switzerland: Springer Nature; 2026. p. 299–314. doi:10.1007/978-3-032-06713-5_15. [Google Scholar] [CrossRef]

26. Shakya AK, Pillai G, Chakrabarty S. Reinforcement learning algorithms: a brief survey. Expert Syst Appl. 2023;231(7):120495. doi:10.1016/j.eswa.2023.120495. [Google Scholar] [CrossRef]

27. Sutton RS, Barto AG. Reinforcement learning: an introduction. IEEE Trans Neural Netw. 1998;9(5):1054. doi:10.1109/tnn.1998.712192. [Google Scholar] [CrossRef]

28. Konda V, Tsitsiklis J. Actor-critic algorithms. In: Advances in neural information processing systems. Cambridge, MA, USA: MIT Press. p. 1008–14. [Google Scholar]

29. Stoyanov D, Mylonas GP, Deligianni F, Darzi A, Yang GZ. Soft-tissue motion tracking and structure estimation for robotic assisted MIS procedures. In: Medical Image Computing and Computer-Assisted Intervention—MICCAI 2005. Berlin/Heidelberg, Germany: Springer; 2005. p. 139–46. doi:10.1007/11566489_18. [Google Scholar] [CrossRef]

30. Pratt P, Stoyanov D, Visentini-Scarzanella M, Yang GZ. Dynamic guidance for robotic surgery using image-constrained biomechanical models. In: Medical Image Computing and Computer-Assisted Intervention—MICCAI 2010. Berlin/Heidelberg, Germany: Springer; 2010. p. 77–85. doi:10.1007/978-3-642-15705-9_10. [Google Scholar] [CrossRef]