[BACK]
Intelligent Automation & Soft Computing
DOI:10.32604/iasc.2022.016543
images
Article

Deformation Expression of Soft Tissue Based on BP Neural Network

Xiaorui Zhang1,2,*, Xun Sun1, Wei Sun2, Tong Xu1, Pengpai Wang1 and Sunil Kumar Jha3

1Engineering Research Center of Digital Forensics, Ministry of Education, Jiangsu Engineering Center of Network Monitoring, School of Computer and Software, Nanjing University of Information Science & Technology, Nanjing, 210044, China
2Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology, Nanjing University of Information Science & Technology, Nanjing, 210044, China
3Faculty of Information Technology, University of Information Technology and Management, Rzeszow, 35-225, Poland
*Corresponding Author: Xiaorui Zhang. Email: zxr365@126.com
Received: 04 January 2021; Accepted: 13 August 2021

Abstract: This paper proposes a soft tissue grasping deformation model, where BP neural network optimized by the genetic algorithm is used to realize the real-time and accurate interaction of soft tissue grasping during virtual surgery. In the model, the soft tissue epidermis is divided into meshes, and the meshes generate displacements under the action of tension. The relationship between the tension and displacement of the mesh is determined by the proposed cylindrical spiral spring model. The optimized BP neural network is trained based on the sample data of the mesh point and vertical tension, so as to obtain the force and displacement of any mesh point on the soft tissue epidermis. The virtual experiment platform is built using a PHANTOM OMNI haptic hand controller and the 3D Max software, by which the simulation experiment of grasping the human abdomen is realized. The experimental results show that the proposed model has good visual interaction and real-time force feedback, which can meet the requirements of deformation simulation for soft tissue grasping in virtual surgery.

Keywords: Soft tissue grasping; genetic algorithm; BP neural network; surgical simulation

1  Introduction

With the rapid development of the information technology, the intelligent era is coming. Various emerging technologies and applications are increasingly widespread. The aggravation of the global aging population [1], the shortage of medical staff [2,3], and the tediouness of surgical training task [4] have brought severe challenges to the traditional surgical training. The traditional surgical training has such shortcomings as few specimens, long cycle, slow effect, and high cost; therefore, the intellectualization of the virtual surgical training system is needed urgently. The emergence of virtual reality technologies and haptic interactive equipments has brought new opportunities to the virtual surgery training system, and its development and application have promoted the intellectualization of virtual surgery [57].

The key of the virtual surgery system is designing virtual soft tissue deformation model. In virtual surgery, the soft tissue model needs to simulate various deformation operations and to feedback the deformation operation, so as to simulate the operation process of real soft tissue. Generally, it is common to grasp the soft tissue with surgical instruments during the surgical simulation. For example, it is often necessary to grasp the wound when removing tumors in soft tissue.

With the rapid development of artificial intelligence technology, many researchers apply deep learning, neural network, and other optimization algorithms to deformable soft tissue models. In order to achieve the requirements of real-time performance and accuracy of the virtual models, Zhong et al. [8] proposed a cellular neural method for interaction of soft tissue deformation by virtue of real-time computing of cellular neural networks, where the soft tissue deformation is executed based on the potential energy propagation [9]. According to the law of conservation of energy, the external force applied to soft tissue is regarded as equivalent potential energy and propagates in soft tissue through nerve propagation based on CNN [10]. De et al. [11] also studied the neural network technology based on machine learning for soft tissue deformation. The calculation process is divided into off-line and on-line stages, in which the response of the FEM model under specified displacement is calculated in advance in the off-line stage, and the coefficients of neurons are optimized by training the radial basis function network (RBFN); the deformation field is reconstructed by using the trained RBFN in the online stage. Lorente et al. [12] further explored the method based on machine learning for calculating soft tissue deformation, by which liver deformation simulation was realized during respiration. In this method, a supervised machine learning model based on deformation data is used to construct the mapping function of input variables to approximate the known model output. The mapping function constructed in the training stage can generate exact output for unknown input in following testing stage, that is, the mapping function can reconstruct the overall deformation of soft tissue. Therefore, the performance of the machine learning model highly depends on the training data and the selected learning algorithm. Tonutti et al. [13] further studied the ability of machine learning method in real-time modeling of soft tissue deformation and experimental results showed that the position error is less than 0.3 mm, which exceeds the general threshold of surgical accuracy in images.

In this paper, a novel variable model of soft tissue deformation is proposed to model and simulate soft tissue grasping deformation in interactive surgical simulation. In this model, the grasping deformation of soft tissue is studied by using BP neural network optimized by the genetic algorithm, and the varieties of soft tissue in the direction of tension are determined. The paper is organized as follows: Section 2 introduces the establishment of the soft tissue grasping deformation model and determination of the functional relationship between key points and stress. Section 3 describes the construction of the virtual surgery simulation system based on the model proposed in this paper. In Section 4, we analyze and compare the experimental results through adjusting the stress range of the soft tissue model and the load of the model.

2  System Design

The soft tissue grasping deformation model consists of four major components: the establishment of a rectangular coordinate system, the study of key points, the functional relationship between key points and stress, and the establishment of the model. The following sections describe in detail the major components of the soft tissue grasping deformation model.

2.1 Establishment of a Rectangular Coordinate System

Under the action of given tension F , when the virtual robot manipulator collides with any point on the virtual soft tissue epidermis with arbitrary shape, under the collision point, it is assumed that the plane of the soft tissue epidermis with arbitrary shape connected with the collision point is XY plane. The midpoint of the longest distance between any two points is regarded as the centroid point O . The centroid point O is considered as the origin point and the line passing through the origin O and perpendicular to the XY plane is considered as the Z -axis. Based on the above definition, the space coordinate system is established. It is assumed that m lines are parallel to the X -axis, and the distance between adjacent lines is 1 mm. The m lines include the X -axis , where the value range of m is m=2,3,4,,300 . It is assumed that n lines parallel to the Y -axis, and the distance between adjacent lines is 1 mm. The n lines include the Y -axis, where the value range of n is n=2,3,4,,300 . Through m lines are parallel to the X -axis and n lines are parallel to the Y -axis, which is shown in Fig. 1, the soft tissue epidermis without deformation can be divided into a series of regions, the number of the divided regions is s . The number of regular block areas and irregular edge areas are s1 and s2 , respectively.

images

Figure 1: Division of soft tissue epidermis

The soft tissue epidermis can be divided into a series of areas. which denotes as,

s=m(n1),m=2,3,4,,300,n=2,3,4,,300 (1)

where the number of regular block areas s1<s , s1 is not less than 80% of the total number.

Note that s is the number of the divided a series of regions in soft tissue epidermis with any shape. m is a straight line parallel to the X -axis and includes the X -axis. N is a line parallel to the Y -axis and includes the Y -axis. If the soft tissue epidermis with arbitrary shape is under tension, only the regular block area is considered in the deformation calculation, and the irregular edge area is mainly generated by the expansion internal force of the soft tissue epidermis.

2.2 Research on Key Points

The origin point O of the XYZ space coordinate system where the soft tissue epidermis with arbitrary shape is placed is the starting point. Parallel to the X -axis, the total number of rows is m . With the direction paralleling to the Y -axis as the column, the total number of columns is n . The soft tissue epidermis with arbitrary shape is divided into a series of regular block regions, with the intersection of rows and columns as the vertices.

The four vertices of each block area are marked with spacial coordinates. In this way, the coordinates of the vertices of each block area in the four quadrants can be expressed as A0,0(x0,0,y0,0,0) , A1,1(x1,1,y1,1,0) , A1,1(x1,1,y1,1,0) , A1,1(x1,1,y1,1,0)Am2,n2(xm2,n2,ym2,n2,0) , respectively, where m=2,3,4,,300 , n=2,3,4,,300 , and the distance di,j from each vertex to the origin point O can be expressed as:

di,j=(Xi,jX0.0)2+(Yi,jY0.0)2+(Zi,jZ0,0)2 (2)

where i and j represent the abscissa and ordinate of the coordinate system, respectively, i=2,3,4,...,300 , and j=2,3,4,...,300 . X,Y,Z are the vertex coordinates. If the vertices of each block mesh are subjected to a slight tension in the Z -axis direction at the origin, the displacement changes of the mesh vertices in the four quadrants in the Z direction are ΔZ0,0 , ΔZ1,1 , ΔZ1,1 , ΔZ1,1 , ΔZ1,1 ,… ΔZm2,n2 , ΔZm2,n2 , ΔZm2,n2 , and ΔZm2,n2 , respectively. If the tension Fi is applied at the origin O , the displacement changes of all points on the circle with the same distance to the origin are the same, so when the coordinate points are selected as the key points, only the XY plane of the cross-sectional coordinate system of the soft tissue epidermis is opted. The vertices on Y=X in the first quadrant can be regarded as key points, which can be expressed as A1,1,A1,2,,An,n , where n=2,3,4,,300 , as shown in Fig. 2.

images

Figure 2: Deformation of key points in spacial coordinate system

2.3 The Functional Relationship Between Key Points and Stress

If any key point on the line Y=X in the first quadrant of XY plane is subjected to the tension Fi in different Z directions at the center of the circle, the change in the vertical direction of Z axis can be measured by experiments, thus can establish the database of key points. The key points are selected according to the vertex of Y=X in the first quadrant of XY plane of the cross-section coordinate system where the soft tissue epidermis is located, that is the vertical displacement variation of any point and the tension Fi in the Z -axis direction applied at point O . The distance from the key point to the origin O is di . The database can be represented as Eq. (3):

[d0,0ΔZ0.0d1,1ΔZ1,1d2,2ΔZ2,2......dn1,n1ΔZn1,n1dn,nΔZn,n] (3)

In order to study the relationship between the Z -axis change and the tension Fi , the micro-cylindrical compact coil spring is used as the deformation model to study the relationship between the Z -axis change ΔZ and the tension Fi . The spring model accords with the elastic characteristics of soft tissue epidermis and is suitable for analysis. If any point on the soft tissue epidermis is stressed, the vertical spring can be simulated as the micro-tension in the direction of the vertical spring cross-section. Through the spring model, the following relationship between the magnitude of tension Fi and ΔZ in the Z -axis direction on the soft tissue epidermis can be obtained as Eq. (4):

{Fi=k|ΔZ|k=NR44tr3 (4)

In Eq. (4), spring tension is proportional to the elongation of deformation, that is, the tension Fi received by the soft tissue epidermis is proportional to the change ΔZ in the Z -axis direction. k is the spring stiffness coefficient, N is the shear modulus, R is the radius of the spring, t is the number of turns of the spring, and r is the radius of the spring cylinder.

The database is used to help the genetic algorithm optimize the BP neural network. Through the BP neural network, the change amount ΔZi of any point Ai on the soft tissue epidermis on the Z -axis and the tension Fi in the Z -axis direction at the origin O can be obtained. There is a non-linear functional relationship between the spatial distances di,j from the vertex of the mesh divided on the soft tissue epidermis to the origin O . The genetic algorithm is used to optimize the initial weights and thresholds of the BP neural network so that the optimized BP neural network can better predict the output, that is, the BP neural network optimized by the genetic algorithm can obtain the exact functional relationship between ΔZi,j , Fi and di,j , as is shown in Fig. 3. The genetic algorithm to optimize BP neural network includes five elements, i.e., population initialization, fitness function, selection function, crossover operation, and mutation operation. The detailed flow chart is shown in Fig. 4.

images

Figure 3: Schematic diagram of neural network training

images

Figure 4: Flow chart of the genetic algorithm

2.4 Model Construction

If a tension Fi in the Z -axis direction is employed at the origin point of the soft tissue epidermis, the change amount ΔZ in the Z -axis direction of any coordinate point on the soft tissue epidermis can be obtained by optimizing the BP neural network through the genetic algorithm and can be obtained through the micro spring model. The force of each point in the Z -axis direction is obtained, so that a planar soft tissue epidermal mesh model can be established and the change ΔZ in the Z -axis direction of any point on the soft tissue epidermis can be obtained.

In order to test the usability of the virtual platform, operate the PHANTOM OMNI hand controller to grasp the modeled lower abdomen. The process of simulated grasping operation is shown in Fig. 5. Experiments show that this model is effective, can grasp the modeled soft tissues, and can provide visual and tactile feedback.

images

Figure 5: Grasping process diagram

In the grasping process, the color map is used to display the deformation characteristics of the soft tissue, which more intuitively and clearly shows the deformation of the soft tissue under local stress. The colors from blue to red indicate the strength of the force, blue indicates no force, and red indicates the area that is deformed by the force, as shown in Fig. 6.

images

Figure 6: Virtual soft tissue stress and deformation map ( K is the deformation process, the deformation of the four models during the deformation process)

3  System Realizations

The virtual surgery simulation system is composed of three modules, including the input module, information processing module, and output module. The input module mainly uses force-tactile interaction devices to input the tension force, including the magnitude, direction, and duration of the force. The information processing module is a soft tissue model programmed by 3D MAX software, which relies on the computer operating system for processing, including force signal processing and model shape processing. The output module is for feeding back the information to the operator after the signal processing by the input module, including the feedback force from the force-tactile interactive device and the deformation of the model displayed on the displayer. The pipeline of the simulation test is shown in Fig. 7.

images

Figure 7: Pipeline of the simulation test

In order to realize the algorithm, a virtual simulation platform is first established. Our platform consists of a computer and a haptic interaction facility called PHANTOM OMNI. The computer is based on Windows 10 with an Intel (R) Xeon (R) CPU, E5-1650 v3 @ 3.5 GHz processor, and NVIDIA GeForce GT 720M graphics. The simulation is carried out on VC++ 2019 and 3DS MAX 2019 software with OpenGL graphics libraries. The PHANTOM OMNI allows the operators to touch and operate on the virtual object simulated by our method. The experimental environment is shown in Fig. 8.

images

Figure 8: Experiment environment

A three-dimensional model of 65536 lower abdomens and forceps is used, which is shown in Fig. 9. Import the 3D model into the 3D MAX software, and render the entire 3D model surface with the colorful lower abdomen CT data obtained from the First Affiliated Hospital of Southeast University, including 36,542 color rendering triangles. The rendering effect is shown in Fig. 10.

images

Figure 9: Three-dimensional model

images

Figure 10: Three-dimensional rendering effect

3.1 Collision Detection Validation

The top-K algorithm in the bounding box hierarchy method has better performance and high real-time detection accuracy [14,15], therefore, the study selects it as the collision detection method. The basic idea of the bounding box hierarchy method is to use a slightly larger bounding box and more simple geometric structure characteristics than the simulation model to approximately represent the complex simulation object, and gradually approach the object by constructing a tree-like hierarchy until the geometric characteristics of the object is completely obtained. If the collision of the object is detected, the bounding box is intersected. Since finding the intersection of the object is easy by the intersection ratio of the bounding box, it is possible to quickly exclude many disjoint objects. The overlapping parts of the bounding boxes are further tested for intersection, thereby speeding up the algorithm.

4  Experimental Results and Analysis

4.1 Simulation Deformation Time

Under the same experimental conditions, 50 grasping operations were performed on the virtual abdomens. The virtual abdomens are constructed by the model proposed in this paper and the spring-mass model [16], respectively. Additionally, we also select fresh living soft tissues for 50 real grasping operations, and the deformation time of the 50 grasping operations is shown in Fig. 11, where the deformation time is from the beginning of the grasping operation on the abdomen to the end of the model deformation. The blue grid points denote the results of the proposed model in this paper, green grid points denote the results based on the Spring-mass model and red grid points denote the results based on living soft tissues.

images

Figure 11: Response time for grasping

4.2 Stress Range of Soft Tissue Model

The soft tissue has the maximum stress range during the deformation process after being stressed. In order to make the deformation of this model more accurate, it is necessary to determine its maximum stress. This paper used real living soft tissues to grab and obtain real data, which is shown in Fig. 12. Based on the obtained real data, we improved the parameter settings of the proposed model.

images

Figure 12: Stress change graph over time

4.3 Stability of Deformation

The stability of model deformation is an important metric of the performance of the model. In order to verify the stability of the proposed model, the traditional spring-mass model [16] and the proposed model are simulated, respectively. The proposed model achieves stable at about 50ms for the soft tissue deformation of 12 cm, but traditional spring-mass model has a slight increase after 75 ms without reaching a steady state. As shown in Fig. 13, the proposed model has better stability after the grasping operation.

images

Figure 13: Displacement change graph

4.4 Model Load Comparison

The finite element model with better load performance [1721] and the model proposed in this paper are simulated, respectively, and the deformation results are shown in Fig. 14. The load-displacement of the local deformation of the proposed model is also in good agreement with the finite element model, and the maximum deviation between the load and displacement is about 5%. The experimental results show that the proposed model is excellently similar to the finite element model in the load performance.

images

Figure 14: Load-displacement behaviors of the proposed model and FEM model

5  Conclusions

This study proposed a soft tissue grasping deformation model to simulate the grasping deformation process on the human abdomen soft tissue by virtual human-computer interaction. Using 3DS MAX 2019 with OpenGL graphics libraries and VC++ 2019 software, the abdominal soft tissue model training system for grasping operation was built through the PHANTOM OMNI force-tactile interactive device. The proposed model combines the displacement and deformation of any point of the irregular cross-section elastic cylinder under grasping action, where the BP neural network optimized by the genetic algorithm is used to calculate the deformation of the elastic tissues intuitively, accurately and quickly. A real-time deformation simulation system was constructed to improve the fidelity of virtual force tactile interaction. The real-time performance and accuracy of the proposed model are verified by comparative experiments. The experimental results show that the proposed model has better stability and on-site perception.

Funding Statement: This work was supported, in part, by the Natural Science Foundation of Jiangsu Province under Grant Numbers BK20201136, BK20191401; in part, by the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD) fund; in part, by the Collaborative Innovation Center of Atmospheric Environment and Equipment Technology (CICAEET) fund; NUIST Students’ Platform for Innovation and Entrepreneurship Training Program; NUIST Undergraduate Excellent Graduation Design (Thesis) Support Program Project.

Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.

References

  1. P. Juha and L. Puustjärvi, “Towards web-assisted self-care in developing countries: A challenge for health posts and pharmacies,” in Proc. IST-Africa Week, South Africa, pp. 1–10, 2016.
  2. D. Wang, J. Jiao, Y. Zhang and X. Zhao, “Computer haptics: Haptic modeling and rendering in virtual reality environments,” Journal of Computer Aided Design & Computer Graphics, vol. 28, no. 6, pp. 881–895, 2016.
  3. X. R. Zhang, P. P. Wang, W. Sun and N. I. Badler, “A novel twist deformation model of soft tissue in surgery simulation,” Computers Materials & Continua, vol. 55, no. 2, pp. 297–319, 2018.
  4. X. Shao, X. Nie and B. Wang, “GPU-based real-time 3D visual hull reconstruction and virtual-reality interaction,” Journal of Computer Aided Design & Computer Graphics, vol. 29, no. 1, pp. 52–61, 2017.
  5. M. Freutel, H. Schmidt, L. Dürselen, A. Ignatius and F. Galbusera, “Finite element modeling of soft tissues: material models, tissue interaction and challenges,” Clinical Biomechanics, vol. 29, no. 4, pp. 363–372, 2014.
  6. X. R. Zhang, X. F. Yu, W. Sun and A. G. Song, “Soft tissue deformation model based on marquardt algorithm and enrichment function,” Computer Modeling in Engineering & Sciences, vol. 124, no. 3, pp. 1131–1147, 2020.
  7. X. R. Zhang, J. L. Duan, J. Liu and N. I. Badler, “An integrated suture simulation system with deformation constraint under a suture control strategy,” Computers Materials & Continua, vol. 60, no. 3, pp. 1055–1071, 2019.
  8. Y. Zhong, B. Shirinzadeh, G. Alici and J. Smith, “A cellular neural network methodology for deformable object simulation,” IEEE Transactions on Information Technology in Biomedicine, vol. 10, no. 4, pp. 749–762, 2006.
  9. J. Zhang, Y. Zhong, J. Smith and C. Gu, “Energy propagation modeling of nonlinear soft tissue deformation for surgical simulation,” Simulation, vol. 94, no. 1, pp. 3–10, 2018.
  10. J. Zhang, Y. Zhong, J. Smith and C. Gu, “Neural dynamics-based poisson propagation for deformable modelling,” Neural Computing and Applications, vol. 31, no. 2, pp. 1091–1101, 2019.
  11. S. De, D. Deo, G. Sankaranarayanan and V. S. Arikatla, “A physics-driven neural networks-based simulation system (phynness) for multimodal interactive virtual environments involving nonlinear deformable objects,” Presence: Teleoperators and Virtual Environments, vol. 20, no. 4, pp. 289–308, 20
  12. D. Lorente, F. Martínez-Martínez, M. J. Rupérez, M. A. Lago, M. Martínez-Sober et al., “A framework for modelling the biomechanical behaviour of the human liver during breathing in real time using machine learning,” Expert Systems with Applications, vol. 71, no. 1, pp. 342–357, 2017.
  13. M. Tonutti, G. Gras and G. Z. Yang, “A machine learning approach for real-time modelling of tissue deformation in image-guided neurosurgery,” Artificial Intelligence in Medicine, vol. 80, no. 8, pp. 39–47, 2017.
  14. S. Gottschalk, M. C. Lin and D. Manocha, “OBBTree: A hierarchical structure for rapid interference detection,” in Proc. SIGGRAPH, New Orleans, USA, pp. 171–180, 1996.
  15. M. L. Yiu and N. Mamoulis, “Efficient Processing of Top-k Dominating Queries on Multi-Dimensional Data,” in Proc. VLDB, Vienna, Austria, pp. 483–494, 2007.
  16. R. Blickhan, “The spring-mass model for running and hopping,” Journal of Biomechanics, vol. 22, no. 11–12, pp. 1217–1227, 1989.
  17. X. R. Zhang, H. L. Wu, W. Sun and S. K. Jha, “A fast and accurate vascular tissue simulation model based on point primitive method,” Intelligent Automation & Soft Computing, vol. 27, no. 3, pp. 873–889, 2021.
  18. J. Goldak, A. Chakravarti and M. Bibby, “A new finite element model for welding heat sources,” Metallurgical and Materials Transactions B, vol. 15, no. 2, pp. 299–305, 1984.
  19. A. A. Ahmed and B. Akay, “A survey and systematic categorization of parallel k-means and fuzzy-c-means algorithms,” Computer Systems Science and Engineering, vol. 34, no. 5, pp. 259–281, 20
  20. Z. Wang, Y. Huang and F. Xiong, “Three-dimensional numerical analysis of blast-induced damage characteristics of the intact and jointed rockmass,” Computers Materials & Continua, vol. 60, no. 3, pp. 1189–1206, 2019.
  21. X. R. Zhang, X. F. Yu, W. Sun and A. Song, “An optimized model for the local compression deformation of soft tissue,” KSII Transactions on Internet and Information Systems, vol. 14, no. 2, pp. 671–686, 2020.
images This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.