Open Access

ARTICLE

EDSM-Based Binary Protocol State Machine Reversing

Shen Wang1,*, Fanghui Sun1, Hongli Zhang1, Dongyang Zhan1,2, Shuang Li3, Jun Wang1
1 School of Cyberspace Science, Harbin Institute of Technology, Harbin, 150001, China
2 The Ohio State University, Columbus, 43202, USA
3 Guangzhou University, Guangzhou, 510006, China
* Corresponding Author: Shen Wang. Email:

Computers, Materials & Continua 2021, 69(3), 3711-3725. https://doi.org/10.32604/cmc.2021.016562

Received 05 January 2021; Accepted 12 July 2021; Issue published 24 August 2021

Abstract

Internet communication protocols define the behavior rules of network components when they communicate with each other. With the continuous development of network technologies, many private or unknown network protocols are emerging in endlessly various network environments. Herein, relevant protocol specifications become difficult or unavailable to translate in many situations such as network security management and intrusion detection. Although protocol reverse engineering is being investigated in recent years to perform reverse analysis on the specifications of unknown protocols, most existing methods have proven to be time-consuming with limited efficiency, especially when applied on unknown protocol state machines. This paper proposes a state merging algorithm based on EDSM (Evidence-Driven State Merging) to infer the transition rules of unknown protocols in form of state machines with high efficiency. Compared with another classical state machine inferring method based on Exbar algorithm, the experiment results demonstrate that our proposed method could run faster, especially when dealing with massive training data sets. In addition, this method can also make the state machines have higher similarities with the reference state machines constructed from public specifications.

Keywords

Network security; protocol state machine; EDSM algorithm; protocol reverse engineering; protocol analyzing

Cite This Article

S. Wang, F. Sun, H. Zhang, D. Zhan, S. Li et al., "Edsm-based binary protocol state machine reversing," Computers, Materials & Continua, vol. 69, no.3, pp. 3711–3725, 2021.



This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 819

    View

  • 501

    Download

  • 0

    Like

Share Link

WeChat scan