Submit

Login Login

Register Register

Home / Journals / CMC / Online First / doi:10.32604/cmc.2026.076083
Journal Menu
Special Issues
Table of Content

All issues

Online First

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Open Access

ARTICLE

ComAlign: A Benchmark Aligning Natural Language with Operating System Commands

Shasha Li, Bin Ji*, Xiaodong Liu, Jun Ma, Jie Yu*
College of Computer Science and Technology, National University of Defense Technology, Changsha, China
* Corresponding Author: Bin Ji. Email: email; Jie Yu. Email: email

Computers, Materials & Continua https://doi.org/10.32604/cmc.2026.076083

Received 13 November 2025; Accepted 29 December 2025; Published online 22 January 2026

Abstract

Aligning natural language with operating system (OS) commands allows users to perform complex computer tasks through simple natural language descriptions. However, due to the complex nature of natural language, it still remains challenging to achieve precise alignment. In this paper, we present ComAlign, a Chinese benchmark dataset that pairs Chinese natural language descriptions with corresponding OS commands. ComAlign covers a broad range of 82 distinct OS command types with a total of 1811 natural language descriptions. We elaborate on the construction of ComAlign and construct three baselines to evaluate the alignment accuracy on ComAlign. Experimental results show that even advanced large language models struggle with certain ambiguously phrased OS commands. Specifically, the best performing baseline achieves 46.9% alignment accuracy. We demonstrate that ComAlign is collected from real-world application scenarios, making it particularly suitable for developing and benchmarking intelligent OS and agent systems that support user-machine interactions through natural language.

Keywords

Large language model; language-command alignment; artificial intelligence operating system

  • 83

    View

  • 21

    Download

  • 0

    Like

Related articles
Copyright© 2026 Tech Science Press
© 1997-2026 TSP (Henderson, USA) unless otherwise stated

Share Link