Toward Trustworthy Chinese Large Language Models: A Multi-Dimensional Evaluation of Toxicity, Bias, and Robustness

1  Introduction

Large language models (LLMs) have rapidly become a transformative foundation across computational science, intelligent systems, and natural language processing (NLP) [1–3]. By leveraging large-scale pretraining and instruction tuning, LLMs now demonstrate remarkable capabilities in question answering, text summarization, reasoning, and code generation [4]. Motivated by the success of ChatGPT, a growing number of Chinese LLMs have been developed and open-sourced [5–9], rapidly expanding the reach of LLM-powered technologies to Chinese-speaking populations and applications.

However, alongside these advances, serious concerns about model security, robustness, and responsible AI deployment have emerged. LLMs are pretrained on large-scale, weakly supervised web corpora that inevitably contain harmful, toxic, and biased content [10]. As illustrated in Fig. 1, Chinese LLMs can generate toxic outputs both when encountering explicitly toxic prompts (Toxic-to-Toxic, T2T) and, more alarmingly, when responding to seemingly benign inputs (Non-Toxic-to-Toxic, NT2T). Furthermore, the lack of human supervision during data preprocessing means that models may silently absorb discriminatory patterns and social stereotypes [11]. These vulnerabilities pose direct security risks in real-world deployments, where LLMs interact with diverse users across sensitive domains. Beyond toxicity and bias, LLMs are also susceptible to adversarial threats such as clean-label backdoor attacks [12], and dedicated defense mechanisms such as LoRA-based expert fusion [13] have been proposed to mitigate such risks. With Chinese LLMs being continuously proposed and open-sourced [14], systematically evaluating their potential security risks has become imperative [15].

Figure 1: Examples of human inputs and model responses, where Toxic to Toxic (T2T), Non-Toxic to Non-Toxic (NT2NT), Toxic to Non-Toxic (T2NT), and Non-Toxic to Toxic (NT2T) denote different types of input-response pairs, respectively. The text inside the red and blue dialog boxes indicates toxic and non-toxic content, respectively. The examples are verbatim model outputs reproduced solely for research documentation and do not reflect the authors’ views.

Despite growing recognition of these risks, existing evaluation efforts remain limited in scope. Most studies focus on English models and datasets, leaving the robustness and security of Chinese LLMs largely uncharted [11,15–17]. Moreover, prior benchmarks concentrate on capability metrics rather than safety-critical dimensions such as toxicity, bias, and avoidance robustness. This gap motivates a dedicated, multi-dimensional evaluation framework tailored to Chinese LLMs.

Our work addresses this gap by investigating three core security and robustness questions:

•   Question 1 (Security—Toxicity): LLMs may internalize harmful values from pretraining data, generating offensive or dangerous content. How toxic are publicly available Chinese LLMs, and under what prompt conditions does toxicity emerge?

•   Question 2 (Robustness—Bias): LLMs may encode discriminatory associations toward sensitive attributes such as gender, race, and region. How severe is the bias in Chinese LLMs, and how does it vary across model families and checkpoints?

•   Question 3 (Reliability—Avoidance): Trustworthy LLMs should reliably recognize and refuse toxic inputs. Can Chinese LLMs effectively avoid generating harmful content, and what are the limitations of their built-in safety mechanisms?

To answer these questions, we propose TrustEval, a dataset- and model-agnostic evaluation framework for the multi-dimensional security and robustness assessment of Chinese LLMs. TrustEval covers three evaluation dimensions: toxicity measurement, bias measurement, and avoidance rate measurement. We apply TrustEval to nine representative Chinese LLMs, together with ChatGPT and GPT-4 as general-purpose comparison models, across three datasets—COLD [18], HateXplain [19], and ToxicSpans [20]—spanning both Chinese and translated English corpora with fine-grained sensitive-attribute annotations. This enables a systematic, multi-dimensional security evaluation of Chinese LLMs at scale.

Key findings. Our empirical study yields the following insights:

•   Toxicity: The evaluated models can generate toxic content under the tested prompting conditions. LLaMA2-Chinese-7B responds to toxic prompts with a 35.36% probability, while MOSS-7B produces toxic responses even to non-toxic inputs (4.78%). These results reveal notable safety weaknesses in the evaluated models.

•   Bias: The evaluated models exhibit systematic bias toward sensitive attributes inherited from unsupervised pretraining. LLaMA2-Chinese-7B’s toxic response rate for race-related prompts is 22.46 percentage points higher than for gender-related prompts. The evaluated checkpoints show no consistent reduction in bias across model families and sizes, revealing a structural limitation of current training paradigms.

•   Avoidance Robustness: The built-in safety and avoidance mechanisms of most Chinese LLMs are insufficient for robust security enforcement. Although some evaluated checkpoints show stronger avoidance than others, the avoidance capability of all evaluated models remains far below the standards required for responsible AI deployment.

2  Related Works

2.1 Evaluating Large Language Models

LLMs have demonstrated powerful capabilities for understanding natural language and solving complex tasks (through text generation), and the field of AI research is being revolutionized by the rapid development of LLMs. In order to evaluate the effectiveness and superiority of LLMs, several studies have empirically evaluated and analyzed LLMs using a large number of tasks and benchmark datasets [1–3]. Despite the tremendous progress, there are still limitations in these superior LLMs, such as generating toxic responses or potential biases in certain contexts [3]. With the broad applications of LLMs, more attention has been paid to toxicity and bias evaluation. Wang et al. [4] evaluated the robustness of ChatGPT under adversarial and out-of-distribution conditions. Bommasani et al. [21] proposed the Holistic Evaluation of Language Models (HELM) to improve the transparency of LLMs, including both toxicity and bias evaluations. Deshpande et al. [10] presented a large-scale, systematic toxicity analysis of ChatGPT-generated text. Cheng et al. [22] introduced Marked Personas, a prompt-based method to measure bias in LLMs for intersectional demographic groups.

2.2 Chinese Large Language Models

Recently, Chinese LLMs have gained increasing attention in academia and industry, where series of Chinese LLMs have been proposed. The CPM family [23] leads the way with the release of the Chinese LLMs. With the proposal of Pangu-α [9], the power of the Chinese language model was increased by pushing the model size to over 200B. The EVA family [7] enhances the conversational capabilities of the model. MOSS [8] is the first open-source Chinese LLM similar to ChatGPT. ChatGLM [14] is a bilingual (English and Chinese) model based on the general language model structure. Qwen [5] is based on the Transformer architecture. In addition, BELLE [24], developed using BLOOM and LLaMA [25], is an open-source Chinese instruction-tuned model; Firefly1 is an open-source instruction-tuning framework; and Baichuan [6] is an independently trained Chinese LLM.

2.3 Evaluating Chinese Large Language Models

The majority of the recently proposed Chinese evaluation benchmarks focus on standardized examination topics, such as college entrance exams and civil servant exams [26,27]. For example, C-Eval [26] contains 52 disciplines divided into four general categories: STEM, Social Science, Humanities, and Other. CMMLU [27] measures massive multitask language understanding in Chinese across 67 subjects.

Regarding Chinese models’ toxicity and bias evaluation, Zhao et al. [11] constructed CHBias, a dataset for evaluating and mitigating biases in Chinese conversational models. Wan et al. [28] proposed an automated framework for identifying and measuring social biases in commercial conversational AI systems. However, existing evaluation methods rely on specialized settings for experiments, leading to difficulties in applying the evaluation to a broader range of models, tasks, and datasets. Moreover, they fail to comprehensively evaluate open-source Chinese LLMs from the perspectives of security and robustness.

2.4 Controllable and Trustworthy Large Language Models

Beyond evaluation, a complementary line of research seeks to directly control LLM behavior at inference time. Representation engineering provides a top-down view of monitoring and manipulating high-level concepts encoded in model activations [29], while activation-steering methods inject steering vectors into intermediate layers to shift generations toward desired attributes without retraining [30]. Inference-time intervention identifies and shifts truthfulness-related attention heads to elicit more reliable answers [31].

From a control-theoretic perspective, Bhargava et al. [32] model prompting as control of a discrete stochastic dynamical system and analyze the reachability of desired output tokens. Nosrati et al. [33] frame the broader interaction between LLMs and control as a bidirectional continuum: in one direction, prompts and LLM reasoning can support control-system design, controller synthesis, and engineering workflows; in the other, control principles can guide prompt/input optimization, parameter editing, activation-level interventions, and safety filters to steer model trajectories toward desired objectives. Karnik and Bansal [34] apply this perspective to safety alignment by using latent-space backward reachability for early detection of unsafe trajectories and minimally restrictive inference-time steering. This progression connects controllability, reachability, feedback, and alignment rather than treating prompt design as an isolated heuristic. TrustEval does not itself implement a controller; instead, its toxicity, attribute-level bias, and avoidance measurements provide observable safety outcomes that can be used to assess whether prompt- or control-based interventions achieve their intended objectives.

Table 1 positions TrustEval relative to representative trustworthiness evaluation efforts [11,15,21,28,35]. Unlike English-centric resources such as RealToxicityPrompts [35] and HELM [21], and unlike Chinese benchmarks centered on multiple-choice safety knowledge [15] or bias mitigation [11], TrustEval offers a unified generation-based protocol covering toxicity, attribute-level bias, and refusal (avoidance) behavior for Chinese LLMs, together with re-annotated Chinese evaluation data.

3  TrustEval Framework

To comprehensively understand and evaluate the potential toxicity and bias in Chinese LLMs, we propose a new evaluation framework TrustEval. Specifically, we design a simple yet efficient toxicity and bias measurement method. In this section, we illustrate the evaluation subtasks and then give the corresponding evaluation metrics for each task.

3.1 Toxicity Measurement

Toxicity measurement aims to detect what kinds of questions or prompts encountered by LLMs will answer toxic or aggressive responses.

Prompt-Response Pair Collection. To measure toxicity in LLMs, a common strategy is to analyze input prompts and model-generated response content [35,36]. Unfortunately, existing datasets are not available for toxicity measurements in dialog-based models. Therefore, we constructed prompt-response pairs using publicly available datasets. To obtain prompt-response pairs, we provide the dataset to pre-trained language models, instructing them to respond. This allows us to collect prompt-response pairs, where prompts come from the dataset and responses come from the language models.

Toxicity Metrics. In contrast to methods that use Google’s Perspective API2, we use the publicly released thu-coai/roberta-base-cold sequence-classification checkpoint3, which is trained on the COLD Chinese offensive-language dataset [18]. For clarity, we refer to this BERT-family classifier as the COLD-based detector (labeled “COLDetector” in Fig. 2); it is our implementation designation, not the name of a model introduced in the COLD paper. For each prompt and response, the classifier assigns a toxicity probability from 0 to 1. Following [36], a prompt or response is labeled toxic when its score is ≥0.5.

Figure 2: The framework of TrustEval, illustrating the pipeline from human-computer interaction to toxicity measurement, bias measurement, and avoidance rate measurement.

Metrics of Toxicity Response Measurement. Toward facilitating quantification, prompt-response pairs were divided into four categories based on the toxicity of the prompt and response: Toxic to Toxic (T2T), Non-Toxic to Non-Toxic (NT2NT), Toxic to Non-Toxic (T2NT), and Non-Toxic to Toxic (NT2T). We define total toxicity (TT) as the proportion of all responses labeled toxic. Because T2T and NT2T are mutually exclusive categories with toxic responses, TT=T2T+NT2T when all three quantities are expressed as percentages of the full evaluation set.

3.2 Bias Measurement

The corpus, training methods, and algorithms involved in the implementation of the various stages of machine learning may have biases that lead to unfair model predictions. Bias is a major source of discrimination and unfairness. However, systematic measurement of bias in Chinese LLMs remains limited.

Metrics of Bias Measurement. Traditional fairness metrics for classification quantify disparities between groups, such as demographic parity (differences in positive prediction rates), equal opportunity (differences in true positive rates), and equalized odds (differences in both true- and false-positive rates) [16]. In contrast, we quantify the output toxicity of LLMs across sensitive-attribute classes. Specifically, we calculate the standard deviation of total toxicity across classes. Formally, let TTc denote the total toxicity of a model on sensitive-attribute class c∈𝒞 (e.g., gender, region, race). We define the quantity measured by TrustEval as attribute-level toxicity disparity, and report three complementary statistics: the standard deviation σ({TTc}), the max–min gap Δ=maxcTTc−mincTTc, and the ratio ρ=maxcTTc/mincTTc. We note that this is an observational, outcome-level fairness measure: it captures disparities in toxic-output rates across attributes, but does not measure implicit associations (e.g., embedding- or likelihood-based probes), context-dependent bias, or causal bias under counterfactual attribute substitution [16]; we discuss these boundaries in Section 5.5.

3.3 Avoidance Rate Measurement

Aligning LLMs with human values is crucial when building trustworthy AI. In the avoidance rate measurement, we primarily intended to measure how these models responded to toxic input prompts.

Avoidance Measurement. To evaluate the practical performance of the models in question with respect to toxic prompt avoidance, we design experiments to compare them to baseline models. Specifically, we adopt the COLD [18] Chinese offensive dataset and randomly select 200 toxic samples per sensitive-attribute category (region, race, and gender), i.e., 600 prompts per model, and obtain the corresponding responses. With 8 open-source models evaluated, a total of 4800 prompt–response pairs are scored.

Metrics of Avoidance Rate Measurement. After obtaining the prompt-response pairs described in Section 3.1, we use ChatGPT as a binary safety judge. For each pair, the judge outputs ai=1 when the response recognizes the harmful prompt and avoids producing harmful content, for example by refusing, warning, or safely redirecting; it outputs ai=0 when the response complies with the harmful prompt, generates harmful content, or repeats it without a safety intervention. For a sensitive-attribute category c containing Nc=200 prompts, the avoidance rate is ARc=100×Nc−1∑i=1Ncai. We report this percentage with a 95% bootstrap confidence interval based on 1000 resamples. The exact judge prompt is provided in the public repository; Fig. 2 summarizes the evaluation pipeline.

4  Experimental Settings

In this section, we conduct a comprehensive study on the toxicity and bias of LLMs to measure the presence of toxicity and inherent bias in the models. The dataset and its corresponding evaluation tasks are shown in Table 2.

4.1 Datasets and Tasks

To comprehensively measure the toxicity and bias of LLMs, we selected the following three datasets.

COLD [18] is the Chinese offensive language dataset. COLD contains 37,480 comments with binary offensive labels covering a variety of topics in terms of race, gender, and region.

HateXplain [19] is a benchmark hate speech dataset that covers multiple aspects of the topic. The dataset contains 20,148 annotated posts in English. Each post is annotated at three levels: (1) category (hateful, offensive, or normal), (2) target community, and (3) token-level rationales identifying the spans that support the annotation.

ToxicSpans [20] contains 11,035 annotated posts of toxic spans in English. The dataset uses posts (comments) from the publicly available Civil Comments dataset, which already provides the entire post-toxicity annotation. The dataset covers offensive language phenomena such as insults, hate speech, identity attacks, or profanity.

4.2 Translation and Annotation of English Datasets

As the available Chinese toxicity datasets are scarce, we adopted two English toxicity datasets: HateXplain and ToxicSpans. We translated them into Chinese and re-annotated them. First, for the aforementioned datasets, we randomly sampled 10,000 samples (6000 from HateXplain and 4000 from ToxicSpans). The sampled posts were translated by machine translation (DeepL), followed by human post-editing by three bilingual annotators to correct mistranslations and preserve offensive intent where translatable. Second, the datasets were labeled into two categories, toxic content and non-toxic content, by manual annotation, conducted in the Chinese context rather than inherited from the original English labels: a sample is labeled toxic only if it is perceived as toxic in Chinese, which absorbs shifts in toxicity intensity introduced by translation. Moreover, a fine-grained annotation was implemented: HateXplain was divided into four subsets (gender, region, politics, and others), and ToxicSpans into three categories (insults, obscenity, and others).

Annotation protocol. A pool of five annotators participated (graduate students in computer science and linguistics, all native Chinese speakers; three female, two male). All annotators were briefed with a written guideline document (released in our repository), which defines toxic content as language that attacks, demeans, or expresses hatred toward an individual or group on the basis of a sensitive attribute, or that contains profanity or explicit insults, with decision rules and examples for each fine-grained category. Each sample was independently labeled by three annotators, and the final label was determined by majority voting; samples with persistent disagreement were adjudicated by a senior annotator (310 samples, 3.1%). The inter-annotator agreement is Fleiss’ κ = 0.74 (HateXplain) and 0.71 (ToxicSpans) on the binary toxic/non-toxic label, and κ = 0.66 and 0.63 on the fine-grained categories, indicating substantial agreement. In addition, a bilingual annotator audited 500 translated samples: 91.4% preserved the original toxicity polarity, and the mismatched cases were corrected during re-annotation. Table 3 summarizes the protocol.

4.3 Baselines

We evaluated nine Chinese LLMs together with ChatGPT and GPT-4 as general-purpose comparison models.

•   Eva-2.8B [7]: A large-scale pre-trained open-domain Chinese dialog model with 2.8 billion parameters4.

•   Pangu-α-2B [9]: The approximately 2.6-billion-parameter variant of the autoregressive Pangu-α Chinese language-model family; the family also includes larger 13B and 200B variants5.

•   BELLE-7B [24]: A Chinese pre-trained language model based on BLOOM and LLAMA, optimized for Chinese and fine-tuned with the help of ChatGPT6.

•   MOSS-7B [8]: A conversational language model supporting Chinese-English bilingualism and multiple plugins7.

•   ChatGLM-6B [14]: A conversational language model supporting bilingual QA with 6.2 billion parameters8.

•   Baichuan-7B [6]: The evaluated checkpoint is based on Baichuan-7B and was instruction-tuned with QLoRA using the Firefly framework9.

•   Baichuan2-13B [6]: A large-scale language model based on 2.6 trillion tokens trained from scratch10.

•   LLAMA2-Chinese-7B: A large-scale language model that supports Chinese-English bilingualism, and the model evaluated in this paper is pre-trained on an expanded Chinese vocabulary11.

•   Qwen-7B [5]: A hyper-scale language model introduced by AliCloud with features such as multiple rounds of dialog, logical reasoning, multi-modal comprehension and multilingual support12.

•   ChatGPT13: A language model applied to conversational scenarios, obtained by using reinforcement learning with human feedback fine-tuned in GPT-3.5.

•   GPT-4 [3]: A large multimodal model which is fine-tuned using Reinforcement Learning from Human Feedback14.

4.4 Implementation Details

For all open-source models, we use nucleus sampling with temperature 0.7, top-p = 0.9, and a maximum generation length of 512 tokens; each prompt is evaluated over three runs with random seeds {42, 43, 44}, and we report the averaged results. For API-based models, we use gpt-3.5-turbo-0125 (ChatGPT) and gpt-4-0613 (GPT-4), accessed on 2026-03-12. The ChatGPT judge for avoidance scoring uses temperature 0 to maximize scoring determinism. All open-source experiments were conducted on 4 × NVIDIA A100 (80 GB) GPUs. The exact prompt templates, chat formats, and scoring scripts are released in our public repository (https://github.com/fenffef/TrustEval); Table 4 summarizes the key settings.

5  Experimental Results and Analysis

5.1 Toxicity in Chinese Large Language Models

5.1.1 Quantitative Analysis

In this section, we quantify the toxic response of each model in the context of different datasets. The specific experimental results are shown in Table 5. To support the reported differences, we compute 95% bootstrap confidence intervals (1000 resamples) for all toxicity rates and conduct two-proportion z-tests for the pairwise comparisons highlighted in the text. Table 6 reports the total-toxicity confidence intervals on the COLD dataset as a compact summary, and full per-cell intervals are provided in the repository. For the full-data settings the confidence-interval half-widths are at most ±1.4 percentage points, while for the models evaluated on 200 samples (LLAMA2-Chinese-7B, ChatGPT, and GPT-4) they reach approximately ±11 percentage points; we therefore interpret those comparisons with corresponding caution.

Experimental results on the COLD dataset show that BELLE exhibits the highest toxicity. When encountering toxic inputs, it responds to toxic content with a maximum probability of 27.26% and to non-toxic content with a minimum probability of 22.02%. Meanwhile, it possesses the highest probability of replying to non-toxic inputs when encountering non-toxic inputs. The above results reflect that the model only considers the realization of the dialog function and lacks safety considerations. For the EVA, it has a 43.40% probability of replying with non-toxic outputs when encountering toxic inputs, which indicates its ability to recognize and provide non-toxic responses. Across the evaluated checkpoints, safety is not determined by parameter count alone; comparisons across model families also reflect differences in model generation, training data, and alignment recipe.

It is worth mentioning that Pangu-α is also moderately toxic, which reflects the fact that although the model was released early, they are much less toxic than several recently released models. This indicates that the toxicity purification ability of the existing models is still relatively poor, and the purification ability of the models is not optimized with the model fine-tuning techniques.

Experimental results from the ToxicSpans and HateXplain datasets similarly demonstrate minimal toxicity of EVA. Even in HateXplain, its total toxicity is only 4.42%. We hypothesize that this is related to two reasons: 1) unlike other LLMs, the dataset used for EVA pre-training is pre-processed, which to a certain extent guarantees the model safety; 2) the ToxicSpans and HateXplain datasets are translated, and the translation process to some extent reduces the toxicity in the text. The first ChatGPT-like Chinese model, MOSS-7B, demonstrated greater toxicity on NT2T, with 7.97% and 8.05%, respectively. The results of BELLE in T2T experiments were similar to the COLD dataset, with 25.34% and 17.66%, respectively. We additionally measured the experimental results of LLAMA2-Chinese-7B and ChatGPT on three datasets. LLAMA2-Chinese-7B is a Chinese-adapted model based on Meta’s LLaMA2, whereas ChatGPT is a general-purpose multilingual model developed by OpenAI. Due to API interface call limitations, only 200 samples were randomly selected for testing. As we can see from the experimental results, ChatGPT shows alarming toxicity on the COLD dataset. Compared with other Chinese models, it is more toxic, which demonstrates the necessity of specialized development of Chinese contextual LLMs.

5.1.2 Comparison of Baichuan Model Generations and Sizes

We compare Baichuan-7B and Baichuan2-13B as two related but distinct model generations that also differ in parameter count. Across the three datasets, Baichuan2-13B has lower T2T and NT2T rates than Baichuan-7B, but it also has lower T2NT rates. These mixed changes do not support a simple monotonic safety improvement. Because model generation, training data, training recipe, and parameter count differ simultaneously, this comparison cannot isolate an effect of parameter scale.

5.1.3 Total Toxicity

To visualize the degree of toxicity of different models, we calculated their total toxicity. For each dataset, total toxicity is the proportion of toxic responses and therefore equals the sum of the T2T and NT2T percentages. Table 7 presents total toxicity alongside the bias metrics for each model across the different attribute classes of the COLD dataset.

5.2 Bias in Chinese Large Language Models

5.2.1 Quantitative Analysis

In this section, we quantify the bias of each model in combination with different datasets. The results of the experiment on COLD, HateXplain, and ToxicSpans are shown in Tables 7 and 8.

Taking the COLD dataset as an example, we evaluate bias by comparing total toxicity across sensitive-attribute classes. Baichuan-7B achieves the smallest STD, while ChatGPT has the largest STD. The two additional disparity statistics introduced in Section 3.2 lead to consistent conclusions: Baichuan-7B also attains the smallest max–min gap (Δ = 3.24%) and the smallest max/min ratio (ρ = 1.18), and the model rankings are largely consistent across the three disparity metrics (Kendall’s τ≥ 0.87). LLaMA2-Chinese-7B, a Chinese-adapted model, and ChatGPT, a general-purpose multilingual model, exhibit two of the largest disparities on COLD. Because their training corpora and alignment details are not fully disclosed, these differences cannot be attributed to the developers’ native language. They may instead reflect uneven Chinese-context coverage and the absence of explicit debiasing objectives; accordingly, we interpret this explanation as a hypothesis rather than a causal conclusion.

On COLD, ChatGPT has the highest total toxicity, whereas EVA has the lowest. This observed difference may reflect variation in training data and Chinese-specific safety alignment, but the evaluation does not support a causal attribution because the models’ complete training corpora are not publicly available. EVA also has the lowest total toxicity across the HateXplain classes. BELLE-7B has the highest total toxicity in every HateXplain class and in all three ToxicSpans classes, showing a consistent pattern across the two translated datasets.

5.2.2 Comparison Across Model Checkpoints

Moreover, Baichuan2-13B does not show consistently lower attribute-level toxicity disparities than Baichuan-7B. Because these checkpoints belong to different model generations and differ in training data, training recipe, and parameter count, their comparison cannot isolate the effect of model scale. The result instead indicates that a later, larger checkpoint does not necessarily exhibit lower bias without explicit debiasing objectives.

5.3 Avoidance Rate in Chinese Large Language Models

In this section, we explore the ability of the model to avoid answering the toxicity question. This ability comprehensively examines the model’s ability to understand and respond. On the one hand, the model needs to have the ability to recognize the toxic content; on the other hand, the model is expected to avoid responding to that question and give a toxicity alert.

5.3.1 Validation of the LLM-as-a-Judge Protocol

Since the avoidance rate relies on ChatGPT-based scoring, we validate this protocol against human judgments. We randomly sampled 300 prompt–response pairs covering all evaluated models, and asked three independent human annotators to score avoidance behavior using the same rubric provided to ChatGPT. The Spearman correlation between ChatGPT scores and the averaged human scores is ρ = 0.83 (p< 0.001; Pearson r = 0.81), and the agreement on the binarized avoidance decision is 87.3% (Cohen’s κ = 0.71). Notably, the ChatGPT–human agreement is comparable to the agreement among the human annotators themselves (Fleiss’ κ = 0.68), indicating that the automatic judge is a reasonable proxy for human evaluation in this task. To assess scoring stability, we repeated the ChatGPT scoring three times on the same subset with temperature 0; the intra-class correlation coefficient across repetitions is 0.91, indicating highly consistent scoring. Table 9 summarizes the validation results. We further note known LLM-judge biases, including self-preference, verbosity, and position effects: we mitigate position effects by scoring pairs in randomized order, and ChatGPT itself is excluded from the avoidance-rate ranking conclusions. The human-validation data are released in our repository.

Table 10 presents the results of the enlarged experiment (200 prompts per category; Section 3.3), which provides a more stable view of avoidance behavior than the initial 20-sample pilot. As shown in Table 10, Baichuan2-13B obtains the highest avoidance rate for region-related toxic prompts, whereas Baichuan-7B obtains the highest rates for race- and gender-related prompts. Pangu-α remains the weakest model across all three categories. Because several category-level confidence intervals overlap, we interpret fine-grained differences among the high-avoidance models cautiously; the main conclusion is that the Baichuan-family models and BELLE-7B exhibit substantially stronger avoidance than low-avoidance models such as Pangu-α, EVA-2.8B, and MOSS-7B.

5.4 Why Do Chinese LLMs Differ in Toxicity and Bias?

Beyond the statistical reporting above, we relate our findings to each model’s training recipe, framing the following as evidence-informed hypotheses since several models only partially disclose their training details. Training data factors. EVA’s consistently low toxicity is plausibly attributable to its preprocessed and filtered pretraining corpus, whereas BELLE’s high toxicity may relate to its instruction data being distilled from ChatGPT outputs without an additional safety-filtering stage; a summary of the publicly documented data pipelines of all evaluated models is provided in our repository. Alignment and RLHF. Contrasting base or lightly aligned models (e.g., Pangu-α, EVA) with instruction-tuned and RLHF-aligned models (e.g., Baichuan2, Qwen, ChatGPT, GPT-4), we observe that RLHF-style alignment primarily improves refusal behavior (avoidance) while attribute-level disparities persist—consistent with our finding that the evaluated checkpoints do not show a uniform reduction in bias. Language mismatch. Models aligned predominantly on English safety data (LLAMA2-Chinese-7B, ChatGPT) underperform on Chinese-specific toxicity, supporting the need for Chinese-centric safety alignment.

5.5 Limitations

Our study has several limitations. First, regarding cross-lingual validity, some culture-specific hate expressions may lose intensity or acquire different connotations after translation, and translated data may under-represent toxicity phenomena unique to the Chinese online environment; we therefore anchor our evaluation primarily on the native Chinese COLD dataset and use the translated datasets as complementary sources, and conclusions drawn from translated subsets should be interpreted with this caveat. Second, our bias measurement is an observational, outcome-level disparity measure (Section 3.2); it does not capture implicit associations, context-dependent bias, or causal bias under counterfactual attribute substitution, which we leave for future work via counterfactual prompt construction. Third, although validated against human judgments (Section 5.3), the LLM-as-a-judge protocol may retain residual evaluator biases. Finally, our findings hold for the evaluated models, datasets, and prompting conditions, and may not generalize to newer model versions.

6  Conclusion

In this paper, we investigate the trustworthiness of Chinese large language models by measuring toxicity and inherent language bias. Specifically, we propose the TrustEval evaluation framework, which measures the toxicity, bias, and avoidance rate of an LLM when encountering toxic or non-toxic prompts. Our main findings are as follows: First, all evaluated models produced toxic outputs under our prompting conditions and exhibited varying degrees of toxicity with different triggering prompts. Second, the evaluated Chinese large language models show bias or discrimination against different sensitive attributes. This indicates that the current model training paradigm only considers optimizing the accuracy of the model on downstream tasks while overlooking the debiasing design of the model. Finally, experiments on the avoidance rates revealed substantial differences across checkpoints, but all evaluated models still failed to handle toxic prompts reliably. In summary, it is clear that trustworthy AI still has a long way to go in terms of safety and debiasing according to the results of our toxicity and bias evaluation of the models. We hope that the findings in this paper will provide experience for future research on trustworthy AI in terms of safety and debiasing.

Acknowledgement: The authors would like to thank all annotators for their assistance in data translation and annotation.

Funding Statement: This research was funded by the Natural Science Foundation of Guangdong Province, China (Grant No. 2025A1515010733), and the General University Key Field Special Project of Guangdong Province, China (Grant Nos. 2024ZDZX1004 and 2022ZDZX1035).

Author Contributions: The authors confirm contribution to the paper as follows: Rong Ma: conceptualization of the three-dimensional TrustEval evaluation framework, formal analysis of experimental results, original draft preparation, review and editing of the manuscript. Jin Ren: methodology design, software implementation of the COLD-based toxicity-scoring pipeline, dataset curation and preprocessing (COLD, HateXplain, ToxicSpans), and all model experiments. Shaobing Shen: crowdsourcing translation and fine-grained annotation of English datasets into Chinese, data quality assurance via manual voting, and validation of results. Yunhe Li: conceptualization, supervision of the overall research direction, funding acquisition, critical revision of the manuscript, and final approval. Man Hu: crowdsourcing translation and annotation support, data quality assurance, manual verification of annotated samples, and manuscript revision. All authors reviewed and approved the final version of the manuscript.

Availability of Data and Materials: The data that support the findings of this study are available at https://github.com/fenffef/TrustEval and from public repositories: COLD (https://aclanthology.org/2022.emnlp-main.796/), HateXplain, and ToxicSpans.

Ethics Approval: This study involved adult graduate-student annotators working with publicly available text datasets. The annotation tasks involved no patients, clinical data, or vulnerable populations; therefore, formal ethics approval was not required. All annotators provided informed consent and were briefed about the potentially offensive content before participating.

Conflicts of Interest: The authors declare no conflicts of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

1https://github.com/yangjianxin1/Firefly-LLaMA2-Chinese

2https://www.perspectiveapi.com/

3https://huggingface.co/thu-coai/roberta-base-cold

4https://github.com/thu-coai/EVA

5https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PanGu-Alpha

6https://github.com/LianjiaTech/BELLE

7https://huggingface.co/fnlp/moss-base-7b

8https://github.com/zai-org/ChatGLM-6B

9Model checkpoint: https://huggingface.co/YeungNLP/firefly-baichuan-7b-qlora-sft; Firefly framework: https://github.com/yangjianxin1/Firefly

10https://github.com/baichuan-inc/Baichuan2

11https://github.com/yangjianxin1/Firefly-LLaMA2-Chinese

12https://github.com/QwenLM/Qwen

13https://chatgpt.com/

14https://openai.com/gpt-4

References

1. Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877–901. doi:10.65525/svup.9788199778009.2026.224-230. [Google Scholar] [CrossRef]

2. Chung HW, Hou L, Longpre S, Zoph B, Tay Y, Fedus W, et al. Scaling instruction-finetuned language models. J Mach Learn Res. 2024;25(70):1–53. [Google Scholar]

3. OpenAI. GPT-4 technical report. arXiv:2303.08774. 2023. [Google Scholar]

4. Wang J, Hu X, Hou W, Chen H, Zheng R, Wang Y, et al. On the robustness of ChatGPT: an adversarial and out-of-distribution perspective. arXiv:2302.12095. 2023. [Google Scholar]

5. Bai J, Bai S, Chu Y, Cui Z, Dang K, Deng X, et al. Qwen technical report. arXiv:2309.16609. 2023. [Google Scholar]

6. Baichuan. Baichuan 2: open large-scale language models. arXiv:2309.10305. 2023. [Google Scholar]

7. Gu Y, Wen J, Sun H, Song Y, Ke P, Zheng C, et al. EVA2.0: investigating open-domain Chinese dialogue systems with large-scale pre-training. Mach Intell Res. 2023;20(2):207–19. [Google Scholar]

8. Sun T, Zhang X, He Z, Li P, Cheng Q, Yan H, et al. MOSS: training conversational language models from synthetic data. arXiv:2307.15020. 2023. [Google Scholar]

9. Zeng W, Ren X, Su T, Wang H, Liao Y, Wang Z, et al. Pangu-α: large-scale autoregressive pretrained Chinese language models with auto-parallel computation. arXiv:2104.12369. 2021. [Google Scholar]

10. Deshpande A, Murahari V, Rajpurohit T, Kalyan A, Narasimhan K. Toxicity in ChatGPT: analyzing persona-assigned language models. arXiv:2304.05335. 2023. [Google Scholar]

11. Zhao J, Fang M, Shi Z, Li Y, Chen L, Pechenizkiy M. CHBias: bias evaluation and mitigation of Chinese conversational language models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics; 2023 Jul 9–14; Toronto, ON, Canada. Stroudsburg, PA, USA: ACL; 2023. p. 13538–56. [Google Scholar]

12. Zhao S, Tuan LA, Fu J, Wen J, Luo W. Exploring clean label backdoor attacks and defense in language models. IEEE/ACM Trans Audio Speech Lang Process. 2024;32(1):3014–24. doi:10.1109/taslp.2024.3407571. [Google Scholar] [CrossRef]

13. Zhao S, Lin Q, Jia Y, Wu X, Li Y, Tuan LA. UniFLE: uniform fusion of multiple LoRA experts for backdoor defense in large language models. IEEE Trans Dependable Secure Comput. 2026;23(3):6620–34. [Google Scholar]

14. Du Z, Qian Y, Liu X, Ding M, Qiu J, Yang Z, et al. GLM: general language model pretraining with autoregressive blank infilling. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics; 2022 May 22–27; Dublin, Ireland. Stroudsburg, PA, USA: ACL; 2022. p. 320–35. [Google Scholar]

15. Zhang Z, Lei L, Wu L, Sun R, Huang Y, Long C, et al. SafetyBench: evaluating the safety of large language models with multiple choice questions. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics; 2024 Aug 11–16; Bangkok, Thailand. Stroudsburg, PA, USA: ACL; 2024. [Google Scholar]

16. Gallegos IO, Rossi RA, Barrow J, Tanjim MM, Kim S, Dernoncourt F, et al. Bias and fairness in large language models: a survey. Comput Linguist. 2024;50(3):1097–179. doi:10.1162/coli_a_00524. [Google Scholar] [CrossRef]

17. Liu H, Dacon J, Fan W, Liu H, Liu Z, Tang J. Does gender matter? Towards fairness in dialogue systems. In: Proceedings of the 28th International Conference on Computational Linguistics; 2020 Dec 8–13; Barcelona, Spain. Stroudsburg, PA, USA: ACL; 2020. p. 4403–16. [Google Scholar]

18. Deng J, Zhou J, Sun H, Zheng C, Mi F, Meng H, et al. COLD: a benchmark for Chinese offensive language detection. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing; 2022 Dec 7–11; Abu Dhabi, United Arab Emirates. Stroudsburg, PA, USA: ACL; 2022. p. 11580–99. [Google Scholar]

19. Mathew B, Saha P, Yimam SM, Biemann C, Goyal P, Mukherjee A. Hatexplain: a benchmark dataset for explainable hate speech detection. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press; 2021. p. 14867–75. [Google Scholar]

20. Pavlopoulos J, Laugier L, Xenos A, Sorensen J, Androutsopoulos I. From the detection of toxic spans in online discussions to the analysis of toxic-to-civil transfer. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics; 2022 May 22–27; Dublin, Ireland. Stroudsburg, PA, USA: ACL; 2022. p. 3721–34. [Google Scholar]

21. Bommasani R, Liang P, Lee T. Holistic evaluation of language models. Ann N Y Acad Sci. 2023;1525(1):140–6. doi:10.1111/nyas.15007. [Google Scholar] [PubMed] [CrossRef]

22. Cheng M, Durmus E, Jurafsky D. Marked personas: using natural language prompts to measure stereotypes in language models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics; 2023 Jul 9–14; Toronto, ON, Canada. Stroudsburg, PA, USA: ACL; 2023. p. 1504–32. [Google Scholar]

23. Zhang Z, Han X, Zhou H, Ke P, Gu Y, Ye D, et al. CPM: a large-scale generative Chinese pre-trained language model. AI Open. 2021;2:93–9. [Google Scholar]

24. Ji Y, Deng Y, Gong Y, Peng Y, Niu Q, Zhang L, et al. Exploring the impact of instruction data scaling on large language models: an empirical study on real-world use cases. arXiv:2303.14742. 2023. [Google Scholar]

25. Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, et al. LLaMA: open and efficient foundation language models. arXiv:2302.13971. 2023. [Google Scholar]

26. Huang Y, Bai Y, Zhu Z, Zhang J, Zhang J, Su T, et al. C-Eval: a multi-level multi-discipline Chinese evaluation suite for foundation models. In: NIPS ’23: Proceedings of the 37th International Conference on Neural Information Processing Systems; 2023 Dec 10–16; New Orleans, LA, USA. Red Hook, NY, USA: Curran Associates, Inc.; 2023. p. 62991–3010. [Google Scholar]

27. Li H, Zhang Y, Koto F, Yang Y, Zhao H, Gong Y, et al. CMMLU: measuring massive multitask language understanding in Chinese. In: Findings of the Association for Computational Linguistics: ACL 2024; 2024 Aug 11–16; Bangkok, Thailand. Stroudsburg, PA, USA: Association for Computational Linguistics; 2024. p. 11260–85. doi:10.18653/v1/2024.findings-acl.671. [Google Scholar] [CrossRef]

28. Wan Y, Wang W, He P, Gu J, Bai H, Lyu MR. BiasAsker: measuring the bias in conversational AI system. In: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering; 2023 Dec 3–9; San Francisco, CA, USA. New York, NY, USA: ACM; 2023. p. 515–27. [Google Scholar]

29. Zou A, Phan L, Chen S, Campbell J, Guo P, Ren R, et al. Representation engineering: a top-down approach to AI transparency. arXiv:2310.01405. 2023. [Google Scholar]

30. Turner AM, Thiergart L, Leech G, Udell D, Vazquez JJ, Mini U, et al. Steering language models with activation engineering. arXiv:2308.10248. 2023. [Google Scholar]

31. Li K, Patel O, Viégas F, Pfister H, Wattenberg M. Inference-time intervention: eliciting truthful answers from a language model. Adv Neural Inf Process Syst. 2023;36:41451–74. [Google Scholar]

32. Bhargava A, Witkowski C, Looi SZ, Thomson M. What’s the magic word? A control theory of LLM prompting. arXiv:2310.04444. 2023. [Google Scholar]

33. Nosrati K, Tepljakov A, Belikov J, Petlenkov E. When control meets large language models: from words to dynamics. Eng Appl Artif Intell. 2026;178(2):115119. doi:10.1016/j.engappai.2026.115119. [Google Scholar] [CrossRef]

34. Karnik S, Bansal S. Preemptive detection and steering of LLM misalignment via latent reachability. arXiv:2509.21528. 2025. [Google Scholar]

35. Gehman S, Gururangan S, Sap M, Choi Y, Smith NA. RealToxicityPrompts: evaluating neural toxic degeneration in language models. In: Findings of the Association for Computational Linguistics: EMNLP 2020; 2020 Nov 16–20; Online. Stroudsburg, PA, USA: ACL; 2020. p. 3356–69. [Google Scholar]

36. Si WM, Backes M, Blackburn J, De Cristofaro E, Stringhini G, Zannettou S, et al. Why so toxic? Measuring and triggering toxic behavior in open-domain chatbots. In: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security; 2022 Nov 7–11; Los Angeles, CA, USA. New York, NY, USA: ACM; 2022. p. 2659–73. [Google Scholar]