| dc.description.abstract | As Large Language Models (LLMs) are widely used in various applications, the potential biases and unfairness embedded in these models have received considerable attention. Existing studies have primarily focused on assessing bias in English LLMs, while research on bias assessment for non-English languages remains relatively scarce. Taking Chinese as an example, Simplified Chinese and Traditional Chinese differ in linguistic environments, language features, and cultural connotations, yet existing research has been overly concentrated on Simplified Chinese, neglecting the unique linguistic and cultural characteristics of regions where Traditional Chinese is the predominant form. The primary objective of this study is to establish a Traditional Chinese social bias benchmark for evaluating the handling of gender and ethnic group bias in large language models in the context of Taiwanese cultural background. Unlike previous research, we delve deeper into the various types of stereotypes in different demographic groups, including personality, institutional, cultural aspects, etc., to provide a more comprehensive bias assessment perspective. We redefine the Bias Specification based on CHBias and adopt the average perplexity as the metric for statistical difference calculation to more accurately identify and evaluate the biases present in language models. Furthermore, this study has improved upon previous research that directly calculated the perplexity (PPL) of collected stereotypical sentences by incorporating prompt templates for PPL calculation. This approach attempts to evaluate biases in a manner closer to actual application scenarios and reduces the overestimation of model biases during the evaluation process. Additionally, this study annotates whether sentences contain harmful speech and explores its impact on the degree of bias. Moreover, the proposed evaluation method can also be used to analyze biases in the training data by evaluating continual pretraining models and inferring biases present in the training data. The contributions of this research lie in the creation of a social bias(gender and ethnic group) benchmark based on the Taiwanese cultural context, which can be used by both academia and industry. Through the methods and findings of this study, we hope to provide valuable references for assessing social biases in Traditional Chinese large language models, promoting fairness and reducing bias in these models to better serve diverse application scenarios. | en_US |