Bias Analysis of L2 Speaking Assessment Systems Usi... | AI Research

Bias Analysis of L2 Speaking Assessment Systems Using Concept Activation Vectors explores how to detect if automated speaking assessment systems are biased against specific speaker attributes, such as first language (L1) or age, rather than focusing solely on language proficiency. Researchers Arya Labroo, Mengjie Qian, and Kate Knill investigate whether these systems—specifically BERT-based text graders and Whisper-based multimodal graders—rely on irrelevant demographic information when assigning scores.

Detecting Bias with Concept Activation Vectors

The researchers use Concept Activation Vectors (CAVs) to determine if a model encodes specific concepts and whether those concepts influence the final score. A CAV represents a human-interpretable concept as a direction within the model's internal activation space. By training a linear classifier to separate examples that exhibit a concept from those that do not, the team can identify the "concept direction."
To determine if this concept actually influences the model's output, the authors use a gradient-based sensitivity metric. This metric measures the alignment between the concept direction and the gradient of the predicted score. If a concept is both linearly recoverable and aligned with the scoring direction, it suggests the model may be using that attribute to influence its assessment.

Probing with Sparse Autoencoders

Because modern Transformer models often represent information in complex, entangled ways, the authors test whether Sparse Autoencoders (SAEs) provide a clearer view of these concepts. SAEs map activations into a higher-dimensional, sparse latent space where features may be more disentangled.
The researchers trained SAEs on the graders' activations and learned CAVs within this sparse latent space. They then mapped these vectors back into the original activation space to evaluate them using the same sensitivity metric. This approach aims to address the limitation that standard CAVs rely on the assumption that a concept can be represented by a single linear direction, which becomes less reliable as model complexity increases.

Findings on Recoverability and Influence

The study reveals that the ability to recover a concept depends heavily on the specific architecture and layer being probed, rather than the concept itself. For instance, the Whisper model’s dense.in layer was the only point where every tested concept was linearly recoverable. In contrast, other layers in both BERT and Whisper models showed "degenerate" results for many L1 concepts, meaning the classifier could not reliably distinguish them.
The authors also found that SAEs make concepts more linearly recoverable but tend to attenuate the sensitivity measured in the original activation space, particularly in low-dimensional layers.

Key Considerations for Bias Auditing

The research concludes that concept recoverability and concept influence are distinct phenomena. Just because a model "knows" a concept (it is recoverable) does not mean it is using that concept to determine a score (it is influential). Therefore, auditing for bias requires distinguishing between these two factors. The authors note that the mapped CAVs from SAEs are constrained by what the autoencoder successfully reconstructs; if the SAE fails to reconstruct the part of the activation space relevant to a specific concept, the resulting bias analysis may be unreliable.