Cognitive Automated Essay Scoring and Representations (CAESAR)

Automated essay scoring (AES) using large language models (LLMs) offers promising use cases for language assessment, yet high-stakes testing requires scores that are both controllable and verifiable, which current LLM-based scorers lack. Given that, we investigate whether LLM-based scorers form distinct internal representations for individual scoring rubrics and whether these representations can be independently controlled. Probing and steering 25 open-weight base LLMs (0.13B-10.31B parameters; 14 families) on human-scored BESTEP English writing tasks, we find that holistic scores are decodable from model activations for the longer tasks (Parts 2 and 3) and only weakly for the shortest (Part 1). Furthermore, scoring rubrics have partly separable directions in representation space for the longer tasks, though not for short responses (Part 1). However, causal steering reveals a critical limitation: applying a matching intervention (e.g., steering "grammar" on a grammar-degraded response) yields only a negligible advantage (about 2% of the effect) over a mismatched intervention (e.g., steering "vocabulary"), and none that is detectable in any of the 10 testable comparisons across reliably measured rubrics. Rather than controlling distinct evaluative dimensions, the rubric directions share a dominant lever whose rubric-specific parts are not selective, which may mirror the halo structure of human ratings. Our contributions are two-fold. First, we show that analytic scoring rubrics designed for human raters do not behave as independent controls in base LLMs, even among the seven models whose scores agree best with human raters (r ≥ 0.6). Second, we establish a methodology that shows that score agreement and decodability do not imply rubric-level control in LLM-based assessment.

While standardized scores provide a macro-level view of proficiency, they often lack the diagnostic granularity required for localized pedagogical intervention. This study addresses this gap by presenting the development of the Writing Learner Corpus, a specialized digital repository designed to enhance assessment literacy through data-informed insights. Theoretically, this project is grounded in Learning-Oriented Assessment (LOA) (Carless, 2007), which advocates for the integration of assessment tasks with instructional goals to promote learner autonomy. Furthermore, it draws on Learner Corpus Research (LCR) (Granger, 2015), similar to other learner corpus studies (Dahlmeier et. al., 2013), treating learner errors not as failures but as "windows" into the developmental state of interlanguage. Our corpus facilitates a Humanistic Approach to assessment: the learner’s output as a unique trajectory of growth rather than a static data point. The study aims to address two core research questions: Sectional vs. Overall Proficiency: Moving beyond the "unitary trait" assumption, we analyze how sub-dimensions interact. We investigate whether high composite scores mask specific linguistic fossilizations that could impede success in writing tasks. Disciplinary Variable Impact: Drawing on Hyland’s (2004) theory of Disciplinary Discourse, we examine how a learner’s academic background (e.g., STEM vs. Social Sciences) influences rhetorical strategies and error patterns. Previous findings suggest that while STEM students may demonstrate high structural accuracy, they often struggle with the "hedging" and "boosting" strategies required for academic persuasion. This study demonstrates how data-driven systems can bridge the "literacy gap" for both teachers and students. For teachers, the corpus provides a "discipline-based performance diagnosis," allowing for the transition from generic feedback to discipline-specific writing support. For learners, the system offers transparent benchmarks for self-regulation for specific section-based scores to overall performance. The session offers a scalable model for how institutions can leverage local assessment data to foster EMI instruction in the Asian context.