VAST: The Valence-Aware Semantics Test for Contextualizing Language Models
Open AccessThis thesis introduces VAST, the Valence-Aware Semantics Test, an intrinsic evaluation task for measuring the quality of word-level semantics in contextualized word embeddings. VAST differs from other intrinsic evaluation tasks by using experimental settings to measure the effects of contextualization, tokenization, and model-specific geometry on word embeddings as they evolve through the layers of a language model.VAST uses the Word Embedding Association Test (WEAT) of Caliskan et al. to measure associations in word representations, and assesses the quality of semantics based on the extent to which word embeddings reflect widely accepted valence norms, building on the ValNorm task for static word embeddings introduced by Toney and Caliskan. While VAST can be used with any contextualizing language model, this thesis focuses specifically on GPT-2, a causal language model which has produced anomalous results when assessed using other intrinsic evaluation methods. Analysis of GPT-2 demonstrates the usefulness of VAST for interpreting the evolution of semantics in contextualizing language models and for informing the extraction of semantically rich contextualized word embeddings for further study or use in downstream applications. Specifically, this thesis finds that the semantics of context significantly alter the semantics of the word representations in the upper layers of GPT-2; that singly tokenized words undergo a different encoding process than do multiply tokenized words; and that removing dominating directions related to the pretraining task of the language model renders the semantics of top layer representations more interpretable. Results are validated by showing improvements on other intrinsic evaluation tasks and by measuring the correlation of embedding associations with dominance and arousal norms.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.