Abstract:

Scaling up Language Models has led to increasingly advanced capabilities for those who can afford to train them. In order to enable community-tailored models for the rest of us, we will examine cross-model consistencies in how LMs acquire their linguistic knowledge-from fundamental syntax and semantics up to higher-level pragmatic features, such as culture. By identifying these consistencies across different models, we highlight opportunities for how they can enable dynamic personalization approaches that improve the accessibility of language technologies for underserved communities, in which collecting sufficient training data is physically impossible.

Bio:

Hej! I’m a postdoc at the IT University of Copenhagen’s NLPnorth Lab and the Danish Pioneer Centre for Artificial Intelligence, working with Anna Rogers. My research centers around identifying and leveraging consistencies in the learning dynamics of language models in order to make their training more efficient. On the data side, we’re looking into how linguistic properties in the training data lead to different generalization capabilities. On the modeling side, we investigate how different types of knowledge are represented across pre-training. We’ve applied findings from both pillars to make model adaptation to low-resource scenarios more efficient: e.g., improving cultural alignment of LMs to Danish, and enabling speech recognition for people with speech disabilities.