Short bio
Masters of Engineering, Ecole des Ponts
Thesis topic
Cheap and expressive neural contextual representations for textual data.
Short abstract
Neural language models are pre-trained using self-supervised learning to produce contextual representations of text data like words or sentences. These representations are shaped by the pre-training procedure: its data, its task, its optimization scheme, among others. I aim at identifying ways to improve the quality of the text representations by leveraging new pre-training approaches, in order to reduce data and/or compute requirements without quality loss.