Servicios Personalizados
Revista
Articulo
Indicadores
- Citado por SciELO
- Accesos
Links relacionados
- Similares en SciELO
Compartir
Computación y Sistemas
versión On-line ISSN 2007-9737versión impresa ISSN 1405-5546
Resumen
URIBE, Diego; CUAN, Enrique y URQUIZO, Elisa. The Impact of Training Methods on the Development of Pre-Trained Language Models. Comp. y Sist. [online]. 2024, vol.28, n.1, pp.109-124. Epub 10-Jun-2024. ISSN 2007-9737. https://doi.org/10.13053/cys-28-1-4718.
The focus of this work is to analyze the implications of pre-training tasks in the development of language models for learning linguistic representations. In particular, we study three pre-trained BERT models and their corresponding unsupervised training tasks (e.g., MLM, Distillation, etc.). To consider similarities and differences, we fine-tune these language representation models on the classification task of four different categories of short answer responses. This fine-tuning process is implemented with two different neural architectures: with just one additional output layer and with a multilayer perceptron. In this way, we enrich the comparison of the pre-trained BERT models from three perspectives: the pre-training tasks in the development of language models, the fine-tuning process with different neural architectures, and the computational cost demanded on the classification of short answer responses.
Palabras llave : Language models; pre-training tasks; BERT; fine-tuning.