Google AI Introduces New Scorer Model Cappy to Enhance Multi-Task Language Model Performance
-
In the latest research paper, Google researchers introduced a pre-trained scorer model named Cappy, designed to enhance and surpass the performance of large multi-task language models. This research aims to address the challenges faced by large language models (LLMs), including high computational resource costs and inefficient training and inference processes.
Currently, multi-task LLMs such as T0, FLAN, and OPT-IML are widely used for various natural language processing tasks and are trained under a unified instruction-following framework. However, due to their massive scale and hardware requirements, these models face challenges when adapting to downstream applications. To address these challenges, Cappy was introduced as a lightweight pre-trained scorer aimed at improving the performance and efficiency of multi-task LLMs. Cappy's architecture is based on RoBERTa, with a linear layer for regression added on top, leveraging a diverse collection of datasets for pretraining to ensure coverage of a wide range of task types. The researchers also proposed a data construction method to meet the need for label diversity in pretraining data and to generate a large and effective regression pretraining dataset. Cappy's applications involve a candidate selection mechanism that can operate independently for classification tasks or serve as an auxiliary component for generation tasks, enhancing the decoding of existing multitask LLMs.
By introducing the lightweight pretrained scorer Cappy, this research addresses the challenge of effectively utilizing large language models in multitask scenarios, demonstrating its superior parameter efficiency and performance across various tasks, while highlighting the potential to simplify the adoption of large language models in practical applications.