Microsoft Launches Innovative Language Model Phi-2: 2.7 Billion Parameters Challenge Performance Boundaries of Large Models
-
On December 13, Microsoft announced its latest language model, Phi-2, which challenges the performance boundaries of large language models with its 2.7 billion parameters. According to Microsoft, through innovations in model scaling and training data organization, Phi-2 demonstrates performance comparable to or even better than models 25 times its size in complex benchmarks.
Phi-2 is a Transformer-based model trained on 1.4T tokens, sourced from synthetic datasets in natural language processing (NLP) and coding, as well as web datasets. The training process on 96 A100 GPUs took 14 days. Notably, Phi-2 is a foundational model that has not been fine-tuned with human feedback reinforcement learning (RLHF) or guided fine-tuning. Despite this, compared to existing open-source models that have undergone alignment, Phi-2 shows better performance in terms of toxicity and bias.
In various comprehensive benchmarks, Phi-2 outperforms Mistral and Llama-2 models with 7 billion and 13 billion parameters. Remarkably, in multi-step reasoning tasks (such as coding and mathematics), Phi-2 achieves performance equivalent to 25 times that of the Llama-2-70B model. Additionally, compared to the recently released Google Gemini Nano 2, Phi-2 performs on par or even better in certain aspects.
This groundbreaking achievement indicates significant progress in Microsoft's research in the field of language models. The successful release of Phi-2 provides new ideas and methods for research in natural language processing and artificial intelligence. Microsoft stated that they will continue to advance the development of language models and look forward to more innovative breakthroughs in the future.