NVIDIA Releases 43B-Parameter Large Model ChipNeMo
-
NVIDIA's newly released 43-billion-parameter large language model ChipNeMo focuses on assisting chip design, aiming to improve engineers' work efficiency. This large language model has a wide range of applications, including question answering, EDA script generation, and bug summarization, making chip design more convenient.
NVIDIA's Chief Scientist Bill Dally emphasized that even a modest improvement in productivity makes using ChipNeMo worthwhile. ChipNeMo's dataset includes bug summaries, design sources, documentation, and hardware-related code and natural language texts. After data collection, cleaning, and filtering, it contains 24.1 billion tokens.
NVIDIA employed domain adaptation techniques, including custom tokenizers, domain-adaptive continuous pre-training, and supervised fine-tuning with domain-specific instructions, to enhance the performance of the large language model in engineering assistant chatbots, EDA script generation, and bug summarization and analysis.
The results show that these domain adaptation techniques not only improved performance but also reduced the model size, though there is still room for improvement. NVIDIA's initiative marks a significant step in the application of large language models in the semiconductor design field, providing a useful generative AI model for specialized domains.