Mini-Gemini: A Simple and Effective AI Framework for Enhancing Multimodal Vision-Language Models
-
Recently, researchers from The Chinese University of Hong Kong and SmartMore introduced a novel framework called Mini-Gemini, which advances VLMs through enhanced multimodal input processing. Mini-Gemini employs a dual-encoder system and an innovative patch information mining technique, combined with a specially curated high-quality dataset, enabling it to effectively process high-resolution images and generate rich visual and textual content, thereby distinguishing itself.
Mini-Gemini's methodology includes a dual-encoder system featuring a convolutional neural network for fine-grained image processing, enhancing visual tokens without increasing their quantity. It utilizes patch information mining to extract detailed visual cues. The framework is trained on a composite dataset that integrates high-quality image-text pairs and task-oriented instructions to improve model performance and application scope. Mini-Gemini is compatible with various large language models (LLMs), ranging from 2B to 34B parameters, achieving efficient arbitrary inference. This setup allows Mini-Gemini to achieve outstanding results in zero-shot benchmarks and supports advanced multimodal tasks. When evaluating the effectiveness of Mini-Gemini, the framework demonstrated leading performance in several zero-shot benchmarks. Specifically, it surpassed the Gemini Pro model in the MM-Vet and MMBench benchmarks, achieving scores of 79.6 and 75.6, respectively. When configured as Hermes-2-Yi-34B, Mini-Gemini achieved an impressive 70.1 score in the VQAT benchmark, exceeding the performance of the existing LLaVA-1.5 model across all evaluation metrics. These results validate Mini-Gemini's efficiency and precision in handling complex visual and textual tasks.
The research introduces Mini-Gemini, advancing VLMs through a dual-encoder system, patch information mining, and high-quality datasets. Mini-Gemini exhibits outstanding performance across multiple benchmarks, surpassing existing models and marking a significant leap in multimodal AI capabilities.
However, as the researchers acknowledge, Mini-Gemini still has room for improvement in visual understanding and reasoning abilities. They assert that future work will explore advanced methods for visual understanding, reasoning, and generation. Project entry: https://top.aibase.com/tool/minigemini
Paper address: https://arxiv.org/abs/2403.18814