What Are the Application Scenarios of Generative AI?

[baoshi.rao]

Generative AI has a wide range of applications, covering various types of content such as text, images, music, and videos. Here are some typical use cases:

Text-to-Image
Text-to-image refers to generating corresponding images based on text descriptions. This application can be used in graphic design, artistic creation, education, and entertainment. Examples include:

• DALL·E 2: Developed by OpenAI, DALL·E 2 is a text-to-image model based on the Diffusion Model, capable of generating realistic, high-resolution images from text.
• Stable Diffusion: A text-to-image model based on the Latent Diffusion Model, Stable Diffusion produces high-definition, stylistically diverse, and visually varied images.
• Imagen: Developed by Google, Imagen combines a Diffusion Model with a pre-trained language model (T5-XXL) to generate high-fidelity, high-quality images from text.
• Parti: Also from Google, Parti uses an Auto-regression Model and an image tokenizer (ViT-VQGAN) to create high-quality, visually diverse images.

Text-to-Music
Text-to-music involves generating music based on text descriptions. This can be applied in music composition, education, and appreciation. Examples include:

• Jukebox: Developed by OpenAI, Jukebox combines a Variational Autoencoder (VAE) with an Auto-regression Model to generate original songs based on artist, genre, and lyrics.
• MuseNet: Another OpenAI creation, MuseNet uses a Transformer-based model to compose original music based on instruments, styles, or composers.
• Coconet: Developed by Google, Coconet employs a Convolutional Neural Network (CNN) to generate original music based on melody, harmony, and rhythm.
• Magenta: A Google open-source project built on TensorFlow, Magenta explores machine learning applications in music and art, including text-to-music, music-to-music, and image-to-music functionalities.

Text-Based Communication
Text-based communication involves interacting with users through text, useful for customer service, consulting, entertainment, and education. Examples include:

• ChatGPT: Developed by OpenAI, ChatGPT is based on a fine-tuned version of GPT-3 (GPT-3.5) and can engage in highly human-like conversations and generate specific text formats based on instructions.
• DialoGPT: From Microsoft, DialoGPT is a fine-tuned version of GPT-2 (GPT-2.7B) capable of fluent multi-turn dialogues and handling complex linguistic phenomena like emotion, humor, and sarcasm.
• BlenderBot: Developed by Facebook, BlenderBot combines Transformer and Retriever models to engage in deep, interesting conversations and enhance dialogue with external knowledge sources.
• Meena: A Google creation, Meena uses an Evolved Transformer and Reformer to conduct flexible conversations with human-like perception and emotion.

Text-Driven Robotics
Text-driven robotics involves controlling robots to perform actions based on text instructions, applicable in robotics control, education, and collaboration. Examples include:

• RoboTHOR: Developed by the University of Washington, RoboTHOR is a virtual environment built on Unity 3D for training and testing robots in indoor scenarios.

ALFRED: ALFRED is a Transformer-based text-driven robotic model jointly developed by the University of Washington and Stanford University. It can control robots in virtual home environments to perform daily tasks such as cooking, cleaning, and organizing based on natural language instructions.

RoboChat: RoboChat is a text-driven robotic service developed by Huawei Cloud, based on multimodal interaction. It can control digital avatars to engage in real-time conversations with users based on text commands and enhance dialogue content using external knowledge sources.

Text2Robot: Text2Robot is a text-driven robotic model based on deep reinforcement learning, developed by Tsinghua University. It can control robots to perform tasks like navigation, transportation, and assembly in virtual scenarios based on text instructions and adjust behavior strategies according to environmental changes and feedback.

TextWorld: TextWorld is a text-driven robotic platform based on text adventure games, developed by Microsoft. It can generate complex virtual worlds and tasks based on text descriptions and train and evaluate robots' performance and learning abilities in games.

Text-to-Video

Text-to-video refers to generating corresponding videos based on text descriptions. This application can be used in scenarios such as video creation, teaching, and entertainment. For example:

Vid2vid: Vid2vid is a GAN-based text-to-video model developed by NVIDIA. It can generate realistic high-resolution videos from text and modify video content and style based on user controls and edits.

VideoBERT: VideoBERT is a text-to-video model combining Transformer and BERT, developed by Google. It can generate video clips related to text and produce text descriptions based on video clips.

DALL-E Mini: DALL-E Mini is a text-to-video model combining DALL-E and CLIP, developed by OpenAI. It can generate dynamic images (GIFs) from text and produce text descriptions based on dynamic images.

VideoGPT: VideoGPT is a text-to-video model based on a fine-tuned version of GPT-2 (GPT-2.1B), developed by Stanford University. It can generate low-resolution videos from text and produce text descriptions based on videos.