Using AI as a Director: Can Such Powerful AI Generate a Movie?

baoshi.rao

In November 2022, OpenAI developed ChatGPT, a natural language processing tool powered by AI technology, which introduced a new way of retrieving information and communication. It can interact with humans and even perform tasks such as writing film scripts, copywriting, code, and academic papers. Within just two months of its launch, ChatGPT achieved over 100 million monthly active users. It can engage in interactive conversations by learning human language and understanding context, and it is capable of questioning and admitting mistakes.

So, can such powerful AI generate a movie?

The application of AI in the film industry has sparked reflections within the field. Oded Avidar, a senior lecturer in the Film Production Department at Vancouver Film School Shanghai, noticed the latest trends in AI and developed specialized AI courses by the end of December last year. These courses, which began this new semester, enable students to proficiently apply AI technology in actual film production.

In her classroom, students use the AI image generation software Midjourney to input keywords that help them create movie concept art. For example, to generate a "Chinese version" of Clint Eastwood's actor image, students need to input detailed keywords such as "shot with 35mm lens," "shallow focus," "close-up photo," etc., in order to obtain images that meet film industry standards. This indicates that the feasibility of AI in film art design is beginning to show promise.

The application of Midjourney offers filmmakers an experimental platform to more efficiently create images that meet the demands of film production. By utilizing AI-generated imagery, filmmakers can quickly obtain the required materials, thereby improving work efficiency. Additionally, AI can provide creative inspiration during the production process, injecting new perspectives into filmmaking.

AI-generated Chinese counterpart image based on Clint Eastwood's likeness. Image source: Internet

However, the final presentation of a film is not solely composed of visual effects. Although AI has achieved some notable progress in art construction, film is a comprehensive synthesis of multiple art forms. Therefore, in processes such as scriptwriting, character performance, director guidance, and post-production editing, AI creation still faces numerous challenges.

Even before AI software like Midjourney went viral, experiments in AI-generated films had already commenced in China, yielding innovations across multiple dimensions. A prominent example is the "Artificial Intelligence Infinite Film (AI-IF)" project by renowned Chinese contemporary artist Xu Bing. Initiated in 2017, this collaborative project between artists and AI scientists has prompted a rethinking of filmmaking methodologies.

In this project, through research in automatic text generation, scene generation, dialogue generation, video retrieval, text-to-speech synthesis, and music synthesis, state-of-the-art deep learning algorithms were implemented to build a software system capable of producing movies in real-time without a film crew and interacting with the audience, with all movie content entirely created by AI.

The final visual presentation of the film is created and edited by AI algorithms based on audience preferences, selecting and capturing relevant video clips from news and other internet content.

At the 2021 Pingyao International Film Festival, Xu Bing and his team first introduced to the audience an interactive AI film version. Viewers can select film genres and durations on a computer interface, and by inputting keywords or sentences, they can generate AI-created films that never repeat. Additionally, audiences can input new vocabulary during playback to alter characters and narrative plots in the film, making them part of the filmmaking process.

Xu Bing introduces his new work 'Artificial Intelligence Infinite Film' (AI-IF, 2020). Source: Internet

The AI Infinite Movie addresses the question of how film, as a medium and technological carrier, generates itself. This film involves four technical frameworks: a script model, a video subtitle model, the matching of generated scripts with video subtitles, and models for generating dialogue audio and background music. The most challenging aspect of this technical framework is not the control of each individual machine learning model, but rather how these four different algorithmic models collaborate with each other to create a series of feedback loops.

The interactive interface for 'Infinite AI Movies' is currently powered by the GPT-2 algorithm.

Based on the four frameworks mentioned above, the research team conducted multiple sets of tests. For the initial NLP (Natural Language Processing) auto-generated script model, they ultimately selected the open-source OpenAI GPT-2 due to its sensitivity to script format and context constraints. The algorithm then annotates the meaningful segments extracted from the clips.

When annotating content, they primarily selected six parameters: characters, locations, number of people, events occurring in the video, actions, and objects. Based on these parameters, the algorithm then matches the most compatible script model with the video model. Finally, automatically generated dialogues and background music are added, producing the final video output according to the sentence content initially input by the viewer.

The AI-generated film 'Infinite' had its first public screening at the Pingyao International Film Festival, where audience members participated in interactive sessions and asked questions.

The 'Artificial Intelligence Infinite Film (AI-IF)' represents the cutting-edge application of AI in the film industry, but it also highlights the potential limitations of AI in film creation. Although AI has potential in generating film content, it still requires humans to set parameters and provide key information to guide the creative process. This underscores the complementary relationship between AI and humans, rather than one of replacement. Furthermore, AI-generated film content may lack emotion, creativity, and human subjectivity - elements that are core to filmmaking.

Film theorist André Bazin once said, 'Cinema is an asymptote of reality.' With the continuous development of AI technology, filmmakers now face the challenge of constructing and integrating the 'real world' with the 'virtual world.' Even if AI masters the logic of life, human nature, and the principles of artistic authenticity, it still cannot replace humans in creating films. In a way, this represents a collaborative division of labor between humans and AI in the creative process of filmmaking, working together to shape cinematic worlds.

Currently, AI is mainly utilized in film special effects. It can create the necessary atmosphere for movie narratives or independently form entire worlds. In the movie Dune, director Denis Villeneuve extensively employed AI-generated effects to shape the film's central narrative thread—'dreams.' The protagonist Paul frequently sees fragmented and hazy scenes in his dreams, such as the endless desert, the blue-eyed Fremen girl, and the faintly visible planet Arrakis.

Poster of the movie 'Dune'. Image source: Internet

The massive silent objects presented by AI effects in films form a visual rhetoric between objects. They are both grand and vast, as well as small and subtle. These divergent, flowing, and chaotic dreams are like fragments floating in the character's consciousness, serving as an important environmental atmosphere for character portrayal.

Still from the movie 'Dune'. Image source: Internet

In contrast to the approach of integrating AI effects into the protagonist's real life, the film 'A Writer's Odyssey' utilizes AI to create an alternate world. The story follows Guan Ning, a father who sets out to assassinate a novelist in search of his missing daughter. However, the tale the novelist is writing eerily mirrors Guan Ning's own life. Thus, dreams, the fictional world, and reality unfold in parallel. The movie is structured between a realistic world and a fantastical, surreal world constructed with AI, juxtaposed and running concurrently.

Poster of 'A Writer's Odyssey'. Image source: Internet

The surreal world is mainly driven by AI digital technologies, such as data and information collection, motion capture and virtual filming, purely virtual production, combined virtual and real filming, digital creature and humanoid creation, digital lighting systems, and more. These methods allow the audience to visually distinguish the AI world from the real world, emphasizing the interaction between the film and the viewers, including immersive physical experiences within the stories of both worlds.

The AI virtual world in 'A Writer's Odyssey'. Image source: Internet

So, the sense of realism brought by films is not about narrowly approximating reality in a strict sense, but rather about creating a 'real world' that audiences can believe in. AI creation offers more diverse possibilities for cinematic realism, where social reality serves merely as a structural element in film narratives, ultimately submitting to sensory reality.

As previously mentioned, Midjourney's AI program opts to use copyright-free images in its material library to avoid copyright disputes. The March 16th Federal Register released by the US government shows that the US Copyright Office (USCO) has explicitly stated in Part 202 of the Code of Federal Regulations (CFR), titled 'Copyright Registration Guidance,' that automatically generated works are not protected by copyright law.

The copyright recognition standard for AI-generated content is as follows: If all elements of a work are produced by a machine without any contribution from a human actor, it does not qualify for copyright registration. From the perspective of the Copyright Office, copyright can only protect the products of human creativity, which is consistent with the U.S. Constitution and the Copyright Act, which limit the term 'author' to 'humans.'

Screenshot of the U.S. Federal Register

Copyright issues have always been an inevitable topic in artistic creation. The emergence of new technologies necessitates the improvement of corresponding laws, regulations, and policies. With the continuous development of AI-generated content, it is essential to establish legal protections for such works. The health and sustainable development of the film industry are crucial components of the cultural sector. It is not only a commercially valuable industry but also carries the mission of cultural heritage and creative expression. In the digital age, the integration of technology and the widespread application of AI have profoundly transformed film production, distribution, and audience interaction. However, as mentioned, properly addressing copyright protection issues is a critical step for the film industry to effectively leverage the advantages of AI technology.