With Sora's Release, Will Special Effects Companies Be the First to Be Replaced?

[baoshi.rao]

We are on the eve of a tremendous transformation brought by AI.

A month ago, OpenAI released Sora, a large AI model for text-to-video generation, which immediately garnered global attention. What Sora will change has become a significant topic of global discussion.

Although OpenAI does not view Sora merely as a video model but rather as a 'world simulator,' the impact of easily generating videos from text input has led many to believe that the film and television industry will be the first to undergo profound changes and disruption. Domestic special effects industry veteran Mi Chunlin believes that this is indeed a 'terrifying upon closer thought' technology, but it is not so easy to completely revolutionize the film industry, especially the movie sector.

As a member of the China Film Association, an expert in the technical expert database of the National Radio and Television Administration, and a member of the Visual Effects Society (VES) in the United States, Mi Chunlin has extensive experience in virtual shooting, visual effects production, pre-shooting guidance for special effects, post-production process management, and team management.

He also participated in the shooting of 'If You Are the One 3', which was released before the Spring Festival. The LED virtual shooting for this movie was provided by his company, Digital Mi Cang. A screenshot from Mi Chunlin's company's demo video, with a virtual Hong Kong street scene behind him.

A few days ago, Mi Chunlin spoke with 20 She, analyzing the potential impact of Sora and AI technology on the film and special effects industry from both technical and industry perspectives.

20 She: Sora's demo videos have made a huge impact on us. As a senior expert in China's special effects industry, what was your first thought when you saw them? Mi: After seeing Sora's demo videos, my first impression was that its greater value lies not in special effects, but in the machine's new level of understanding about the world. The scenes presented by Sora are not just images, but reconstructions of the real world. This is a technology that's "terrifying when you think deeply about it."

This is what OpenAI calls a "world simulator" - it understands the operational logic of the entire world, with inherent physical laws. The characters and objects appearing in the videos are all combined according to these laws.

Therefore, I believe Sora's real game-changing capability hasn't been revealed yet. Currently, people are only seeing the video aspect and think it's nothing special. But its true value will far exceed our imagination. 20社：这可能就是Sora厉害的地方，虽然我们知道具体的影响会在未来一段时间才会释放出来，但是它的影响之大，不得不让我们现在就开始思考未来一段时间可能发生的各种变化，以及我们要如何应对。在你看来，特效乃至整个影视都要被颠覆了吗？

**米：**其实AI对影视行业的影响最近几年一直在持续发生。 However, AI and AIGC (AI-generated content, content entirely produced by AI) should be viewed separately.

AI tools are now widely used in many aspects of film production. For instance, some commonly used production software has integrated numerous AI tools to assist in the process. In traditional production, many steps can be replaced by algorithms, meaning AI has consistently helped reduce costs and improve efficiency.

AIGC has now evolved to tools like Sora, which can generate videos directly from text descriptions. The biggest challenge in its application, I believe, is controllability. Therefore, I believe the application of AIGC, including its more suitable future scenarios, is to provide auxiliary design functions in the early creative stages, offering more creative presentations. This is currently the most valuable application in my opinion.

However, feature films have very high requirements for controllability. If you say that AIGC can completely overturn the entire traditional film industry, I think it's not that easy yet.

Looking at it separately, in the entire video field, the higher the audience's demands for content, the less likely it is to be overturned. I think the first to be overturned might be short videos, primarily because people's requirements for short video content are not that high, the cost investment is not that high, and the cost of trial and error is also low. The most challenging industry for AI to impact may be film. From scriptwriting and shooting to final theatrical release, films are both labor-intensive and capital-intensive products. This inherently sets very high standards for all aspects of such products, including visuals, music, storytelling, and more.

If one day in the future, films were to be completely revolutionized by AI, it would first disrupt the entire industry chain. At that point, films might become similar to contemporary theater—a protected art form.

By then, just like with short videos, everyone could potentially become a creator of video content. 20社：我们应该怎么理解你说的这种改变？

**米：**Sora以及所有AIGC技术在传统影视拍摄制作行业中可能会带来重大的变革，但是否会彻底颠覆这一行业仍然取决于多个因素： First is technological progress. The advancement and development speed of AIGC technology will determine its application in film and television production. Based on the current presentation effects and application experience, it may bring significant changes to short video production, but it still cannot meet the quality and creative requirements for professional film and television production.

Another factor is audience demand and acceptance. The needs and acceptance of the audience for film and television content will also affect the application of AIGC technology in this industry. If audiences prefer traditional filming and production methods or are skeptical of content generated by artificial intelligence, AIGC may not completely disrupt this industry.

Of course, there are also legal and ethical issues. The use of AIGC technology in film and television production may trigger a series of legal and ethical issues, such as intellectual property, false information, and privacy. These issues may limit the scope of AIGC technology's application, thereby affecting its potential to disrupt the industry. AIGC encounters various resistances in reality. Let me give a simple example: when shooting celebrities for advertisements or movies, if you generate a replica of a celebrity, their income will be affected. Calculating the subsequent related benefits will take a long time to resolve.

Additionally, although artificial intelligence can generate content, in the film and television industry, creativity and human creativity are still indispensable. The imagination, emotional expression, and aesthetic concepts of human creators still possess unique value, which current artificial intelligence cannot completely replace.

For movies, directors have high requirements for the overall visual presentation. What AI produces may be roughly similar, but it will definitely differ from the specific results the director wants. This is why, even today, film production still requires extensive human involvement, as it is a refined creative process. In the production process, aspects such as the director's control over actors' rhythm, the pacing of camera movements, the selection of camera angles, and subtle adjustments are crucial. In traditional production workflows, these elements offer a high degree of controllability.

Therefore, although AIGC technology could have a major impact on the film and television production industry, completely disrupting this sector would likely require overcoming many challenges and depend on the interplay of multiple factors.

The recently released Madame Web received poor reviews and dismal box office performance, with the lead actor hinting that the script was AI-generated, highlighting that creative work cannot rely solely on algorithms. In traditional film production, there are indeed many people performing low-tech jobs, and AI is replacing this part. However, positions like directors, cinematographers, art directors, and action choreographers still require human decision-making.

The result may be that projects that previously required hundreds of people can now be completed by just a few dozen in the future.

20社: Specifically, what do you see as the most important potential or assistive capability of Sora? Mi: Sora may not be disruptive, nor will it disrupt the special effects and film industry, but it will serve production.

Currently, the fastest application of Sora is to generate different footage materials to assist production.

For example, the previous production process was relatively cumbersome, with back-and-forth adjustment steps. In the future, it might involve AI working with virtual shooting to directly generate 3D scenes, display them on screens, and then composite real actors into them, achieving what-you-see-is-what-you-get. I've heard that Hollywood has already started using this method to help directors better grasp actors' performances, with everything else being generated by AI.

Currently, AI is primarily serving pre-production work, such as creating concept art and storyboard scripts. Many people both domestically and internationally are working on this, and it's indeed very labor-saving. However, when it comes to performance aspects, even from a technical standpoint, it might not be achievable yet - after all, humans are the most complex parameters.

Even at this level of application, the impact on the industry is enormous. Film production is actually a labor-intensive job, with various departments consisting of a supervisor leading a team of workers - essentially one brain needing many hands to realize its vision. In the future, a single brain coupled with AI could accomplish tasks. So, how will those who are laid off survive? This is a question worth pondering.

Of course, the entire filming and production process in the film industry will undergo changes—not merely replacing steps in the existing workflow but constructing an entirely new process.

20社：Given that Sora might be difficult to use directly in China, and considering your previous orders from Hollywood, will your future business be affected? Mi: In fact, in recent years, our country's share in Hollywood and global film VFX outsourcing has remained relatively low. Our labor costs have been increasing, leaving us with no competitive advantage compared to India or even Singapore.

Previously, our advantage when dealing with Hollywood studios was cost. Now, most low-budget outsourcing projects are sent to India because their labor is cheaper.

A few years ago, I went to India for a research trip. There, engineers with Hollywood experience and compositing expertise were only charging RMB 3,000 to 8,000 per month. In contrast, decent domestic engineers in China typically demand RMB 30,000 to 40,000, with some even commanding 70,000 to 80,000. In reality, special effects companies are struggling to survive, and this is a global issue. In some countries and regions with higher labor costs, the special effects industry can receive government subsidies. For example, Canadian special effects companies can obtain subsidies covering 40% of project labor costs, which is unlikely for us.

I believe China's post-production industry has reached a critical stage of survival. Many film and television projects have very limited budgets, which are insufficient to sustain the entire industry, let alone support its development.

Even if a company secures one or two profitable projects, the profits are not enough to cover personnel costs. During project gaps, the company continues to operate without income but still has to pay employee salaries. As a result, it's extremely challenging for special effects companies to retain staff, despite the need for a large number of highly skilled professionals. The current problem is that the earnings from a single project may not even cover two months of operational costs during downtime. 20社：如果未来不能使用Sora，会不会让行业雪上加霜？你有没有想过一些对策？

**米：**现在全球电影都处于复苏期，整个行业都很难。不单是中国，全世界都一样。前两天我听说迪士尼裁员7000人，也有AI的影响因素在。老外也在面临一样的问题，他们的后期制作团队也很难赚到钱。 The current issue is that after the widespread adoption of AI technology, it remains quite challenging to use it normally in China. Moreover, I am increasingly concerned that with the rapid development of AI and AIGC technologies, the gap in production capabilities between China and other countries will widen further. At present, there is no effective solution, and the only option is to keep striving to keep up.

20社: Domestic companies like Baidu and ByteDance also have similar text-to-image applications based on large models.

米: In the field of special effects, high-quality data is particularly essential for training, and we are actually quite lacking in this area. The relationship between Sora and data is akin to needing both an engine and fuel.

20社：你们最近虚拟拍摄做得不错，在电影和广告业都有新的作品，去年年底上映的《非诚勿扰3》用了你们的虚拟拍摄。

米： Virtual production involves using camera tracking and real-time rendering technology to display virtual digital scenes that match the perspective of the real camera on a large LED screen as the shooting background. During the filming process, the virtual scenes are synchronized with real actors or physical environments to achieve a 'what you see is what you get' effect. This technology is widely used in Hollywood productions, such as Disney's series The Mandalorian, which utilized it extensively for shooting scenes. However, the technology still has certain limitations, particularly in aspects like lighting design and spatial arrangement.

20社: I once watched a documentary where the actor playing Gandalf in The Lord of the Rings performed with a ball, and because it was so lonely, he ended up crying during the performance.

米: This is precisely why, after the advent of virtual production, Hollywood began using it extensively—actors can have a stronger sense of presence by directly seeing the visuals. 20社： Compared to Hollywood, the application of virtual shooting in domestic films is not as rapid.

米： Virtual shooting is still in its early stages in China. 'If You Are the One 3' chose to shoot at the Bird's Nest Villa in Sanya, but there were many uncontrollable factors, such as the unpredictable local weather—sunny one moment and rainy the next, with lots of clouds making it difficult to maintain scene continuity. Additionally, Director Feng wanted to experiment with color changes, but it's impossible to alter someone else's property. We filmed in the studio for 20 days, unaffected by external weather conditions like wind, rain, or typhoons. Every day, the team worked like clockwork, with all progress, risks, and ultimately the shooting process being fully controllable.

Virtual filming does come with certain prerequisites. In the industrialization of the film industry, one trend is the front-loading of post-production. For instance, the virtual filming we conducted is a prime example of this front-loading approach. We often emphasize that technology is merely a means, while filming is the ultimate goal. This requires the creative team, starting from the project planning phase, to have a profound understanding of the final visual presentation.

Since you're handling pre-production for post-production, you must first understand post-production techniques while also being knowledgeable about pre-production filming, visual effects, and the technical principles of real-time shooting. Only then can you truly meet practical requirements. In terms of overall effect, our integration of virtual and live-action filming is better than that of other domestic companies. This is because we have independently developed a lighting system that ensures seamless blending of lighting and color between the live-action and virtual parts, making the final product more immersive.

20社：Will the virtual production business be affected by Sora?

米： This is a technology that combines virtual backgrounds with real-life sets. Many people are concerned that with the emergence of Sora, traditional film and advertising industries might no longer need this technology, especially given the high costs associated with virtual production. International peers are also considering this issue, with greater anticipation for the new productivity brought by the combination of the two technologies.

A recently funded AIGC technology development team, PIKA, shared their views on the future integration of AIGC technology and film production at a recent technical seminar. They believe that combining LED virtual filming as a shooting method with AIGC as a background content generation technology can enhance the realism of virtual scenes, improve production efficiency, and reduce costs.

Specifically, AIGC technology can quickly generate various types of content, including scenes, characters, and special effects. When combined with LED virtual filming, it can significantly boost production efficiency. Filmmakers can adjust virtual scenes in real-time as needed, eliminating the long wait for post-production. Over the past decade, we have been facing various impacts. While we pay close attention to the development of AI, it is equally important to focus on our current work and keep pace with our global peers.