Broadcasting companies and film studios alike are beginning to explore the vast potential of modern technology to bring a new generation of film industry to our televisions and cinemas. Artificial intelligence, machine learning and deep learning are buzzwords that excite video executives with the promise of revolutionary new possibilities for creating and editing video.
Deep learning, in particular, is a new frontier for the video industry, allowing the video professional to do everything automatically that would have taken weeks of work in the past, as well as some things that would have been impossible at all. How is deep learning different from other machine learning algorithms? And what are its practical applications for television and film? What is science and its business results?
Artificial intelligence, machine learning and deep learning
Artificial intelligence is an attempt to make a computer look like it has intelligence. The computer can be told exactly what to do in a given situation, in which case it has learned nothing. Machine learning seeks to allow a computer to learn how to perform certain tasks. There are many methods for doing this, and almost all of them rely on the computer, constantly changing the parameters through a process of trial and error. One of the more sophisticated ways to do this is by mimicking the neurons of a biological brain. When we make these artificial brains, or neural networks, more complex, we have deep knowledge.
Deep learning allows a computer to take something as complex as input, such as pixels in a video frame, and output something as complex as all the pixels in a new, modified, frame from the video. For example, they may be showing frames with unwanted grain as input, and its output compared to pure frames. Through trial and error, he will learn to extract grain from the input. As more and more images are passed through it, he may learn to do the same for images he has never been shown.
- Find the best big screen TVs here
Perhaps the first impressive use of deep learning was when Google trained a neural network to play Go, a famously difficult and challenging board game. The game is too complex for human instruction to create a viable opponent, and one neural network layer is never enough. Deep learning has become possible.
Deep learning is used for a wide range of other tasks. It is used to match generated speech to human speech, so a text-to-speech program will sound more natural. In similar tasks, she used translation from teaching computers how to translate from one language to another. The self-driving car that several companies are working on is driven by deep learning. Marketing departments use it to study shopper habits and think about how a customer will behave and what strategies they will best respond to. Digital assistants use it to better understand the requests we make of them.
Deep Learning for TV and Film
There are many opportunities to apply deep learning technology in the fields of video production, editing and cataloging. But the technology isn't limited to automating repetitive tasks; it can also enhance the creative process, enhance video quality and help maintain the massive video archive that many studios keep.
Generation of video editing and
Warner Bros. recently had to spend $25 million on reshoots for 'Justice League' and part of that money went to the digital mustache removal that star Henry Cavill grew up to be unable to shave due to overlapping commitments. It's not just Justice League - the post-production stage of any movie is time-consuming and costly. Deep learning will be a game changer for these types of tasks.
Consumer grade, easy to use solutions like Flo let you use deep learning to automatically create a video describing what you want in it. The software will find matching videos from your library and edit them together automatically.
Google has a neural network that can automatically separate the foreground and background of a video. What used to require a green screen can now be done without special equipment.
Deepfakes have hit the news quite a bit lately - when one person's face is on video to another, in a similar way, deep portraits that apply movement to still paintings like the Mona Lisa. The potential for using this technology in special effects is vast.
For example, the mustache issue at Warner Bros., which drew Henry Cavill into a controversy with fans. Cavill needed to grow a mustache for Mission: Impossible: Fallout, and at the same time needed to reshoot for Justice League. Cavill mustache for Fallout, but you have to be clean-shaven Superman. He chose to keep his mustache, so the Justice League editing team had to digitally remove the hairy lips for every scene he reshoots.
Unfortunately, this was noticed by the fans and it caused a stir. If work-from-home enthusiasts can put Nicolas Cage in films he's never been to using deep learning tools, one can only guess how much time and money Warner Bros. could have saved by replacing Henry Cavill with older footage himself.
Recovery Video
According to the Los Angeles Film and Television Archive, nearly half of all films made before 1950 have disappeared. Worse, 90% of the classic film prints that exist are in poor condition. The process of restoring these films is long, tedious and expensive. This is an area where deep learning is going to make a big difference.
The process of coloring black and white footage has always been lengthy. There are thousands of frames of frames in a movie and coloring each one takes a long time. Even with advanced tools, this process can be so automated. Thanks to NVIDIA technology, deep learning can dramatically speed up the process, with tools that only require an artist to color a single frame of a scene. From there, deep learning network automatically handles the rest.
Previously the show-stopping problem was missing or corrupted frames from the video. You can't reshoot something that happened many years ago.
Restoring that type of film used to mean editing the missing footage. Now, deep learning networks with Google's goal to change that. They have developed a technology that can actually recreate part of a scene based on the start and end frame.
Face/Object Recognition
By detecting the face of everyone in the video, deep learning will allow you to quickly classify video collections. You can, for example, search for any clip or movie that has a specific artist. Alternatively, you could use technology to calculate the exact time for each actor in a video. Recently, Sky News used facial recognition to identify famous faces at a royal wedding.
The technology isn't just limited to face detection, though; sports broadcasts rely on people's cameras to follow the movement of a ball, or to reveal other key elements in a game, such as a goal. Using object recognition, AI-powered tools can be used to automate sports broadcast production.
Video Analysis
While Flo can determine what a scene is and use that data to create a video of whatever you want, that same technology can be used to sort and categorize videos to make it easy to find specific footage by simply searching for the people or actions that appear in him.
This can be used to detect and remove objectionable content from videos to ensure they remain suitable for the target audience. In a similar vein, it can be used to match new videos with old videos that a person has expressed interest in and provide them with a personalized list of recommendations.
Better than streaming
As we move into 4K streaming, and TV manufacturers start rolling out 8K displays, streaming is using more data than ever before. Anyone with a bad connection knows what the problem might be. The shiny 4K display utility will be weakened if your internet connection cannot handle the bandwidth to take full advantage of it. Thanks to neural networks that can recreate high definition frames from low definition input, we could soon be streaming definition streams over our internet while still enjoying the high definition glory that our screens are capable of.
Future
The deep meaning of learning into film and broadcast has only begun to bite off the edges that this will be used in the future. I believe that his future in the video industry is especially bright. However, like all new technologies, deep learning is not without its drawbacks. As with deepfakes or the face of misrepresentation, there are legitimate privacy and trust concerns that arise from the rapid development of this technology.
As with any new technology, the industry needs to address a number of issues. The video industry and tech professionals must come together to develop standards for what tomorrow's new normal might look like. However, with the right approach, the benefits of this other than the elements will be greater than can be imagined now, and with the advent of sound film and color film and made before it, deeply study film and television to a new level.
Max Kalmykov is Vice President for Media and Entertainment at by DataArt.
- Find the best data visualization tools