Many of us will have seen Queen Elizabeth II apparently delivering an improbable dance as part of an address to the nation or, perhaps, even tuned in to a very unusual phone call seemingly taking place between former US Presidents Obama and Trump. These are just some examples of so-called “deepfakes”, in which the face of one person in a video is convincingly replaced with that of another using sophisticated machine learning technology. Deepfakes are gaining in popularity and, while the technology has raised some existential questions about the threats of misinformation on our democracies, it has also found its uses in the film industry.
With the Oscars only days away, Netflix’s “The Midnight Sky” has captured the attention of the Academy’s visual effects panel with its cutting-edge computer-generated facial replacements, securing a nomination for Best Visual Effects.
A particularly impressive and unexpected use of face replacement technology was also showcased in HBO’s “Welcome to Chechnya”, the first documentary to be shortlisted in the Visual Effects category. It follows the stories of LGBT+ Chechens fleeing their country through an underground network of rescuers. Despite missing out on the Oscar nomination, the documentary has been hailed as a game-changer by critics and filmmakers due to its novel approach to identity protection. Director David France needed to protect the identity of both LGBT+ Chechens and their rescuers so as not to expose them to the Chechen authorities. Nonetheless, he was determined to preserve the emotional states and the fear that was part of their stories. It was only with face replacement technology that New York City-based LGBT+ activists could lend their likeness to LGBT+ Chechens and give them a chance to tell the world their stories with real faces displaying real emotions in a seemingly natural way.
Visual effects and patents
Undoubtedly, such outstanding results are only obtained with intensive work and high production costs. In some cases, film producers may use novel visual effects (VFX) technologies straight from their R&D departments for the first time, pushing the entire industry forwards. For these reasons, obtaining patent protection is potentially very useful in helping attract research investment and incentivise further development efforts. It can also be easier to rely on registered rights like patent protection than “softer” forms of intellectual property protection (such as trade secrets or confidentiality clauses) in the highly competitive and people-fluid movie business.
When it comes to patenting VFX technology in the US, McRO, Inc. v. Bandai Namco Games America Inc. (Fed. Cir. 2016), in which the generation of automated lip-synchronization and associated facial expressions for 3D animated characters was found to be patent eligible, has been highly influential and is generally seen as positive for those looking to patent VFX inventions. But what about Europe?
VFX inventions in Europe
At the European Patent Office (EPO), VFX inventions will normally fall within the definition of computer-implemented inventions (CIIs). Such inventions will not be formally excluded from patentability if a technical means (e.g. a computer) is defined in the claims. However, to establish the presence of an inventive step for a CII, it must be shown that the CII has one or more non-obvious features which have “technical character”, i.e. a "further technical effect” going beyond the normal physical interactions that would be expected between the program and the computer on which it is run. Ability to demonstrate this kind of technical effect is crucial, as failing this means the features concerned will be seen as “non-technical” and cannot be taken into account when assessing inventive step.
An effect will generally be seen as “non-technical” if it falls within one of the exclusions from patentability defined in the European Patent Convention. For VFX inventions, which typically relate to mathematical techniques to generate or change digital images, the most relevant exclusions are likely to be presentations of information, aesthetic creations and mathematical methods.
Presentations of information and aesthetic creations
As amazing as the output of a ground-breaking computer-implemented VFX technique might look, a claim defining the output without defining the technical features which produce the output may be seen by the EPO as providing merely a non-technical presentation of information or aesthetic creation. However, the EPO Guidelines for Examination make clear that if an aesthetic effect is obtained by a technical structure or other technical means, even though the aesthetic effect itself is not of a technical character, the means of obtaining it may be. They also teach that, although “presentations of information” concerns both the cognitive content of the information and the manner of its presentation, it does not extend to the technical means used for generating such presentations of information. Such a “non-technical” objection for VFX inventions might therefore be avoided at the EPO by defining the technical features which produce the VFX output in the claims.
The technical features which produce a VFX output will typically comprise a set of mathematical steps which take input data (e.g. pixel values indicating an input image) and convert it to output data (e.g. pixel values indicating an output image).
The EPO Guidelines for Examination specifically define that a mathematical method (which could be an artificial intelligence or machine learning technique) may contribute to the technical character of an invention if it serves a technical purpose. “Digital audio, image or video enhancement or analysis” is given as a specific example of an acceptable technical purpose. This would seem to bode well for VFX techniques which, by their very nature, are applied to digital images. The claims must, however, be functionally limited to this technical purpose in order for the technical contribution made by the mathematical steps of a new VFX technique to be recognised.
A mathematical method may also contribute to the technical character of an invention if the claim is directed to a specific technical implementation of the mathematical method and the mathematical method is particularly adapted for that implementation in that its design is motivated by technical considerations of the internal functioning of a computer. This may be useful if, for example, an invention allows a known VFX technique to be implemented on specific computer hardware resources more efficiently. In this case, the applied mathematical steps of the VFX technique may already be known and therefore cannot render the invention novel and/or inventive. Nevertheless, the claimed specific implementation may be new and may be able to contribute to an inventive step if it is associated with a technical effect. Ensuring the technical effect is clear from the description can be very helpful in convincing an EPO Examiner of the technical contribution made by the claimed invention during prosecution.
Drafting Tips for Europe
Appropriate drafting of the patent application is crucial when it comes to successfully patenting VFX inventions in Europe. It should be ensured that the claims define: (i) suitable technical means (e.g. a computer); (ii) the mathematical steps of the VFX technique; and (iii) how those mathematical steps are applied to generate a final image from input data. It is also advisable to ensure any technical effect relating to the specific implementation of the VFX technique is apparent from the application as filed. This can be achieved by explicitly referring to the technical effect in the description of the application and showing a link between this effect and the technical features of the claims. This is essential if, for example, the VFX technique is already known (or turns out to be following the EPO search) and the inventive step of the invention therefore lies in its specific technical implementation.
Visual effects are advancing at a staggering rate and are finding new, unexpected uses in industry. Patent protection can be a valuable tool to help encourage costly R&D efforts in this field. Taking care over your claiming strategy and making sure your patent applications conform to the EPO’s specific requirements for computer-implemented inventions may be a deciding factor in your chances of success at the EPO.
To chat about this article (or anything else patent-related, for that matter!) please contact Alessio, Arun or your usual Kilburn & Strode advisor.