One of the most popular open-world games by Rockstar Games is Grand Theft Auto V, or GTA V. Grand Theft Auto 5 is a fine addition to the Grand Theft Auto Franchise. Generating $800 million alone on the first day and earning $1 billion after three days of release, it has got a place in the list of best-selling games. Its story involves the three main protagonists: Franklin, Michael, and Trevor, and the graphics were praised a lot. Modders have made many graphical overhauls, such as Enhanced Natural Beauty (ENB) for GTA V, making it more graphically impressive. But still, after using the overhauls, GTA V looks like a game. Well, not anymore.

Imagine playing the game inside a photo. That is what this project makes you feel like. A group of researchers has applied an enhanced photorealism makeover to Rockstar Games’ Grand Theft Auto 5 using AI machine learning. A new project from Intel Labs has been started called  “Enhancing Photorealism Enhancement,” which takes realism in an unsettlingly photorealistic way. The purpose of the project is to enhance photorealism in games. They used Grand Theft Auto 5 to illustrate their latest photorealism enhancement, and the results are impressive.

“We present an approach to enhancing the realism of synthetic images.”

The processes researchers Stephan R. Richter, Hassan Abu Alhaija, and Vladlen Kolten produced a surprising result, very much like the visual look you would get if you took photos from the smudged window of your car. The stills taken are impressive, but the videos uploaded while testing in GTA V are phenomenal. The slightly faded lighting, smoother pavement, and realistically reflective cars feel like you are looking out your window in real life, watching traffic move gracefully, making you forget that it is a game.

“The images are enhanced by a convolutional network that leverages intermediate representations produced by conventional rendering pipelines. The network is trained via a novel adversarial objective, which provides strong supervision at multiple perceptual levels.”

In simple words, images from the game are rendered to a separate dataset, called Cityscapes. This enables the researchers to make every image in the game photorealistic. Intel ISL uploaded a video to YouTube delineating the approach behind its photorealism enhancement. The footage is hugely differing from what the players are used to seeing as it has approached a photorealistic environment. With the enhancement, it feels like footage of actual cars not of the game.

The researchers say their enhancements go further than what other photorealistic conversion processes can be by integrating geometric information from GTA V itself. ‘G-buffers’ can include data like the distance between objects in the game and the camera, and the quality of textures, like the glossiness of cars.

“We analyze scene layout distributions in commonly used datasets and find that they differ in important ways. We hypothesize that this is one of the causes of strong artifacts that can be observed in the results of many prior methods. To address this, we propose a new strategy for sampling image patches during training. We also introduce multiple architectural improvements in the deep network modules used for photorealism enhancement. We confirm the benefits of our contributions in controlled experiments and report substantial gains in stability and realism in comparison to recent image-to-image translation methods and a variety of other baselines.”

The modifications by our method are geometrically and semantically consistent with the original images.

It greens the parched grass and hills in GTA’s California:

It adds reflections to the windows and increases the fresnel effect (e.g., at the roof of cars):

It rebuilds the roads:

This project uses machine learning to a whole new level to create photos of the game identical to real-life through shading and then use these photos to create a video-like structure running at 20 – 24 fps. This thing shows the true potential of AI in this modern world.

There have been instances where a game has benefited from another kind of machine learning — AI upscaling. The process of using machine learning smarts to increase graphics to higher resolutions but has been featured in Nvidia’s Shield TV and in several different mod projects focused on upgrading the graphics of older games.

If this is later released as a mod, it will be elevating the graphics of the games to a whole new level. It may also take away the feel of a game as it is too realistic. Games are usually vibrant, which is a whole lot different from this project. We can take GTA V as an example; the game on which the project is being tested looks different from the project results.

Grand Theft Auto expects a new audience, but there has been no official announcement, so there is no knowing what Rockstar is planning to do. Rumors are that it may be released in 2023 and set in Miami. We may see similar graphics in GTA 6 but not too realistic or dull like this as Rockstar would not want to take away the game’s feel, would they?

Source: https://intel-isl.github.io/PhotorealismEnhancement/