Generative AI can not only simulate a playable version of Doom, but also the Nintendo classic Super Mario Bros., according to new research. On Wednesday, Virtual Protocols, a company that’s been developing AI-powered characters for games, debuted a research project called MarioVGG, which uses text-to-video generation to recreate a version of the 1985 Nintendo game. Its research paper concludes that AI-enabled video generation programs show real potential in providing a new way to develop games. “Trained on just a single consumer-grade GPU (an Nvidia RTX 4090), our text-to-video diffusion model, MarioVGG can generate controllable game frames that are consistent and follow game mechanics,” the paper adds.
MarioVGG was developed by training the AI model on 737,000 frames taken from a Super Mario Bros. session. This included identifying frames where the main character Mario “runs” or “jumps.”The model was then trained to generate copycat images of the Nintendo game based on input from the player, which arrives as a text prompt, either run or jump. “Specifically, our model takes an initial video frame of the game along with text of the desired action (e.g. ‘jump’), and learns to generate a sequence of frames that visually depicts the desired action,” the paper adds.
(Credit: Virtual Protocols)
The result has some lag and is not easily playable. According to the paper, MarioVGG “takes approximately 4 seconds to generate a 6-frame video sequence of the game at inference.” Each frame is also generated at 64-by-48 or a 128-by-96 resolution when real Nintendo games from the era feature a 256-by-224 resolution. Even so, the AI model can accurately simulate the 2D Mario experience, including the various mechanics, like the real 1985 game.“We observe that MarioVGG successfully simulates and generates the gravity mechanics of the game,” the paper says. “The generated video demonstrates Mario falling off a platform as the ‘Run’ action is used, all this while maintaining consistency of the surrounding environment and generating coherent new tiles.”
(Credit: Virtual Protocols)
In addition, MarioVGG will generate its own unique Mario levels, with the first input frame the only one “grounded from a true game state.”But the model still faced some limitations. This included MarioVGG failing to always follow the expected behavior and rules of the game when generating the video frames. For example, “levels and objects in MarioVGG are not controllable and are generated procedurally,” the paper says. Another issue is that the AI model will keep rendering video frames even after Mario dies, rather than revert to the last checkpoint.
Recommended by Our Editors
That raises questions over how an AI-generated game will work like a traditional computer game, which can involve tracking the character’s stats and progress through a level. “An open research question is how terminal states are decided in a fully generative game that has no ‘ground truth’ equivalent,” the paper says. Still, Virtual Protocols says its research offers a glimpse at future applications of generative AI for game development, pointing to how MarioVGG only used a relatively small sample of training data and one consumer-grade GPU. “The era of infinite interactive worlds is here,” the company wrote in a tweet.
Get Our Best Stories!
Sign up for What’s New Now to get our top stories delivered to your inbox every morning.
This newsletter may contain advertising, deals, or affiliate links. Subscribing to a newsletter indicates your consent to our Terms of Use and Privacy Policy. You may unsubscribe from the newsletters at any time.
About Michael Kan
Senior Reporter
I’ve been with PCMag since October 2017, covering a wide range of topics, including consumer electronics, cybersecurity, social media, networking, and gaming. Prior to working at PCMag, I was a foreign correspondent in Beijing for over five years, covering the tech scene in Asia.
Read Michael’s full bio
Read the latest from Michael Kan
