NitroGen: Open AI Foundation Model for Gaming Agents
Introduction
Gaming has long been a playground for artificial intelligence research. From simple rule‑based bots to sophisticated reinforcement‑learning agents, developers continually push the boundaries of what a computer can do in a virtual world. NVIDIA’s NitroGen takes this ambition a step further: an open foundation model that can play a wide range of video games straight from pixel input.
In this article we explore NitroGen’s design, training data, and practical usage. Whether you’re a researcher, hobbyist, or a game developer curious about AI, you’ll find a clear roadmap for getting NitroGen up and running on your machine.
What is NitroGen?
- Open‑source foundation model: The code and weights are freely available on GitHub and Hugging Face.
- Generalist gaming agent: NitroGen is not tuned for a single title; it learns from a massive video‑action gameplay dataset sourced entirely from the internet.
- Behavior cloning architecture: The model mimics human play by reproducing the actions that produced the recorded gameplay frames.
- Pixel‑to‑control: Input is raw RGB pixels; output is discrete gamepad commands suitable for most Windows games.
The project’s research paper (Magne et al., 2026) details the dataset construction and model architecture. The open‑source release allows the community to adapt, extend, and validate the model on new games.
Training & Data
NitroGen was trained on over 1 million hours of gameplay captured from publicly available videos. Key points:
- Dataset size – Massive scale is critical for a generalist model. The diversity of games, camera angles, and player styles teaches the network to generalize.
- Behavior cloning – For each frame, the original action labels (e.g., button presses) are extracted. The model learns to map image → action.
- Normalization & augmentation – The raw frames are resized to 84×84 and normalized. Data augmentation includes random cropping and brightness changes to improve robustness.
Because the model was trained entirely on internet video, the code is entirely free of copyrighted game binaries. You provide the executable in a subsequent step.
Installation Guide
Below are step‑by‑step instructions for getting NitroGen on a Windows laptop or a Linux server.
1. Clone the Repository
git clone https://github.com/MineDojo/NitroGen.git
cd NitroGen
2. Create a Python Environment
We recommend using Python ≥ 3.12 (see the repo’s pyproject.toml).
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -e . # Installs dependencies and the local package
3. Download the Pretrained Checkpoint
The checkpoint lives on Hugging Face under the nvidia/NitroGen repository.
hf download nvidia/NitroGen ng.pt
ng.pt can be moved wherever you prefer; the server script expects a path to it.
4. Run the Inference Server
python scripts/serve.py <path_to_ng.pt>
8000 and listens for inference requests.
5. Play a Game
You must have a Windows game installed. Find the exact executable name with Task Manager (Ctrl + Shift + Esc) → Details → Right‑click → Properties.
python scripts/play.py --process 'mygame.exe'
mygame.exe with the actual process name. The script captures the screen, sends frames to the inference server, and translates predictions into controller inputs.
Tip: If your game uses DirectX 11 or greater, NitroGen’s screen‑capture routine performs well. For low‑frame‑rate titles, consider increasing the capture frequency in
scripts/play.py.
Key Features & Limitations
| Feature | Description |
|---|---|
| Cross‑Game | Works with any Windows game as long as it runs as an .exe. |
| Python API | Simple server‑client architecture; can be integrated into custom pipelines. |
| Open‑source | No closed components; research code is available under the MIT license. |
| No game binaries | You supply the game; no licensing conflicts. |
| Windows‑only | Games must run on Windows; server inference can run on Linux. |
| Behavior cloning limitations | If the game’s action set differs from training data, you may need fine‑tuning. |
Extending NitroGen
- Fine‑tuning on a New Title – Gather a small set of gameplay videos for the target game and use the provided
train.pyscript to adapt the model. - Custom Action Spaces – Modify the
action_spacedefinition in the training config to match a new controller schema. - Multi‑CPU/ GPU – The server is single‑threaded; you can deploy multiple instances behind a load balancer for higher throughput.
For detailed instructions, refer to the docs/ directory in the repo.
How to Contribute
- Bug Reports – File issues on GitHub.
- Pull Requests – Code improvements, new training recipes, or additional demo scripts. Please follow the contribution guidelines in
CONTRIBUTING.md. - Community Discussion – Join the NVIDIA AI Research Slack channel or the
#nitrogenDiscord for real‑time help.
Summary
NitroGen demonstrates that a single foundation model can handle a surprisingly wide range of games at the pixel level. By building on an open‑source repo, NVIDIA invites the research community to test, extend, and remix the technology.
Whether you’re testing the limits of behavior cloning or want a ready‑made agent for a game‑testing pipeline, NitroGen offers a solid, well‑documented starting point. Clone the repo today, run your first game, and join the conversation around open gaming AI. Happy gaming!
Citation (for academic use):
@misc{magne2026nitrogen,
title={NitroGen: An Open Foundation Model for Generalist Gaming Agents},
author={Loïc Magne and Anas Awadalla and Guanzhi Wang and Yinzhen Xu and Joshua Belofsky and Fengyuan Hu and Joohwan Kim and Ludwig Schmidt and Georgia Gkioxari and Jan Kautz and Yisong Yue and Yejin Choi and Yuke Zhu and Linxi "Jim" Fan},
year={2026},
eprint={2601.02427},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2601.02427},
}