arduinomanceron Hacker News • Recommends
On “Neural Rendering: How Low Can You Go in Terms of Input?”
The video linked of applying a neural net to GTAV for photorealism is really impressive if you haven't seen it. link to this video
0:00 / 0:00
#photorealismenhancement#neuralrendering#gtav
Enhancing Photorealism Enhancement
ISL and Collaborators
12K subscribers • 1 video on Hallway
- Recommended by 2 people: 2 on Hacker News.
- Linked in the descriptionisl-org/PhotorealismEnhancementarXiv 2105.04619
Intermediate. Assumes working knowledge.
6.2M views5 years ago2 recommendations
Graphics and audioAI and machine learningRenderingGPUs, shaders and CUDANeural networks and deep learning
Intel researchers present a convolutional network that enhances the photorealism of synthetic images, demonstrated on Grand Theft Auto V footage.
Enhancing Photorealism Enhancement Stephan R. Richter, Hassan Abu AlHaija, and Vladlen Koltun Paper: arxiv.org/abs/2105.04619 Code and data: github.com/isl-org/PhotorealismEnhancement Project page: isl-org.github.io/PhotorealismEnhancement/ We present an approach to enhancing the realism of synthetic images. The images are enhanced by a convolutional network that leverages intermediate representations produced by conventional rendering pipelines. The network is trained via a novel adversarial objective, which provides strong supervision at multiple perceptual levels. We analyze scene layout distributions in commonly used datasets and find that they differ in important ways. We hypothesize that this is one of the causes of strong artifacts that can be observed in the results of many prior methods. To address this we propose a new strategy for sampling image patches during training. We also introduce multiple architectural improvements in the deep network modules used for photorealism enhancement. We confirm the benefits of our contributions in controlled experiments and report substantial gains in stability and realism in comparison to recent image-to-image translation methods and a variety of other baselines.Why it's here
2 people recommend this, 1 disputes itWhat people say. Recommended as an impressive demonstration of using neural networks to enhance low-resolution or synthetic game graphics in real time.
valgazeon Hacker News • Recommends
On ““Can it run Doom” will never be the same, thanks to new ray-tracing mod”
DLSS & ideas like it outside an idea are really interesting: link Ex. run the base game at an insanely crummy/low resolution. On low(er) it’s possible to run a model to kinda “upscale” in each frame in a performant way. Or even this CNN initiative from Intel where they took Grand Theft Auto & “painted” it with dashcam footage from rural Germany: link to this video The results are pretty interesting— all “brown” southern CA mountains were green like in Germany
ameliuson Hacker News • Disputes
On “Enhancing Photorealism Enhancement”
I noticed some strange (unrealistic) effects on the back of the red car at the 8:19 mark in this video: link to this video
thih9on Hacker News • Mentions
On “Enhancing Photorealism Enhancement”
From the "Enhancing Photorealism Enhancement" video 1: "It is a convolutional network which produces images frame by frame and can be run at interactive rates.". 1: link to this video (0:19)
L_on Hacker News • Mentions
On “Enhancing Photorealism Enhancement”
There's also a version trained on Mapillary Vistas, which has images from all around the world. Results from that are at the bottom of the page and at the end of the video ( link to this video )
wruzaon Hacker News • Mentions
On “Software Dev Can't Be Automated – It’s a Creative Process with an Unknown Goal”
Likely not, but that was not my point either, it was just an non-stretchable analogy. AI is not a single network usually, it is a pipeline of networks of different kinds, see e.g. link to this video (seek to 1:35 for beginning of an architecture overview). You can’t call it “just pattern matching” without fetching it too far.
valgazeon Hacker News • Mentions
On “Stable Diffusion with Core ML on Apple Silicon”
Video is really a series of frames, the framerate for film/human can get away with 24 frames/second-- so maybe ~40ms/image for real-time at least? What's cool about the era in which we live is if you look at high-performance graphics for games or simulations, for instance, it may in fact be faster to a the model to "enhance" a low-resolution frame rather than trying to render it fully on the machine. ex. AMD's FSR vs NVIDIA DLSS - AMD FSR (Fidelity FX Super Resolution): link - NVIDIA DLSS (Deep Learning Super Sampling): j link AMD's approach renders the game at a crummy, low-detail resolution then each frame uses "upscales" Both FSR and DLSS aim to improve frames-per-second in games by r…
FoodWThrowon Hacker News • Mentions
On “How do I become a graphics programmer? – A guide from AMD Game Engineering team”
> AI will be doing graphics programming? I doubt that. I think the bigger concern (if you could call it that) is that certain AI techniques may be able to replace entire stacks once they get advanced/stable enough. Something along the lines of this perhaps, where you render simplified geometry and let the AI fill in the blanks: link to this video Even without experimental technologies, you can get a glimpse of how shipped tools like ray reconstruction can morph the field in the future, forever entrenching themselves into graphics programming one way or another. As far as AI writing graphics code? No way, at least not in the next couple of decades. Button snippets are a far cry from rendering virtualize…
joegibbson Hacker News • Mentions
On “Sora: Creating video from text”
I think for the near future we’ll see something like this: link to this video From a few years ago, where the game is rendered traditionally and used as a ground truth, with a model on top of it that enhances the graphics. After maybe 10-15 years we will be past the point where the entire game can be generated without obvious mistakes in consistency. Realtime AI dialogue is already possible but still a bit primitive, I wrote a blog post about it here: link
meinersburon Hacker News • Mentions
On “Games Look Bad: HDR and Tone Mapping (2017)”
Researchers trained a deep learning network on photos to make GTA5 more realistic-looking: link to this video I was surprised that most of it is is just tone mapping. It looks too green-ish to me, needing some white balancing, but realistic. I think Video games are intentionally over-saturated and contrasty to make them look better, just as paintings are1. I took many screenshots in Horizon Zero Dawn that would be look bland if it was realistic like the modfified GTA5 footage above2. You also don't want animation movies3 with realistic tone mapping. But for a horror game you would do exactly that, it's about immersion (or you would never be frightened), not beauty. 1 link 2 link 3 [link…