UE4 360 Stereo Image and Movie Capture
Back in 2016 I did some research into this topic, coming from a motion picture background this stuff really interests me, especially as real-time rendering and visualization effects are becoming more practical and widely used.
The need to capture 360 video and images is still a thing. Surprisingly not much has changed over the last four years.
The two strongest and most easily available are the 'Panoramic Capture' plugin by Kite and Lightning devs, and Nvidia's 'Ansel' plugin.
The Panoramic Capture plugin for UE4 allows you to capture still images or movies that can then be viewed or played back in stereo, giving the illusion that the world is being rendered in real time. In the following sections, we will take a look at how to set up and use the Panoramic Capture plugin and also take a look at how you can view the stereo images you create.
https://docs.unrealengine.com/en-US/Platforms/VR/DevelopVR/StereoPanoramicCapture/index.html
https://docs.unrealengine.com/en-US/Platforms/VR/DevelopVR/StereoPanoramicCapture/QuickStart/index.html
Both can be enabled in the UE4 plugin menu. A restart is required after.
Panoramic Capture
The prebuild Blueprint is found in the 'PanoramicCapture Content' folder
The Blueprints can be full customized for your project requirements
The need to capture 360 video and images is still a thing. Surprisingly not much has changed over the last four years.
The two strongest and most easily available are the 'Panoramic Capture' plugin by Kite and Lightning devs, and Nvidia's 'Ansel' plugin.
The Panoramic Capture plugin for UE4 allows you to capture still images or movies that can then be viewed or played back in stereo, giving the illusion that the world is being rendered in real time. In the following sections, we will take a look at how to set up and use the Panoramic Capture plugin and also take a look at how you can view the stereo images you create.
https://docs.unrealengine.com/en-US/Platforms/VR/DevelopVR/StereoPanoramicCapture/index.html
https://docs.unrealengine.com/en-US/Platforms/VR/DevelopVR/StereoPanoramicCapture/QuickStart/index.html
Both can be enabled in the UE4 plugin menu. A restart is required after.
Panoramic Capture
The prebuild Blueprint is found in the 'PanoramicCapture Content' folder
The Blueprints can be full customized for your project requirements
This is the ue4 default:
This is my custom Blueprint for 60FPS 360 Stereo at 8k, 8 seconds
Mono or Stereo mode (0,1)
360 Single Frame or movie modules. Simply change the number at the end for the required number of frames output. (Example. 8 seconds of video at 60 frames per second will have an output of 480. Each frame at 8k takes about 6 minutes. That's a total render time of approximately 2880 minutes, or 48 hours! )
360 Stereo image example
360 Stereo Movie, 8 seconds at 60fps in 8k coming soon!
360 Single Frame or movie modules. Simply change the number at the end for the required number of frames output. (Example. 8 seconds of video at 60 frames per second will have an output of 480. Each frame at 8k takes about 6 minutes. That's a total render time of approximately 2880 minutes, or 48 hours! )
360 Stereo image example
360 Stereo Movie, 8 seconds at 60fps in 8k coming soon!
Oh yeah! 480 frame later we have a short video. However, the original working composite from After effects is nearly 80gb so some compression editing is needed. :)
Media Encoder has some issues with 8k and there is not way for me to successfully test the video as nothing (I have) will play it. I'll need to reduce it to 4k to view in VR. A successful test at 363 MB shows three things. 60fps is beautiful and super smooth. Moving in VR needs to be done much slower as this feels like a roller coaster. 8 seconds is not nearly enough time.
You can watch the video here.
Whilst Panoramic Capture is great there are some points to consider:
'Ansel' was great great for taking detailed 360 stereo images with some powerful tools. Taking a perfect 16k image in 360 with stereoscopy was totally possible. However, movie capture is not a selectable option and using blueprints to control Ansel is also impossible for multiple single frame capture. Manually moving a frame at a time (60 frames per second) would be impractical, but running a keystroke macro can solve this issue and you can just leave it running for a few hours (or days).
Sadly, there seems to be some issues with later versions of UE4 and I continuously get this error message.
Surreal Capture software
The final and newest option, is to use a sideloaded application with your UE4 project called 'Surreal Capture'.
Whilst Panoramic Capture is great there are some points to consider:
Not all post process will work.
Does not capture sound
Some frames will have errors / artifacts and will need to be re-rendered
Dynamic lighting can cause issues, scenes work best with built lighting
Particle effects in cascade wont render correctly, however Niagara effects should.
All moving actors will need to be set up on a sequencer and move alone splines.
No free camera movement, the camera needs to move around scene but not rotate to look at objects.
The single image files will need to be composited into a movie file and sound added.
A great video demonstrating set up and using this plugin:
Does not capture sound
Some frames will have errors / artifacts and will need to be re-rendered
Dynamic lighting can cause issues, scenes work best with built lighting
Particle effects in cascade wont render correctly, however Niagara effects should.
All moving actors will need to be set up on a sequencer and move alone splines.
No free camera movement, the camera needs to move around scene but not rotate to look at objects.
The single image files will need to be composited into a movie file and sound added.
A great video demonstrating set up and using this plugin:
Rendering Notes:
Set the Project Setting to the required FPS
Run in standalone mode
Nvida's Ansel
'Ansel' was great great for taking detailed 360 stereo images with some powerful tools. Taking a perfect 16k image in 360 with stereoscopy was totally possible. However, movie capture is not a selectable option and using blueprints to control Ansel is also impossible for multiple single frame capture. Manually moving a frame at a time (60 frames per second) would be impractical, but running a keystroke macro can solve this issue and you can just leave it running for a few hours (or days).
Sadly, there seems to be some issues with later versions of UE4 and I continuously get this error message.
Great news!
There is a brutal hack that can be performed to create real-time 360 video using Ansel
It will need more time to investigate and test on later version of Unreal Engine, but it looks very promising. I'm guessing this is gonna crush the GPU. :)
Read the article here:
Watch the test video here:
The output file is still hundreds (if not thousands) of image files which will need to be composited together to make a movie, with sound fx being added later.
Surreal Capture software
The final and newest option, is to use a sideloaded application with your UE4 project called 'Surreal Capture'.
https://www.surrealcapture.com/
The free 15 days demo is easy to use and works straight away with some options to adjust depending on your requirements.
The benefit of using Surreal capture is that it creates a video in real-time, no need to wait for each frame to render and then composite.
Real-time rendering is definitely the future, however there are currently some trade-offs. Even having a powerful computer to process the video whilst playing is impressive but there are some things that re currently impossible.
I found it incredibly easy to set up and use, however with a maximum resolution of 4k (2 for each eye) is too low, also I could never get the post process effect to capture, and stereoscopic video was not working for left and right eye, so it fails to meet my requirements. This is perfect for gamers who maybe want to capture some epic gameplay scene and share on social media, or for players in a VR arcade who want to watch their game again.
The free 15 days demo is easy to use and works straight away with some options to adjust depending on your requirements.
The benefit of using Surreal capture is that it creates a video in real-time, no need to wait for each frame to render and then composite.
Real-time rendering is definitely the future, however there are currently some trade-offs. Even having a powerful computer to process the video whilst playing is impressive but there are some things that re currently impossible.
I found it incredibly easy to set up and use, however with a maximum resolution of 4k (2 for each eye) is too low, also I could never get the post process effect to capture, and stereoscopic video was not working for left and right eye, so it fails to meet my requirements. This is perfect for gamers who maybe want to capture some epic gameplay scene and share on social media, or for players in a VR arcade who want to watch their game again.
