Matchmoving a Stereo Project using 3DEqualizer and Maya
  • Matchmoving a Stereo Project using 3DEqualizer and Maya by Albrecht Steinmetz - Weta Digital Ltd.

Matchmoving is one of the invisible effects in this industry. It is vital to combine CG and live-action. Weta Digital's camera departmenthad a challenging role to deliver over 300 stereoscopic matchmoves forAvatar.
Albrecht Steinmetz
Albrecht Steinmetz, who has been working as a CG Artist and Matchmover on a large range of TV series and award-winning films (including the Lord ofthe Rings Trilogy, King Kong and Lovely Bones to mention a few), worked as Senior Matchmover on Avatar. Albrecht explains some of the challenges the camera department faced on Avatar:

"I have always been fascinated by integrating CG into real life footage. Robots or any type of creature interacting with real actors. Extending film sets with CG environments to make them look the size of a city or bigger. For Avatar it was mostly the other way round and we had to integrate real actors into a CG environment. And that in stereo which means instead of one plate we had to deal with two plates mimicking the human perception. We hadn't worked on a stereo project before and the fact that the cameras' interocular, convergence and focal length could change at the same time was a very new concept.

While the actual Fusion 3D Camera System (developed by J.Cameron and VincePace)  was a rather complex machine, our stereo-rig in Maya could be represented by two similar cameras more-or-less parallel to each other.

We had to sensibly recreate a stereo-rig that reflected the real cameras'motion, interocular-shift, change in convergence and zooming which took a good deal longer than matchmoving plates for a non-stereo show.
With 3DE V3 we had some initial difficulties adjusting for minor differences between the left-eye/right-eye of the plates that we could overcome with 3DEqualizer4. Most importantly the ability to simultaneously calculate focal length, depth shift, vertical shift and distortion for both cameras at the same time. Initially this calculated for hours but with 3DEqualizer4's adaptive calculation this was reduced down to minutes! This way we could create a set of cameras that exactly matched the plate. From here we could deliver our setups to animation and compositing.
When it comes to determine the scale of a 3D setup we rely on what we call 'survey'.  Survey is a set of 3D data that gives us precise measurements of sets and props. Usually tracking markers are spread on green screens and set environments. They are visually easy to pick up and get surveyed with a theodolite by our onset team. Then we get a set of 3D data for Maya and 3DE to help us solve our shots with the correct size and depth. Survey is a tremendous help to solve complex shots and stay true to the real camera.

We are very lucky to have survey for most of our sets. Occasionally we have shots with only a limited amount of survey. For such situations extensive photo-lineups within 3DE helped a lot: additional reference pictures taken by the onset team 'fill-up' our survey. To be sure we always use the correct lens settings for this photogrammetry within 3DE we read out the reference pictures'metadata and compare them with previously taken grid shots that we useto measure the lens used by the onset team.
Similarly we made grid shots for every camera used on set. Those measured values were used in 3DE to describe the lens distortion and could be used by compositing, too. This approach allowed us to solve very complex shots with dynamic i/o-changes and zoom for critical contact between CG creatures and real actors as well as set extensions.

For one sequence the survey and additional camera information had become unreadable. Since we needed a CG camera for set extensions and critical contact on moving objects within the scene, an iterative approach was needed to establish the right relation focal length, interocular and recalculated points. Comparing the plates with shots of other sequences we could already guess where we would have to start. We used these 'guestimates' as a starting point in 3DE then checked if the recalculated result still made sense.

One sequence was particularly exciting to work on:
We got a filmed set element and separately an actor filmed in front of a green screen. Both had to work perfectly together since we had to implement Neytiri with critical contact in both plates. Additionally the glass on a flexible rebreather that is first in Neytiri's hand then put onto Jake's face had to be added in CG and needed accurate matchmoving. The CG cameras for each plate were almost identical. However, since every tiny error in stereo shows up so much more than in anon-stereo film we had to correct for the slight difference in motion and convergence.
We decided to use one plate as the main plate and adjust the second plate by pre-stabilizing using an elaborate 3D/2D setup. That locked the plates together, corrected all differences in convergence and allowed for set extensions and critical contact of our CG character. Both plates could be combined now and a new CG stereo camera that worked for Neytiri with critical contact in both plates was created.
Additionally we had to add CG glass to the rebreathers to add environment reflections and for additional breathing condensation effects or even smashing glass. Most of these glass elements could be tracked in 3DE as object matchmove as part of our setup. Some of the shots required deforming glass because the rubber frame got stretched. Here I could still obtain a generally 'correct' move from 3DE. With a couple of extra tools in Maya I piped additional 2d tracks into my setup to animate and lock-in the animated deformations.

We work with a variety of tools to be able to deliver 3D setup scenes with CG cameras, dewarped plates, set geometry, rigid object matchmoves like guns or helicopters or matchmoves of deforming objects like skin or even water. But we also deliver 2d setups to combine various 2D elements (different miniature passes, etc.) into one shot. Cardrigs are something in between. With those we can place live action elements onto CG creatures or in CG environments anywhere we want to. The resulting changes in scale and perspective due to different distance sand angles to the CG camera are piped back to our compositing software.
On a non-stereo show we can cheat additional elements into place with 2D/3D tricks but have to be much more careful in stereo. Because of the fact that we see every element through two eyes everything has to be at the correct depth otherwise the resulting composition will look wrong. With relatively small elements like actors we got away with it quite easily by matching the convergence to the CG environment render. For close-ups or objects with a lot of parallax due to their size we would have to recreate the second eye by reprojecting the hero plate onto 3D dummy objects.

In some instances we tried to pre-vertically align the plate in comp. This proved to be useful before the release of 3DEqualizer4 since we couldn't properly calculate vertical offset or depth shift between the cameras then. Since 3DEqualizer4 we can calculate all these values in relation to each other at once and deliver a CG camera setup matching the original plate. This is a huge leap forward in solving cameras in stereo for critical contact. The vertical alignment happens as a last step in comp now.

In the old days we struggled with long shots in full resolution. Today we can easily load in huge stereo plates in HD or 2k without loosing any performance. That way we work with the most detail available and have a greater freedom to extract tracking markers off a blue/green screen using either the image controls within 3DE or alternatively Shake or Nuke if we believe the original scans have more image detail than those we use in 3DE. For animation we hand over CG cameras and dewarped plates with smaller resolutions to keep their workflow fast and easy.

We had quite a steep learning curve for this project and were stoked to see Avatar on the big screen in stereo."

Albrecht Steinmetz
Weta Digital Ltd
Please save your recent changes.
You Forgot to Accept the License Agreement
Please accept the Software License Agreement in order to proceed with your download.
Your script has been rejected! Please fix all issues listed below before uploading again. Thanks!