Multi-Method Approach to 3D Content Generation

Matt DeJohn and David Seigle, In-Three
With contributions from:
Rob Engle, Sony ImageWorks
Jason Goodman, CEO of 21st Century 3D
Lenny Lipton. CTO of RealD
Chris Townsend, VFX supervisor on “Journey to the Center of the Earth”
Chris Ward, President of Lightspeed

In-Three creates 3D content from 2D movies using its proprietary process called Dimensionalization®. You can read about how we do this on our web site. In addition the web site contains material that describes what it takes to create “perfect 3D”, that is, 3D that achieves a director’s vision and has no disturbing artifacts or disparities.

We are often asked to comment on ways to keep the cost of creating such content low. That is, for any given type of shot, which approach costs less, -- Dimensionalization or using computer generated images or stereo shooting (i.e., dual computer cameras or dual real cameras, respectively).

As a guide in discussing this question we have created a matrix describing the potential advantage of each of these techniques. The chart describes the types of shots that are most likely to be realized at a lower marginal cost and why. That is, it describes the approaches that might be worth gearing up for if there are enough particular kinds of scenes.

From In-Three’s point of view we know that we can Dimensionalize any 2D content to produce perfect 3D. However, if content contains a preponderance of certain types of shots, then it might be most cost effective for our client to use a “hybrid*” or “multi-mode*” approach to creating it. As an example, if a movie is to have a large number of long shots / establishing shots where the closest objects are fairly far away; such scenes might be captured most cost effectively with a stereo 2nd unit, while other scenes are captured in 2D for Dimensionalization.

While this chart deals with multiple-techniques in 3D creation, there are certain requirements where only one mode is appropriate:

• Stereo shooting: Live action sports and entertainment when multiple dual rigs can be set up and pre calibrated to capture “the best seat in the house” for any action.
• CG: Any full animation.
• Dimensionalization: Any legacy 2D content to be converted to 3D.

We have thought a lot about the content of this chart and have asked others to comment and contribute, but most of our direct experience is in Dimensionalization per se. If readers question the chart entries or have additions or comments we welcome their input.

* By hybrid we mean 3D layers within a shot produced using different techniques then composited together. By multi-mode we denote entire shots of a movie produced using alternate techniques. 9/2008

Optimum Approaches for Producing 3D Content
Using a Multiple Method Approach

Overview

This matrix of the best approach assumes the following: the 3D created must accommodate traditional cinematic styles, not cause discomfort, maintain depth continuity, and be a compelling and immersive experience. Dimensionalization® is the specific 2D to 3D conversion process cited in this document.

Note

Multiple categories listed below may apply to 1 shot. It is up to the director and VFX supervisor to weith their options.

Key

X			Dual-camera only approach is most appropriate.
	X		CG-3D only approach is most appropriate.
		X	Dimensionalization only approach is most appropriate.
X	X	X	One of the selected option is most appropriate (depending on the nature of the shot).
X	X	X	A hybrid approach employing all the selected options is most appropriate.

 

Shot / Situation	Best Approach to Create 3D			Reason
	Dual-Cam	CG-3D	DZN (2D to 3D)
OTS & Closeups			X	During dialogue scenes, over the shoulder (OTS) shots need to seamlessly cut together with close-ups (CU) and other OTS shots that may have been shot with a different lens. The internal depth of the actor should be similar to the internal depth of the other actor in the reverse shot or in CU. Also, the bounds of positive parallax should be similar. These are rigorous requirements, but they should be thought of as similar to keeping continuity of lighting between different angles. Dimensionalization is the best option to handle these types of shots because it is the best tool to maintain the depth continuity between shots. Dimensionalization provides full control over internal depth and the shot's range of parallax.
Long shot / establishing	X			Long shots and establishing shots often are shot with medium length lenses, with most objects a good distance from the camera, and can be done by a second unit. This means that comfortable and compelling depth can be cost-effectively achieved with dual-cameras. This makes dual cameras a good choice for this type of shot.
Telephoto lens			X	In 2D a telephoto lens tends to compress a scene because of it's narrow angle of view. This translates to 3D as well. This compression makes objects look flat. In stereoscopic photography the inter-axial would be increased to compensate for this, but this can often lead to too much overall parallax for comfortable viewing in a cinema. The control available through Dimensionalization means that a comfortable range of parallax can be maintained while obtaining pleasing internal depth for the subject.
Complex organics	X			If complex organics like trees, bushes, or a field of grass are prominently placed in the frame one should be able to perceive extraordinary nuance in depth. Every leaf on a tree exists at a different point in Z-space, meaning that there are many thousands of levels of perceivable depth. Because dualcameras mimic the human eye for the 3D effect, they can easily capture this detail and are the appropriate choice for this type of shot.
Composite shots			X	Composited shots are built from a wide variety of sources. Matte paintings, CG characters, particle effects, fire elements, and green screen photography are often used to create one single shot. Each of these elements needs to look like they were photographed at the same time in the real world. This is a daunting task even in 2D. In 3D one now has to ensure that every element is generated or captured with the appropriate internal depth relative to all other elements in the scene. In addition to that fact, FX houses often use elements, such as fire, found in their own library. These likely only exist in 2D. These facts make Dimensionalization the best choice for composite shots because every element is given the depth appropriate for its part in the shot as a whole. One caveat to this is 1 or 2 main elements can be captured or generated with two perspectives and the rest of the elements can be Dimensionalized to accommodate them.
Specular highlights			X	Shots with specular highlights are sometimes hard to understand or are uncomfortable if they are captured with dual camera rigs. This is due to a variety of factors, one of which is that each camera captures a slightly different highlight because of their different position in space. This can lead to L/R eye discrepancies. This makes using Dimensionalization the only way to ensure that specular highlights don't discomfort.
Concert or Park Lights			X	Similar to specular highlights and reflections, dual cameras recording from different perspectives means one will get a different glare in each camera. This difference is often a star pattern where the L eye will have one point on top of late where the other will not. This kind of discrepancy can be quite confusing and uncomfortable. This problem can be alleviated with stereo-painting, but the only way to avoid this problem is with Dimensionalization.
Fully CG environment		X		Fully CG environments, whether it is from a fully animated movie or just a particular shot, are generated in software where the virtual cameras can be "perfect". That is, they can avoid any issues that arise in stereoscopic photography due to lenses. Also, with techniques like multi-rigging and dynamic inter-axial control they can achieve high quality results efficiently. This makes CG-3D the best choice for these types of shots.
Major CG element in liveaction environment		X	X	This situation touches on the Composite shot scenario listed above. In this situation the CG character likely fills most of the frame and is used in conjunction with a few other 2D elements to create the final shot. Similar to the fully CG environment shot, CG characters can be generated with "perfect cameras". Similar to the composite shot, the 2D elements are more easily matched to this character through Dimensionalization. This makes CG the appropriate choice for the bulk of work, generating the CG character, and Dimensionalization the appropriate choices from marrying the 2D elements with that character.
Minor CG element in liveaction environment			X	In this situation the CG-element is not the main element filling the frame. It may be a small CG bird or something of that sort. Dimensionalization is the appropriate choice for this shot because the assistance that rendering another perspective for the CG element would provide would be minimal.
Major liveaction element in CG Environment	X	X	X	In this Situation the CG-element is the environment and the character(s) is live-action. The best approach here is to employ a hybrid approach between CG-3D and Dual-cam or CG-3D and Dimensionalization. The choice between the two options rest with which approach should be "slaved" to the other approach. In the CG-3D and Dual-cam approach the CG-3D choices would be slaved to the Dual-cam 3D. Meaning that the choices made on set (in terms of inter-axial) would have to be matched when generating the CG environment. In the CG-3D and Dimensionalization approach, Dimensionalization would be slaved to the CG-3D choices. Meaning, the CG-3D choices would be made first and Dimensionalization would be used to conform the live-action elements as necessary to match. This option opens up the possibility of techniques in CG-3D that would not be able to be employed under the Dual-cam and CG-3D paradigm (such as multi-rigging).
Crowd shots	X			Similar to complex organics, crowd shots have hundreds or thousands of distinct layers of depth. While it is possible to achieve this nuance with Dimensionalization, it is more efficient to shoot this with a dual-camera rig. Also, these shots are often obtained from fixed locations so the necessary interaxial separation can be planned for.
Miniatures	X		X	Depending on the size of the miniature and the size of the dual-camera rig it may be acceptable to shoot stereoscopically. Usually the production crew is not as large as the main one, so the extra time needed to set-up and operate the dual-camera rig may be negligible. However there is the potential that the inter-axial between the cameras cannot be sufficiently closed to avoid making the miniature look miniature. Dimensionalization is a completely viable option if the additional time and cost is significant and if the dual-rig cannot avoid making the miniature look miniature.
35mm shoot	X		X	35mm cameras have a long history in 3D and can be as effective as their digital counterparts for creating 3D content. Their limiting factor, as compared to digital cameras, is size. In the case where the 35mm camera must be mobile, as in hand-held shots, Dimensionalization is the appropriate choice.
Complex Rigremoving			X	Stereoscopic paint tools are arriving on the market, but they are only adequate for simple wire removal or at most flat surfaces. The issue with more complex stereo rig-removing tasks is that the paint stroke needs to be distorted in the secondary perspective as if it were painted on a full 3D model. If it's not, then one may be able to perceive a dent in object when viewed in 3D. Dimensionalization avoids this problem because rig-removing would happen before the alternate perspective is generated.
Staging "cheats"			X	Occasionally a director needs to "cheat" an actor toward or away from the camera. This may or may not be in conflict with the other actors line of site. The other actor may be directed to perform the scene while looking at the other actor's ear instead of eyes. Because dual-camera rigs mimic how our eyes capture the world, this cheat will be evident and look odd. If this cheat is a necessity, Dimensionalization is the appropriate choice because depth is created to "look right", not necessarily scientifically accurate.
Crane	X			During a crane shot motion depth cues provide the human brain with extensive amount of information about the depth of objects in the scene. Any conflict between the motion depth cues and the chosen depth during Dimensionalization is easily evident. That means Dimensionalization for this type of shot is possible, but much more difficult. That is why dualcamera rigs are the best option for this type of shot.
Steadicam	X			Similar to a crane shot, the motion depth cues in the shot require a higher degree of accuracy of depth that is more easily achieved by dual-camera rigs.
Shaky			X	Shaky shots are often problematic in 3D because the ability to perceive depth exaggerates the effect. Dimensionalization is the appropriate choice for this type of shot because depth can be reduced once this problem becomes evident in editing.
"Run & Gun"			X	When one needs to "run & gun" to capture an event or follow inspiration, he needs to work with easy to use equipment that can respond reliably and quickly. Dual-camera rigs need a significant amount of set-up time, calibration, and forethought to get a good results. This makes Dimensionalization the appropriate choice. The directors can worry about the depth once they've gotten the shot they need.
Particles	X	X		Shots where the volume is described by particles, such as fog, rain, dust, snow, or smoke, can be the some of the most compelling in 3D cinema. To be effective these particles (thousands +) need to accurately play in depth. This level of detail is most easily and cost effectively achieved with stereoscopic photography or CG renderings.

Note

Multiple categories listed below may apply to 1 shot. It is up to the director and VFX supervisor to weith their options.