Smart Cropping to get a sharper Panorama

<<How to>>

Part III

Assembly and blend of two half-images in a Panorama

How to get rid of Nadir obstructions without any retouching at all

Introduction

When the photographs where shot with a camera on top of a panorama head and a tripod, those two elements shall protrude in center of the field of view at the Nadir.

The obstructed part of the images belongs to the Nadir shot itself or to the photographs that were shot laterally around the vertical axis.

The usual solution - to clear the Nadir of the final panorama from the intrusion - is to shoot (often with the hand held camera) a specific photo. Then one patches the obstruction out in a graphic application by cloning the unobstructed area onto the offended Nadir one way or another. This is done generally, but not necessarily at the latest step of panorama making workflow. The rendered panorama is thus retouched outside of the GUI. The workflow is more complicated and the result depends on the skill of the retoucher.

Sometimes, no specific Nadir shot is necessary (i.e. when enough digital information of the Nadir area is already present in other images) but the Nadir is anyhow spoiled by the obtrusive element as well. Retouching to be done.

Some pan-photographers work around the difficulty by simple and light retouching when the obstruction has been purposely previously reduced to a minimum during shooting session.

Even more simply, others pan-photographers patch a personal logo onto the Nadir, thus giving commercial information by the same token.

Object

This is related to an alternative way that can be applied in the stream of the workflow in PTGui 5 and directly with the support of this Application.

The effort is transfered upstream to the shooting phase as two symmetrical images of the Nadir are used: the first one shall have a half of the Nadir view (almost) free from obstruction and the second shall bring the complement. This is done, for example by means of a special hardware configuration.

The presentation shown below is based on a DSLR with an APS-C sensor (Canon EOS 20D) and a circular fisheye lens (Sigma EX 8mm). But the process can be ran with most other combination used to make spherical (cubic) panoramas.

The basic principle that shall be applied is "Smart Cropping" in PTGui 5. This functionality of the GUI is a powerful tool in itself since the moment when Control Points of "negative coordinates" where supported. A Control Point has negative coordinates when it is located outside of the Crop area boundaries (in any of the PanoramaTools GUI).

Anyhow, doing the Cropping outside of the PanoramaTools GUI is a possible alternative in a similar manner that was previously developed (click on "HOW TO: Part I" button from the navigation bar at the top) but more complicated due to the large offset of the cropping area. The process shall be less streamlined but it is the only way that I found for the PTMac application.

Smart Cropping Basics

Technical preamble (can be skipped over)

A fundamental feature of PanoramaTools (and that is also adopted in the recent PTGui v5) is that every image to be stitched can be singularly described in both the Optimizer and Stitcher scripts.

All the parameters and coefficients of every "i" line can have a different value from any other "i"line. That is to say: any to-be-warped/stitched/blended image can be different e.g. have been shot with another lens, or with another camera or come from different source(e.g. scanner), etc.

A directly related fact to be known is that the "Lens Type" which is specified in one place (i.e. "Lens Settings Tab" in PTMac and PTGui 4) or two places now (i.e. "Project Assistant Tab" and "Lens Settings Tab" in PTGui 5+) becomes arbitrary and are without effect on a given image as soon as this image is given a independent status (i.e. when "Use individual lens parameters for:" box is check marked for this image). Hence, the word "Lens Type" in the "Images Parameters" table means more accurately "Cropping Type" under these circumstances because it then does specify the Cropping characteristics applied in the image rather than the lens that was used to shoot this photograph...

The confusion between real meaning of the "type of lens" in the different Tab pages has thus to be avoided. For example, when the lens type that was used for shooting could be a "circular fisheye", the "Lens type" in the two said Tab pages could be set (note: they are synchronized by PTGui) as "Fulframe" while the setting in the column "Lens type" of the "Images parameters" table could be "Fulframe" (another image could being set as "Circular" and have also a different "fov" as well). This apparent hodgepodge can be actually correct and simultaneously set on purpose.

In the described method, different Crop characteristics shall be specified for two of the images from the other five. Settings such as size and location in the source image coordinates ("Crop" tab page) will vary from those two image to all others. The parameters "fov", "d" and "e" will be simultaneously and accordingly "Optimized". The "d" an "e" parameters shall have very unusual high value corresponding to the actual location wrt the center of the coordinate system (center of the source image i.e. different from the Crop system coordinate which is centered at the top left of the image).

After warping, this shall result in an optimal patchwork of parts of these sources images in the final Panorama.

"How to" Crop two Nadir shots in halves to get rid of the obtrusive tripod head

This is the ultimate and more sophisticated way using Smart Cropping.

Just to make things even more spectacular, two "Lens Types" shall also be used simultaneously in PTGui 5: "FullFrame (i.e. rectangular) fisheye" shall be used to crop the two "half-Nadir" images and "circular fisheye" for the other five faces of the spatial cube.

Naturally this is not absolutly required. All the images could be kept as the same "Type of Lens" namely "FullFrame".

Illustrated workflow for Optimization, Stitching and Blending in PTGui 5 (Click on Picts to enlarge)

Project assistant Tab

FOV~113 degrees shall be meaningless at the end of the process, when all the images to be stitched have individual characteristics It could be anything else as well.

USE the "avanced" presentation mode to run the rest of the process

Lens type is Fullframe (i.e. rectangular shape). It could be specified as circular as well.

Source Images Tab The Tif images comes from conversion from RAW format.

See enlarged version of te image in the "Control Points tab" section below.

2336 x 3504 pixels

Lens Settings Tab All images have "Individual Parameters Settings" selected.

The shown coefficients here are also meaningless for this reason: every image shall have its own characteristics

Note that this page shall have to be accessed in several occasion and iteratively in the course of the process.

Panorama Settings Tab Standard settings for a Full spherical equirectangular panorama

Crop Settings Tab 1

The Red grid shall serve as tracer to clearly see what part of this image is visible in the final panorama.

The crop settings are the Chosen to be supported by PTGui 5 (i.e. FOV<110 degrees)

Advanced users: Other type of crop could be used (see "Images parameters")

Crop Settings Tab 2(a)

The Black-grid shall serve as tracer to clearly see what part of this image is visible in the final panorama.

The settings shall command the crop of ~1450 x 2266 pixels. Just to cover a little bit more than half of the image width.

Advanced users: Other type of crop could be used (see "Images parameters")

This is one first image of the Nadir: the tripod and the vertical arm of the panorama head are spoiling most of it.

Crop Settings Tab 2(b)

The white shall serve as tracer to clearly see what part of this image is visible in the final panorama.

The settings shall command the crop of ~1450 x 2266 pixels. Just to cover a little bit more than half of the image width.

Advanced users: Other type of crop could be used (see "Images parameters")

This second image of the Nadir has been symmetrically shot compared to the first one above it (Tab 2 (a)).

The tripod and the vertical arm of the panorama head are spoiling the some area but on a symmetrical position: The tripod has been rotated 180 degrees around the vertical axis of the panorama.

Crop Settings Tab 3

The Green grid shall serve as tracer to clearly see what part of this image is visible in the final panorama.

The crop settings are the same as for the first image.

Advanced users: Other type of crop could be used (see "Images parameters")

Crop Settings Tab 4

The Blue grid shall serve as tracer to clearly see what part of this image is visible in the final panorama.

The crop settings are the same as for the first image. processing.

Advanced users: Other type of crop could be used (see "Images parameters")

Crop Settings Tab 5

The Cyan grid shall serve as tracer to clearly see what part of this image is visible in the final panorama.

The crop settings are the same as for the first image.

Advanced users: Other type of crop could be used (see "Images parameters")

Crop Settings Tab 6

The Magenta grid shall serve as tracer to clearly see what part of this image is visible in the final panorama.

The crop settings are the same as for the first image.

Advanced users: Other type of crop could be used (see "Images parameters")

Images parameters Tab The settings come from the final optimization.

"Individual Parameters Settings" were selected (Images Parameters Tab). The Nadir cropped images are rectangular the other are circular. The fov is <110 deg as expected.

Note the VERY unusual HIGH value of "Hshift" and "VShift" for the two Nadir cropped images #1 and #2

This page above shows that all the parameters and coefficients were optimized individually. This could have been done otherwise and with "fixed and calibrated" values. However this would produce not too much difference in the apparence of the final result.

Control Points Tab The CP were first created by the "Auto Create" new functionality of PTGui 5.

Note the wide and good spreading of these points and note also the grid color that differentiate the source images one from each other.

Optimizer Tab All individual parameters and coefficients were optimized individually.

Preview Tab I rarely use this functionality. These are default settings, I can imagine.

Create Panorama Tab

One of numerous combination possibilities as the FOV<110 degrees allows using PTGui new internal engines amongst the other older possibilities (e.g. Enblend, Smartblend).

Panorama Editor The six images participate about equally: look at the "colored grid" relative area and location.

Note that the tripod legs are still prominent on the "white grid" Nadir

Theflat blended equirectangular Panorama The six images participate about equally: look at the "colored grid" relative area and location.

Note that the Nadir is absolutely clean: no more obstruction! The cleaning was done by blending.

A more obvious representation of the blending effect that cleans out the Nadir from obstruction:

The blended Panorama and all of the Layers in an VR Object QT movie Click once on the picture and use the Arrow Keys of your keyboard to see all the layers and the panorama one by one.

Alternatively: click & drag the mouse sideways left or right.

IMPORTANT:

Please remember that it is not mandatory to mix the types of lens (that is to say the "shape" of crop). Mixing was shown here for demonstrative purpose only.

Rectangular cropping (i.e. Fullframe "Lens Type" in the "Image Parameters" Tab table) for all of the images (Refer to the previous Chapter). This would do the same visual final effect.

In fact I would recommand this as it is more "regular" and easy to handle, especially for non-experienced PTGui users.

By the way: if good calibrated values were known for the lens, it could be advised to use those for all the images instead of using individual settings.

Of course, the very large Hshift (d) and Vshift (e) parameter values together with the lack of sure repeatability for the Cropped Nadir images parameters as to be accounted for. Complete Optimization is required for these two images.

Conclusion:

The method that is presented here may possibly lead to the best Panorama quality for a given Fisheye lens. No retouch should be necessary depending only on the care taken during the shooting of the seven photographs.

I have personnally at first found a little confusing having to go not linearly in the steps of the PTGui 5 workflow but I got quickly familiar with the process.

Due to the manner the two Nadir shoots were shot, precise repeatability is not possible from one panorama to an other, one can therefore conclude that this method could not be subject of Batch Stitching... Best possible quality is may be at this price level?