Photographic lenses projections: computational models, correction, conversion...

Fisheye projection: an unknown and neglected species


Image remapping for Panospheric Imaging

A general imaging technology which is able to capture and present substantially spherical fields of view has been developed, and a new discipline known as panospheric imaging (PI) has been recognised. PI is a technology which allows a substantially spherical field-of-view to be captured, digitally processed, and presented to an observer in the form of a fully immersive spherical perspective image, in both still and full motion video formats. This technology greatly simplifies the process of acquiring and presenting panoramic and immersive still images, and offers for the first time a practical technology for true panoramic and panospheric full motion video. PI has become feasible because of a number of separate advances, including the development of appropriate optics, the emergence of a general digital image remapping capability able to correct even severely distorted images, and the availability of wide angle virtual reality (VR) headsets with head orientation sensing, which provide an appropriate device for viewing such images.

Visionary abstract from [ "An introduction to panospheric imaging" by Bogner, S.L. 22-25 Oct 1995 Systems, Man and Cybernetics, 1995. IEEE International Conference on Intelligent Systems for the 21st Century.]

Perspective projection is still thought and still taught as being the unique holly graal for photograph(er)s

Camera modelling has been the subject of a considerable research effort during the last three decades, thanks to the new capabilities given by computation means allowing to manipulate digital images. Just to get an idea of the present amount of knowledge, one could read for instance the list of references at the end of "Camera Models and Fundamental Concepts Used in Geometric Computer Vision" by Peter Sturm et al: it presently comprises 562 titles and the authors admit that it's far to be comprehensive (and they planned explicitely to update it on a web site with the help of the readers of the article ): "While aiming at a certain level of exhaustiveness in the coverage of the literature, such a survey can never be complete; we apologize for any significant omission"... After such a statement by these authoritative and renown scientists, I certainly would not pretend to do a better job eventhough I have found and read dozens of excellent papers, documents, books and websites which could also be appended to their huge list!

As a matter of fact, a vast majority of the studies related to lens projection models and to image correction was written to describe better than previous existing means of correcting the distortion of images... Thus they all proposed a way to get near-perfect "ideal" (rectilinear) images or to get them faster or easier if not better. I have not found on the 562 items long list a reference that would purposely propose or suggest to correct a photographic image in order to output another projection than the "holly" Perspective... with one notable and formidable very smart exception suggesting to "un-distort" but not to make rectilinear images. And Sturm et al. apparently failed to recognize this pioneering innovation...

About image distortion correction and about image projection correction

Any user of graphical programs knows that from a digital image, one can get another different image by changing each pixel location to an other place in the image plane (using clever interpolation to take care of possible missing information during the transformation). With an adequate way to compute and to interpolate the new ccordinates of the location of every moved pixel in relation with its initial location, the initial image can thus be transformed to another new image. This whole and global (or alternatively partial and local) transformation of the image can possibly be a "correction". We may say this in a more precise way by distinction of two cases:

Underlaying general principles for camera modelling and image correction

The underlying geometric principle of image projection and models
Image distortion correction
Image conversion from a projection to another (process sometimes designated image correction)

rd= radial distance from the Image distortion center to the Distorted image point

ru= radial distance from the image distortion center to the Undistorted image point after image distortion correction

rsrc= radial distance from the Image center to the image point before image transformation

rdest= radial distance from the image center to the corresponding Image point after image projection conversion

(*) In the vast majority of the literature the perspective (rectilinear) projection is considered as the Ideal Reference.

The radial distance from the Principal point (the center of the image) to the Image point depends on on the incoming angle θ measured from the lens axis and on the projection. In the traditional case, the perspective (gnonomic) projection makes the dotted line to hit the image plane at the conventional "Undistorted image point" . The diagram above is to generalize this proposition: when the intention is to convert from an initial projection to an other one, and thus possibly leaving the image with geometrical aberration traditionally known as "distortion", then the meaning of "undistorted image point" becomes "converted image point".

The perpective (aka rectilinear) projection is generally considered to be the Reference (remember the saying : << a straight line must be straight (the straight lines in 3D space are then straight on the 2D image)>>). This is IMO a convention (often not mentionned) and it is the fruit of tradition: most of the mathematical formulas and camera models for computation of projection transformation are intended to get rid of "unwanted distortion" that affects the raw camera image. This is the case for a huge majority of scientific papersabout camera calibration and modelling: I could not find more than a dozen of documents that explicitly not use perspective rectilinear as the Reference BTW most of these exceptions were about image mosaicing, image stitching or similar topics.


Models for rotational symetric refractive lenses

Fundamental principle

First, let us recall this underlaying principle which is a simplified ray tracing from the 3D space to the 2D image plane:

Recall: the 5 "classical" projections models

Classical projection modelling: A camera model is often built to represent (i.e. to describe the specific characteristics) of the projection function of certain lenses. Optical design considerations drive the formula as it illustrates the light transport rays from the object 3D space to the image plane through the lens. All the image points in the focal plane are defined from a function of the Angle (in the object plane) in conjunction with the Psi angle (around the lens axis).The process follows the light transport direction from the object to the focal plane. The reversed projection function goes from the focal plane back to the object space and is for instance the representation (back-projection) of the projector in a cinema theater (where the 3D space is more than often an over-simplified 3D reflective medium reduced to a flat screen however).

Equations for projection modelling
Rectilinear aka Perspective
Equidistant aka equi-angular
Equisolid angle aka Equal area
Orthographic aka Sine law
Projection function rd= f * tan θ rd= 2f * tan (θ/2) rd= f *θ rd= 2f * sin (θ/2) rd= f * sin θ
Reverse (back) projection function θ= arctan (rd / f) θ= 2arctan (rd / 2f) θ= rd / f θ= 2arcsin (rd / 2f) θ= arcsin (rd / f)

Camera calibration, modelling and distortion correction: Photogrammetric sciences and applications following by computational imaging disciplines have developped another way to describe the same phenomenon but most often by focusing on the relation between the radial distances to the distorted point of the real 2D image and to the undistorted point (rd and ru respectively), both being measured from the from the distortion center. To completely recover the whole camera image coverage, the angle of incoming light is the driving parameter. Calibration of the camera system is done directly when rd=f(ru) and distortion correction is directly done when ru=f(rd)... but modern computational means allows now also to do the reversed ways too, and therefore the boundaries that used to separate the computational imaging world from the optics world have nearly completely disappeared.

The classical models and the inverse function for the classical five projection functions are:

Equations for distortion correction or camera calibration
Rectilinear aka Perspective
Equidistant aka equi-angular
Equisolid angle aka Equal area
Orthographic aka Sine law
ru= f(rd) [direct equation] ru= rd (i.e. no distortion) ru= f * tan (2 arctan (rd/2f)) ru= f * tan(rd / f) ru= f* tan(2* arcsin (rd/2f)) ru= f* tan (arcsin (rd/f))
rd= f(ru) [inverse equation] rd= ru (i.e. no distortion) rd= 2f * tan ((arctan (ru/f))/2) rd= f * arctan(ru / f) rd= 2f* sin (arctan (ru/f)/2) rd= f * sin (arctan (ru/f))

Alternative physical parametric fisheye models to replace or to complete the classical formulas

As most of the studies were done to "correct distortion" from "distorted images", most of the research effort has produced equations, formulas and other cameras systems models in that end in mind: viewing in fine a rectified rectangular image is the permanent motto. This means that the suggested equation is often not valid for very large incoming angle and therefore it cannot handle images of the fisheye along its whole fied of view (that can be >> 180 degrees and thus θ >> 90°). Here after are a few of such formulas:

rd= fθ+Σ [i = from 1 to 5] ki * sin ((i πθ)/θmax) where θmax is the camera’s field of view (assumed known).

rd= k1 log (1+f * tanθ)

rd= k1 * sin (k2 θ )

Unfortunately, this formula has only a sine function and thus can map neither the Stereographic nor the Rectilinear projection. Nevertheless this simple equation supports very well the projections of most of the high-end photographic fisheye lenses that are currently (2012 ) commercially available. Some lenses with stereographic projection have been lately introduced by Samyang though.

rd= k1 * tan (θ/k3) + k2 * sin (θ/k4)

rd= sin (βθ) / k cos (max(0, βθ))

One may read the formula background and other details about this model on page 13 of this document.

Polynomial parametric models

Most of the physical models most often based on trigonometric formulas were proposed by Optics engineers that were studying the way an image is formed or viewed during photographic shooting. These formulas often describe how the lens contributes to the imaging process.

Other scientists were more interested by the Computational handling of this shot much more than by the picture taking part. As they can be integrated into graphic programs and commanded by the user with adjustable parameters, (too?) many sorts of parametric polynomial models have been proposed during the last three decades.

As a matter of fact and by nature, there is no absolute requirement for conventional and specific relation to a kind of trigonometric projection or another! Thus, diverging from the Optics literature common basics, some new models became de facto the novel "classical" (polynomial) models after some time of existence and popularity in the related Computer Vision literature. These models should not to be confused with the preceding defined "classical", "traditional" or "conventional" non-parametric models.

Some of these parametric polynomial models are presented in a Radial form where at this stage the distortion is assumed to be axis-centered (and about a fixed center viewpoint) and radially symetrical. When considering a distortion center diffrent from the origin, the given back-projection equations can be modified in a straightforward manner, by preceding them with the translation bringing the distortion center to the origin. When considering a distortion center different from the principal point, the modifications of back-projection equations are equally straightforward. However other models are presented in the classical Cartesian form and sometimes taking possible decentering and radial dissymetry into account.

The use of polynomials to model radial distortion in lenses is currently standard practice. From an embedded implementation point of view, polynomials are desirable as they do not require the estimation of functions based on log and tan operations (although, with the use of look-up tables, this advantage is lessened). Practical problems with polynomial models arise due to the fact that there is no analytical method to invert them, i.e. there is no general method to invert a forward model to its inverse for use in correction of radial distortion.

The following notations can be used. The radial distance is denoted by rd (in Distorted images) and ru (in Undistorted images). The angle between a camera ray (in the object space) and the optical axis is denoted by θ. Radial distortion models are usually defined as a mapping between any of these three entities. For simplicity, we always assume in the following that the radial distortion center is known... and located at the origin.

Beside some famous more intricate modelling (e.g. division models, rational models, FET etc.,) there are two ways to model a lens "projection" model or a lens "distortion correction" model: they are casually designated respectively direct projection models where U= f(D), and back-projection models where D= f(U). Here again, D stands for "Distorted images" and U means "Undistorted images"

Back-projection Vs forward projection.

One defines the other but one or the other may be more difficult to formulate algebraically. For example, if the "classical" radial model is written using polynomials, even with only one distortion coefficient, the polynomial is probably cubic, and it is cumbersome to write the inverse model. This "simple" model is traditionally used for back-projection although some researchers used the same expression for forward projection. Other polynomial models are also used for either direction. Generally speaking though, back-projection is often easier to formulate.

A main property of camera models is that some have a direct expression for forward projection, others for back-projections, some work easily both ways. Because the undistorted point location can be determined directly from the corresponding distorted point, the forward projection is known to work better for distortion correction of images and bundle adjustment, back-projection for minimal methods for various structure-from-motion tasks, such as pose and motion estimation, but back-projection can be also elegantly used in computer vision for images distortion correction or projection transformation (cf Dersch's Panorama tools and its siblings).

A direct projection model is used by Norman Koren in the tool Distortion that is a module of the Imatest tools suite: ru = rd + h1 rd^3 + h2 rd^5. I am not certain if this second order model is really good enough for representing the (circular) fisheye projections correctly along the whole huge FoV of most of these lenses. In any case, correct modelling of fisheye lenses is not really a driving requirement from the Imatest customer base who mostly are Photographic magazines and Photo reviews website: these specific patrons don't care too much about exhaustive and precise measurement results about fisheye projection: (BTW, Norman Koren obviously shares the opinion of Alexandre Jenny, the developer who wrote in the User's manual documentation of AutoPano Pro: Distortion model : 3rd order : Further complex distortion. Few lenses still get this type of distortion (not very useful in practice). Hmmm....

As Polynomial expressions, we may use Taylor series adequatly limited with "optimized" coefficients (up to the seventh power when necessary) for plotting the concerned SINE and TANGENT functions of the classical traditional formulas for the various lens projections. Result is as follows when using the seventh powered terms of the polynomial:

Projection type and "classical" trigonometric formulas
Rectilinear undistorted projection ............. rd= ru= f tan θ .................~ f * (θ + k1* θ^3 + k2* θ^5 + k3* θ^7)
k3 is needed for a precise appxoximation
Stereographic projection............................ rd= 2f tan (θ/2)...........~ f * ((θ/2) + k1* (θ/2)^3 + k2* (θ/2)^5+ k3*(θ/2)^7)
k3 brings better accuracy in the approximation
Equidistance projection...............................rd = f θ........................~ f * ((θ/2) + k1* (θ/2)^3 + k2* (θ/2)^5+ k3*(θ/2)^7)
Equisolid angle (a.k.a. Equal Area)............. rd = 2f sin (θ/2)...........~ f * ( (θ/2) - k1* (θ/2)^3 + k2* (θ/2)^5- k3*(θ/2)^7)
- 1/6
- 0.00019516895
k3 can be neglected without much accuracy degradation
Orthographic (a.k.a. Sine-law) projection... rd = f sin (θ)................~ f * (θ - k1* θ^3 + k2* θ^5 - k3* θ^7)
- 1/6
- 0.00019516895
k3 can be neglected

Note: In a paper by (2010) titled 'Accuracy of fish-eye lens models 'Ciarán Hughes et al suggested a so-called simplified model for orthographic projection: ru= rd/(√ (1- rd^2 / f^2)) and the inversed function rd= ru/(√ (1+ ru^2 / f^2)).

Using MS Excel, and plotting the various projections with the two alternate models in each of the 5 cases, we can check that the curves nicely overlap as expected, letting only one of the pair of curves to be visible for each projection. In a way this observation illustrates the high accuracy that is obtained from the computer algorithms when they simulates analog trigonometric expressions with ...polynomials:


Adobe's LPC model (2010)

Fisheye projection: A typical example is the Adobe Camera Model: in the frame of the Adobe Lens Profile Creation project, a simple and apparently elegant radial model (neglecting decentering and dissymetrical projection) was introduced in 2010 for (camera + fisheye lenses combo) calibration by the Adobe labs: rd= f * (θ + k1 * θ^3 +k2 * θ^5) where θ = arctan (r / f). This equation is designed to correct the "distorted" (fisheye) image into a "pin-hole camera" ideal image. Do not ask me to explain why only even powered terms and only two coefficients are used here: it is generally considered that these polynomial models are insufficient to describe the level of distortion introduced by fisheye lenses. One has however to remember that θ shall be restricted in practice to less than about 55 degrees (because of the rectilinear projection FoV shortcomings): along this 'reduced angular range', the Adobe formula is then probably adequate...

Perspective projection: an other separate model is also proposed by Adobe Labs for the case of Rectilinear lens where five coefficients and amongst which, three (k1, k2, k3) are dedicated to pure radial distortion modelling: rd= ru + ru^3 + ru^5 + ru^7. In Cartesian equivalent form , it can model the so-called tangential distortion (that we do not deal with on this paper for simplicity). Then Adobe added k4, k5 that are polynomial coefficients for the "tangential" distortion. Let's observe that an apparently similar approach has been adaopted by Kolor when they introduced a supplementary third "parameter" on option in their rectilinear projection model in AutoPano Pro and AutoPano Giga v 2.6+ .

Apple QTVR Group's fisheye model (1996)

Yalin Xiong and Ken Turkowski proposed to use a r= f(θ) general polynomial model (i.e. with both even and odd terms). This radial" distortion" function models the higher order effects in the mapping between the latitude angle and the polar distance: r= c1*θ + c2* θ^2 +c3 * θ^3 +... where the order of the polynomial can be determined 'experimentally'. Interestingly, this is one of the few exceptions that does not intend to correct "distortion" in the usual and conventional way: the authors have explicitly selected the Equidistance projection for first order approximation and have experimentally verified the model adequate accuracy with the venerable Nikon 8 mm f/8 (Note: as a matter of fact, their measurment shows that this lens is actually half-way between Equidistant and Equi-solid angle projection, as are most of the commercial fisheye lenses nowadays....)

Polynomial Fish-Eye Transform model (PFET, Basu and Licardie, 1995)

A general polynomial that uses both odd and even coefficients too has been proposed, and has been referred to as the PFET : rd = k0 + k1 * ru^1+ k2 * ru^2 + ... .+kn * ru^n +....

This PFET polynomial model was used as it makes the model independent of the underlying fisheye mapping function and can take into account some errors in the manufacture of the lenses. It has been suggested in many papers that a fifth-order form of the model is more than adequate to simulate the radial displacement introduced by fish-eye lenses. Some authors cite the same polynomial but letting k0= 0 (read next).

PanoTools lens correction model (H. Dersch, 1998)

Dr Dersch proposed in 1998 [rsrc= (a *rdest^3 + b*rdest^2 + c* rdest + d) * rdest] where src stands for "source image". This radial lens model by Helmut Dersch's Panorama Tools formula is universal as it can refer to the classical Rectilinear when a rectilinear lens is considered for distortion correction, but it is able also to correct very effectively a fisheye projection into equidistance as alternate reference. You may check that Dr Dersch was cautious not to call 'distortion' the deviation of the radial mapping of real fisheye lenses compared implicitely to the perpective projection or explicitely to the 'theoretical angle dependance' when he wrote the paragraph "Correct / Radial" of its most recent Panorama Tools readme1 note back in 2000. It is beyond doubt that Dersch's model is one sort of PFET (logically with k0 = 0), it is probably the most known equation amongst the panorama photographers community. Dr Dersch is a genius for Panospheric Imaging.

If you want to see how changing the parameters influences distortion correctionfollow this link and get abc.xls. (Don't deactivate macros on loading). PTGui as well as Hugin are a software descendant from PTools that have built up on this features.

References and Links:

A paper sharing usefully ways of building "Faster Math Functions" by Robin Green – Sony Computer Entertainment America

A comprehensive compilation and analysis of models: "Camera Models and Fundamental Concepts Used in Geometric Computer Vision" by Peter Sturm, Srikumar Ramalingam, Jean-Philippe Tardif, Simone Gasparini and Joao Barreto: there are more than 500 references at the end of the paper!

Michel Thoby

23 november 2012