A Minimal Set of Parameters Based Depth-Dependent Distortion Model and Its Calibration Method for Stereo Vision Systems

Abstract

Depth position highly affects lens distortion, especially in close-range photography, which limits the measurement accuracy of existing stereo vision systems. Moreover, traditional depth-dependent distortion models and their calibration methods have remained complicated. In this work, we propose a minimal set of parameters based depth-dependent distortion model (MDM), which considers the radial and decentering distortions of the lens to improve the accuracy of stereo vision systems and simplify their calibration process. In addition, we present an easy and flexible calibration method for the MDM of stereo vision systems with a commonly used planar pattern, which requires cameras to observe the planar pattern in different orientations. The proposed technique is easy to use and flexible compared with classical calibration techniques for depth-dependent distortion models in which the lens must be perpendicular to the planar pattern. The experimental validation of the MDM and its calibration method showed that the MDM improved the calibration accuracy by 56.55% and 74.15% compared with the Li’s distortion model and traditional Brown’s distortion model. Besides, an iteration-based reconstruction method is proposed to iteratively estimate the depth information in the MDM during three-dimensional reconstruction. The results showed that the accuracy of the iteration-based reconstruction method was improved by 9.08% compared with that of the non-iteration reconstruction method.

I Introduction

Lens distortion existing in optic measurement, which is usually caused by refraction and focusing of light rays through camera lenses, will result in changes in the shape and size of an image, subsequently affecting the image’s quality and accuracy. The lens distortion is mainly classified as a radial distortion and a decentering (or tangential[1]) distortion[2]. Radial distortion occurs when straight lines in the image appear curved or bulging at edges, causing a distortion in the proportion between the center and edges of the image. On the other hand, the decentering distortion refers to bending or war** of straight lines in the image along the horizontal or vertical direction, resulting in an overall skewed or distorted shape. The occurrence of lens distortion is due to the construction of the lens and the propagation of light rays within it. In the design and manufacturing process of lenses, certain unavoidable flaws exist, leading to inaccurate refraction and focusing of light rays, thereby causing distortions.

Accurate lens distortion calibration is the basis of precise vision measurement, especially in stereo vision systems [3]. Conventional lens distortion models, which consider radial and decentering lens distortions [4], [5] usually take the spatial constraints of the calibration target as the target and employ the nonlinear least-square optimization method to conduct the calibration (Zhang’s [6] method). However, these models neglect the effect of depth information on lens distortion [7], [8]. When measuring a distant target, the performance of these models is satisfactory. However, for large depths in close-range photogrammetry, depth information significantly impacts lens distortion; thus, adopting these models [9]-[11] can result in the low accuracy of the vision measurement system.

Magill [12] explained the relationship between radial distortion and magnification for the first time. However, the unknown parameters (distortion function for inverted infinite focus) in the distortion model could not be calibrated. Then, Brown presented a more applicable depth-dependent radial distortion model [13]. Consequently, Fryer proposed a depth-dependent decentering distortion model [14]. In these depth-dependent distortion models, no distortion function exists for inverted infinite focus, making the calibration of the lens distortion model possible. However, distortion functions at different focused depths must be calibrated, which entails adjusting the lens to focus on planes at different depths during the calibration process. To reduce the complexity of the calibration process, Alvarez derived a new radial distortion model by combining the Brown and Fraser [15] models. During the calibration, the lens focus does not need to be adjusted, increasing the stability of the calibration [16]. However, the decentering and in-plane affine distortion parameters are not calibrated. Consequently, Sun [17] and Li [18] proposed two depth-dependent distortion models, which consider both decentering and radial lens distortions. However, these depth-dependent lens distortion models have more than 16 unknown parameters, which makes the depth-dependent lens distortion models complicated. For example, the depth-dependent radial lens distortion model and decentering lens distortion model in [18] have 9 unknown parameters and 8 unknown parameters respectively, which need to be calibrated separately in 9 steps in total. The above-mentioned lens distortion models are listed in Table I. During the calibration procedures, especially when estimating the radial and decentering lens distortion parameters in the defocused plane in the depth-dependent lens distortion model, the checkerboard orientations need to be adjusted by mounting the checkerboard on an electric control platform and the camera needs to be adjusted to be perpendicular to the checkerboard. Thus, it will take plenty of time to adjust the checkerboard orientations during the calibration procedures and the additional equipment such as the electric control platform contribute to an increase in costs, which makes the calibration procedures inefficient. Besides, no effective three-dimensional (3D) reconstruction method is proposed in the reference for handling the input-output coupling (depth information) issue.

TABLE I: Comparison among different lens distortion models.

Method	Number of Parameters	Considering Depth- dependent	Considering Radial Distortion	Considering Decentering Distortion	Advantages	Disadvantages
Brown’s Model [13]	6	✓	✓	✓	The foundation of other distortion models	No distrotion functions exists for inverted infinite focus
RFDM [7]	23	✗	✓	✓	Accurate and flexible in 3D shape measurement	Depth information is not considered compared with [9]
DRDM [9]	20	✓	✓	✓	Accurate and flexible under large-field-of-view	More parameters to be calibrated compared with [16]
ADDM [16]	16	✓	✓	✗	Valid in any scenario	Decentering distortion is not considered compared with [18]
Li’s Model [18]	16	✓	✓	✓	Avoiding manually adjusting of the focus	More parameters to be calibrated compared with MDM
MDM	8	✓	✓	✓	The camera is not required to be perpendicular to the pattern	Hard to find accurate optimized initial values

In this paper, we propose a minimal set of parameters based depth-dependent lens distortion model (MDM), which considers the radial and decentering distortions of the lens to improve the accuracy of stereo vision systems and simplify their calibration process. Compared with [18], the minimal set of parameters in the MDM has a total of 8 unknown parameters of both radial and decentering lens distortion models, which makes the model more simplified. Besides, we present an easy and flexible calibration method for the MDM of stereo vision systems with a commonly used planar pattern, which requires cameras to observe the planar pattern at different orientations. Compared with some general calibration methods for the depth-dependent distortion models, such as the model proposed by Li and Sun, the proposed technique does not need the camera to be adjusted to be perpendicular to the planar pattern during the calibration, which is easy to use and flexible. Several experiments have been conducted to validate the MDM and its calibration method. The results showed the MDM improves the calibration accuracy by 56.55% and 74.15% compared with the Li’s distortion model and conventional Brown’s distortion model. Moreover, an iteration-based reconstruction method is proposed to iteratively estimate the depth information in the MDM during 3D reconstruction for handling the input-output coupling (depth information) issue. The results showed that the iteration-based reconstruction method improved the calibration accuracy by 9.08% compared with the non-iteration reconstruction method.

This paper is organized as follows: Section II introduces the minimal set of parameters based depth-dependent lens distortion model. Section III introduces the calibration method for the MDM and elaborates on the 3D reconstruction method for the MDM. Section IV provides the experimental details, and Section V outlines the conclusions of the paper.

II Minimal Set of Parameters Based Depth-Dependent Lens Distortion Model

In this section, the minimal set of parameters based depth-dependent radial lens distortion model (MDM-R) and the minimal set of parameters based depth-dependent decentering lens distortion model (MDM-D) are proposed.

II-A Minimal Set of Parameters Based Depth-Dependent Radial Lens Distortion Model

In this section, we simplified the classical depth-dependent radial lens distortion model based on the Magill model. The Magill model was first simplified by Brown in 1971 [13], which overcame the limited practicality. The simplified model is shown in (1):

$$\left\{\begin{array}[]{l}\delta r_{s}=\alpha_{s}\cdot\delta r_{s_{m}}+\left(1-\alpha_{s}\right)\cdot\delta r_{s_{k}}\\ \alpha_{s}=\frac{\left(s_{k}-s_{m}\right)\cdot(s-f)}{\left(s_{k}-s\right)\cdot\left(s_{m}-f\right)}\end{array}\right.$$ (1)

where $f$ denotes the focal length. When the lens is adjusted to focus on the distances $s_{m}$ and $s_{k}$, the radial distortions in the focal planes are defined as $\delta r_{s_{m}}$ and $\delta r_{s_{k}}$. Then, the distortion function $\delta r_{s}$ for the lens adjusted to focus on distance $s$ can be computed. The $I$-th radial distortion coefficients $K_{I}^{s}$ for the focused object plane at distance $s$ are described by:

$$K_{I}^{s}=\alpha_{s}\cdot K_{I}^{s_{m}}+\left(1-\alpha_{s}\right)\cdot K_{I}^{s_{k}}$$ (2)

where $I=1,2.$, and $K_{I}^{s_{m}}$ and $K_{I}^{s_{k}}$ represent the $I$-th radial distortion coefficients when the lens is adjusted to focus on the distances $s_{m}$ and $s_{k}$, respectively. However, an empirical model was proposed by Fraser and Shortis for the distortion of any defocused plane [15]:

$$K^{s,s_{p}}=K^{s}+\varepsilon\cdot\left(K^{s_{p}}-K^{s}\right)$$ (3)

where $K^{s,s_{p}}$ represents the radial distortion coefficient at defocused distance $s_{p}$ for the lens focused at distance $s$; $\varepsilon$ is a constant correction factor; $K^{s_{p}}$ and $K^{s}$ denote the radial distortion coefficients in the focal planes at distances $s_{p}$ and $s$, respectively. According to (3), the radial distortion function $\delta^{s,s_{n}}$ at $s_{n}$ can be obtained when the lens is adjusted to focus at distance $s$ :

$$\left\{\begin{array}[]{c}\delta r^{s,s_{m}}=\delta r^{s}+\varepsilon\cdot\left(\delta r^{s_{m}}-\delta r^{s}\right)\\ \!\delta r^{s,s_{n}}-\delta r^{s,s_{m}}=\varepsilon\!\cdot\!\left(\delta r^{s_{n}}-\delta r^{s}\right)\!-\!g\!\cdot\!\left(\delta r^{s_{m}}-\delta r^{s}\right)\end{array}\right.$$ (4)

Thus, by adopting the results to the distortion of a point in the defocused plane at distance $s_{k}$, the relationship among $\delta r^{s,s_{n}}$, $\delta r^{s,s_{m}}$, and $\delta r^{s,s_{k}}$ can be described as:

$$\left\{\begin{array}[]{l}\delta r^{s,s_{n}}=\delta r^{s,s_{m}}+\frac{\left(s_{m}-s_{n}\right)}{\left(s_{m}-s\right)}\!\!\cdot\!\!\frac{(s-f)}{\left(s_{n}-f\right)}\!\!\cdot\!\!\left(\delta r^{s}-\delta r^{s,s_{m}}\right)\\ \delta r^{s,s_{n}}=\delta r^{s,s_{k}}+\frac{\left(s_{k}-s_{n}\right)}{\left(s_{k}-s\right)}\!\!\cdot\!\!\frac{(s-f)}{\left(s_{n}-f\right)}\!\!\cdot\!\!\left(\delta r^{s}-\delta r^{s,s_{k}}\right)\\ \delta r^{s,s_{m}}=\delta r^{s,s_{k}}+\frac{\left(s_{k}-s_{m}\right)}{\left(s_{k}-s\right)}\!\!\cdot\!\!\frac{(s-f)}{\left(s_{n}-f\right)}\!\!\cdot\!\!\left(\delta r^{s}-\delta r^{s,s_{k}}\right)\end{array}\right.$$ (5)

Then (6) can be obtained by eliminating the focus distance, and the distortion in the focal plane:

$$\left\{\begin{array}[]{l}\delta r^{s,s_{m}}=\delta r^{s,s_{k}}+\frac{s_{k}-s_{m}}{s_{k}-s}\cdot\frac{s-f}{s_{m}-f}\cdot\left(\delta r^{s}-\delta r^{s,s_{k}}\right)\\ \delta r^{s,s_{n}}=\delta r^{s,s_{k}}+\frac{s_{k}-s_{n}}{s_{k}-s}\cdot\frac{s-f}{s_{n}-f}\cdot\left(\delta r^{s}-\delta r^{s,s_{k}}\right)\end{array}\right.$$ (6)

According to (6), $\delta r^{s,s_{n}}$ can be written as

$$\!\delta\!r^{\!s,\!s_{n}\!}\!=\!\frac{\delta r^{\!s,\!s_{k}\!}\!\!\cdot\!\!\left(\!s_{m}\!\!-\!\!f\!\right)\!\!\cdot\!\!\left(\!s_{k}\!-\!s_{m}\!\!\right)\!\!+\!\!\left(\!s_{k}\!\!-\!\!s_{n}\!\!\right)\!\!\cdot\!\!\left(\!\!s_{k}\!\!-\!\!f\!\right)\!\!\cdot\!\!\left(\delta r^{\!s\!,\!s_{m}\!}\!\!-\!\!\delta r^{\!s\!,\!s_{k}\!}\right)}{\left(\!s_{m}\!-\!f\!\right)\!\cdot\!\left(\!s_{k}\!-\!s_{m}\!\right)}$$ (7)

When the lens is adjusted to focus at distance $s$, the radial distortion coefficients in any defocused plane with the depth of $s_{n}$ can be obtained via two distortions corresponding to object distances $s_{m}$ and $s_{k}$, respectively [18]:

$$\!K_{I}^{\!s\!,\!s_{n}\!}\!=\!\frac{\!K_{I}^{\!s\!,\!s_{k}\!}\!\!\cdot\!\!\!\left(\!s_{m}\!\!\!-\!\!\!f\!\right)\!\!\cdot\!\!\left(\!s_{k}\!\!-\!\!s_{m}\!\right)\!\!+\!\!\left(\!s_{k}\!\!\!-\!\!\!s_{n}\!\right)\!\!\cdot\!\!\left(\!s_{k}\!-\!f\!\right)\!\!\cdot\!\!\left(\!K_{I}^{\!s\!,\!s_{m}\!}\!\!-\!\!K_{I}^{\!s\!,\!s_{k}\!}\right)}{\left(\!s_{m}\!-\!f\!\right)\!\cdot\!\left(\!s_{k}\!-\!s_{m}\!\right)}$$ (8)

According to (8), $K_{I}^{s,s_{m}}$, $K_{I}^{s,s_{k}}$, $s_{m}$, $s_{k}$, and $f$ are known since two object planes are intentionally set. Thus, $K_{I}^{s,s_{n}}$ in (8) depends on $s_{n}$ only, and (8) can be further simplified as follows:

$$\!\!\left\{\begin{aligned} \delta r_{u}^{R}&\!=\!\bar{u}\!\!\cdot\!\!\left[1\!\!+\!\!\left(A_{1}\!\!-\!\!B_{1}\cdot s_{n}\right)\cdot r^{2}\!\!+\!\!\left(A_{2}\!\!-\!\!B_{2}\cdot s_{n}\right)\cdot r^{4}\!\!+\!\!\ldots\right]\\ &\!=\!\bar{u}\!\!\cdot\!\!\left[1\!\!+\!\!K_{1}^{s,s_{n}}\cdot r^{2}\!\!+\!\!K_{2}^{s,s_{n}}\cdot r^{4}+\ldots\right]\\ \delta r_{v}^{R}&\!=\!\bar{v}\!\!\cdot\!\!\left[1\!\!+\!\!\left(A_{1}\!\!-\!\!B_{1}\cdot s_{n}\right)\cdot r^{2}\!\!+\!\!\left(A_{2}\!\!-\!\!B_{2}\cdot s_{n}\right)\cdot r^{4}\!\!+\!\!\ldots\right]\\ &\!=\!\bar{v}\!\!\cdot\!\!\left[1+K_{1}^{s,s_{n}}\cdot r^{2}+K_{2}^{s,s_{n}}\cdot r^{4}+\ldots\right]\\ K_{I}^{s\!,\!s_{n}}&=A_{I}-B_{I}\cdot s_{n}\\ A_{I}&\!=\!\frac{K_{I}^{s,s_{k}}\!\!\cdot\!\!\left(\!s_{m}\!\!-\!\!f\!\right)\!\!\cdot\!\!\left(\!s_{k}\!\!-\!\!s_{m}\!\right)\!\!+\!\!s_{k}\!\!\cdot\!\!\left(\!s_{k}\!\!-\!\!f\!\right)\!\!\cdot\!\!\left(\!K_{I}^{s,s_{m}}\!\!-\!\!K_{I}^{s,s_{k}}\!\right)}{\left(s_{m}-f\right)\!\!\cdot\!\!\left(s_{k}-s_{m}\right)}\\ B_{I}&\!=\!\frac{\left(s_{k}-f\right)\!\!\cdot\!\!\left(K_{I}^{s,s_{m}}-K_{I}^{s,s_{k}}\right)}{\left(s_{m}-f\right)\!\!\cdot\!\!\left(s_{k}-s_{m}\right)}\end{aligned}\right.$$ (9)

Here $A_{I}$ and $B_{I}$ are the $I$-th simplified radial lens distortion parameters in the short-distance defocused plane with the depth of $s_{n}$; $\delta r_{u}^{R}$ and $\delta r_{v}^{R}$ are the radial distortion functions in the $u$ and $v$ directions, respectively; $(\bar{u},\bar{v})$ is the uncorrected corner point image coordinate in the checkerboard image; $\left(u_{0},v_{0}\right)$ is the principal point coordinate; $r=\sqrt{\left(\bar{u}-u_{0}\right)^{2}+\left(\bar{v}-v_{0}\right)^{2}}$ is the distortion radius of the image point. Then, the MDM-R can be expressed as shown in (9).

II-B Minimal Set of Parameters Based Depth-Dependent Decentering Lens Distortion Model

In this section, we simplified the classical decentering lens distortion model based on the Brown’s distortion model. The decentering lens distortion model proposed by Brown in 1966 [19] is described as follows:

$$\left\{\begin{array}[]{c}\delta r_{u}\!=\!\left(1-\frac{f}{s}\right)\!\!\cdot\!\!\left[P_{1}\cdot\left(r^{2}+2\cdot\bar{u}^{2}\right)+2\cdot P_{2}\cdot\bar{u}\cdot\bar{v}\right]\\ \delta r_{v}\!=\!\left(1-\frac{f}{s}\right)\!\!\cdot\!\!\left[P_{2}\cdot\left(r^{2}+2\cdot\bar{v}^{2}\right)+2\cdot P_{1}\cdot\bar{u}\cdot\bar{v}\right]\\ r^{s,s^{\prime}}=\frac{s-f}{s^{\prime}-f}\cdot\frac{s^{\prime}}{s}\end{array}\right.$$ (10)

where $\left(1-\frac{f}{s}\right)\cdot r^{s,s^{\prime}}$ denotes the compensation coefficient; $\delta r_{u}$ and $\delta r_{v}$ represent the components of the decentering distortion in the $u$ and $v$ directions, respectively; $s$ and $s^{\prime}$ are the object distances corresponding to the two focal planes, respectively.

Based on (10), a depth-dependent decentering distortion model is proposed in [18], which is described as follows:

$$P_{I}^{\!s\!,\!s_{n}\!}=\frac{P_{I}^{\!s\!,\!s_{k}\!}\!\cdot\!\left(1\!-\!s_{m}^{2}\right)\!+\!P_{I}^{\!s\!,\!s_{m}\!}\left(\!s_{m}\!\cdot\!s_{k}\!-\!1\!\right)}{P_{I}^{\!s\!,\!s_{k}\!}\!\cdot\!\left(1\!-\!s_{m}\!\cdot\!s_{n}\right)\!+\!P_{I}^{s\!,\!s_{m}\!}\left(s_{n}\!\cdot\!s_{k}\!-\!1\right)}\!\cdot\!\frac{s_{n}}{s_{m}}\!\cdot\!P_{I}^{s\!,\!s_{m}\!}$$ (11)

where $P_{I}^{s,s_{m}}$, $P_{I}^{s,s_{k}}$, and $P_{I}^{s,s_{n}}$ represent the $I$-th decentering distortion parameters in the defocused planes at the object distances $s_{m}$, $s_{k}$, and $s_{n}$ when the lens is adjusted to focus at distance $s$, respectively. According to (13), the decentering distortion parameters $P_{I}^{s,s_{n}}$ in any defocused plane depend on only the object distance $s_{n}$ since $P_{I}^{s,s_{m}}$, $P_{I}^{s,s_{k}}$, $s_{m}$ and $s_{k}$ are known. However, (13) also has 6 decentering lens distortion parameters to be calibrated separately, which makes the decentering lens distortion model complicated. Similarly, the checkerboard orientations need to be adjusted by mounting the checkerboard on an electric control platform, and the camera also needs to be adjusted to be perpendicular to the checkerboard to estimate $P_{I}^{s,s_{m}}$ and $P_{I}^{s,s_{k}}$ during the calibration procedures, which makes the calibration procedures inefficient. Thus, a minimal set of decentering distortion parameters is described, and the decentering distortion model can be further simplified as follows:

$$\left\{\begin{aligned} \delta r_{u}^{D}&=\frac{1}{C_{1}+\frac{D_{1}}{s_{n}}}\!\cdot\!\left(r^{2}+2\!\cdot\!\bar{u}^{2}\right)+2\!\cdot\!\frac{1}{C_{2}+\frac{D_{2}}{s_{n}}}\!\cdot\!\bar{u}\!\cdot\!\bar{v}\\ &=P_{1}\!\cdot\!\left(r^{2}+2\!\cdot\!\bar{u}^{2}\right)+2\!\cdot\!P_{2}\!\cdot\!\bar{u}\!\cdot\!\bar{v}\\ \delta r_{v}^{D}&=\frac{1}{C_{2}+\frac{D_{2}}{s_{n}}}\!\cdot\!\left(r^{2}+2\!\cdot\!\bar{v}^{2}\right)+2\!\cdot\!\frac{1}{C_{1}+\frac{D_{1}}{s_{n}}}\!\cdot\!\bar{u}\!\cdot\!\bar{v}\\ &=P_{2}\!\cdot\!\left(r^{2}+2\!\cdot\!\bar{v}^{2}\right)+2\!\cdot\!P_{1}\!\cdot\!\bar{u}\!\cdot\!\bar{v}\\ P_{I}^{s,s_{n}}&=\frac{1}{C_{I}+\frac{D_{I}}{s_{n}}}\end{aligned}\right.$$ (12)

Here $C_{I}$ and $D_{I}$ are the $I$-th decentering lens distortion parameters in the short-distance defocused plane with the depth of $s_{n}$; $\delta r_{u}^{D}$ and $\delta r_{v}^{D}$ are the decentering distortion functions in the $u$ and $v$ directions, respectively. The MDM-D is described as shown in (14), which has a minimal set of 4 dencentering lens distortion parameters.

III Calibration and 3D Reconstruction Methods for the MDM

In this section, a flexible three-step calibration method for the MDM is proposed. First, the intrinsic and extrinsic camera parameters of the stereo vision system are calculated by the Zhang’s calibration method. Then, the MDM is estimated using linear constraints. Finally, the intrinsic and extrinsic camera parameters of the stereo vision system and the parameters in the MDM are optimized based on the distance constraints between adjacent points on the checkerboard. Besides, an iteration-based reconstruction method is proposed.

III-A Calibration of Intrinsic and Extrinsic Camera Parameters

As shown in Fig. 1, stereo vision systems consist of two cameras and two lenses. The MDM calibration method employs the most commonly used checkerboard. $m$ horizontal lines and $n$ vertical lines are arranged on the checkerboard. The two lines intersect to form a corner point. The distance between adjacent corner points is precisely known. In image acquisition, the lenses and cameras are first fixed on the optical platform, and the checkerboard is randomly placed in multiple orientations within a certain depth range. For each orientation, the two cameras capture the images of the checkerboard at the same time. A total of $2\times w$ images of the checkerboard are acquired.

Figure 1: Stereo vision system and transformations of different frames. ${O_{W}}$ is the origin of the world coordinate system, ${O_{C}^{L}}$ is the origin of the left camera coordinate system, ${O_{C}^{R}}$ is the origin of the right camera coordinate system, ${K_{L}}$ and ${K_{R}}$ are the intrinsic parameters of the left and right cameras, respectively. [$R_{L}^{W}$,$T_{L}^{W}$] and [$R_{R}^{W}$,$T_{R}^{W}$] are the transformation matrices between the world coordinate system and the left and right camera coordinate systems.

For the corrected $i$-th row and $j$-th column corner point $P_{ij}$ on the checkerboard plane in the world coordinate system $O_{W}X_{W}Y_{W}Z_{W}$, the map** relationship between its 3D reconstructed information $\left[X_{ij},Y_{ij},Z_{ij}\right]$ and the left image coordinate $\left(u_{ij}^{L},v_{ij}^{L}\right)$ and that between its 3D reconstructed information $\left[X_{ij},Y_{ij},Z_{ij}\right]$ and the right image coordinate $\left(u_{ij}^{R},v_{ij}^{R}\right)$ are introduced as follows:

$$\left\{\begin{aligned} z_{L}\left[\begin{array}[]{c}u_{ij}^{L}\\ v_{ij}^{L}\\ 1\end{array}\right]&=\boldsymbol{K}_{L}\left[\boldsymbol{R}_{L}^{W},\boldsymbol{T}_{L}^{W}\right]\left[\begin{array}[]{c}X_{ij}\\ Y_{ij}\\ Z_{ij}\\ 1\end{array}\right]\\ &=\left[\begin{array}[]{llll}m_{11}^{L}&m_{12}^{L}&m_{13}^{L}&m_{14}^{L}\\ m_{21}^{L}&m_{22}^{L}&m_{23}^{L}&m_{24}^{L}\\ m_{31}^{L}&m_{32}^{L}&m_{33}^{L}&m_{34}^{L}\end{array}\right]\!\!\!\!\left[\!\!\!\begin{array}[]{c}X_{ij}\\ Y_{ij}\\ Z_{ij}\\ 1\end{array}\!\!\!\right]\\ z_{R}\left[\begin{array}[]{c}u_{ij}^{R}\\ v_{ij}^{R}\\ 1\end{array}\right]&=\boldsymbol{K}_{R}\left[\begin{array}[]{ll}\boldsymbol{R}_{R}^{W},\boldsymbol{T}_{R}^{W}\end{array}\right]\left[\begin{array}[]{c}X_{ij}\\ Y_{ij}\\ Z_{ij}\\ 1\end{array}\right]\\ &=\left[\begin{array}[]{llll}m_{11}^{R}&m_{12}^{R}&m_{13}^{R}&m_{14}^{R}\\ m_{21}^{R}&m_{22}^{R}&m_{23}^{R}&m_{24}^{R}\\ m_{31}^{R}&m_{32}^{R}&m_{33}^{R}&m_{34}^{R}\end{array}\right]\!\!\!\!\left[\!\!\!\begin{array}[]{c}X_{ij}\\ Y_{ij}\\ Z_{ij}\\ 1\end{array}\!\!\!\right]\end{aligned}\right.$$ (13)

Here, $\boldsymbol{R}_{L}^{W}$ and $\boldsymbol{T}_{L}^{W}$ are the rotation matrix and the translation matrix, expressing the transformation between the left image coordinate system and the world coordinate system; $\boldsymbol{R}_{R}^{W}$ and $\boldsymbol{T}_{R}^{W}$ are the rotation matrix and the translation matrix, expressing the transformation between the right image coordinate system and the world coordinate system; $\boldsymbol{K}_{L}$ and $\boldsymbol{K}_{R}$ are the intrinsic parameter matrices of the left and right cameras, respectively; $z_{L}$ and $z_{R}$ are the scaling factors; $m_{ij}^{L}$ and $m_{ij}^{R}$ are the values of the $i$-th row and $j$-th column in the left and right camera projection matrices, respectively; $\left(u_{ij}^{L},v_{ij}^{L}\right)$ and $\left(u_{ij}^{R},v_{ij}^{R}\right)$ are the corrected image coordinates in the $i$-th row and $j$-th column in the checkerboard images under the left and right image coordinate systems, respectively. The image coordinates of corner points in the checkerboard images are extracted, and the initial values of the intrinsic $\left(\boldsymbol{K}_{L},\boldsymbol{K}_{R}\right)$ and extrinsic $\left(\boldsymbol{R}_{L}^{W},\boldsymbol{T}_{L}^{W},\boldsymbol{R}_{R}^{W},\boldsymbol{T}_{R}^{W}\right)$ parameters of the cameras are calculated using the Zhang’s calibration method [4].

III-B Calibration of MDM Parameters

In this section, the radial and decentering lens distortion parameters in the MDM are roughly estimated using linear constraints. For any corrected $i$-th row and $j$-th column corner point on the checkerboard plane, its left image coordinate $\left(u_{ij}^{L},v_{ij}^{L}\right)$ and right image coordinate $\left(u_{ij}^{R},v_{ij}^{R}\right)$ are obtained by (16), where $A_{1}^{L}$, $A_{2}^{L}$, $A_{1}^{R}$, $A_{2}^{R}$, $B_{1}^{L}$, $B_{2}^{L}$, $B_{1}^{R}$, and $B_{2}^{R}$ are the radial lens distortion parameters; $C_{1}^{L}$, $C_{2}^{L}$,$C_{1}^{R}$, $C_{2}^{R}$, $D_{1}^{L}$, $D_{2}^{L}$, $D_{1}^{R}$, and $D_{2}^{R}$ are the decentering lens distortion parameters; $\left(\bar{u}_{ij}^{L},\bar{v}_{ij}^{L}\right)$ and $\left(\bar{u}_{ij}^{R},\bar{v}_{ij}^{R}\right)$ are the uncorrected corner point image coordinates in the $i$-th row and $j$-th column in the checkerboard images under the left and right image coordinate systems, respectively. $\left(u_{0}^{L},v_{0}^{L}\right)$ and $\left(u_{0}^{R},v_{0}^{R}\right)$ are principal point coordinates under the left and right image coordinate systems, respectively; $r_{ij}^{L}=\sqrt{\left(\bar{u}_{ij}^{L}-u_{0}^{L}\right)^{2}+\left(\bar{v}_{ij}^{L}-v_{0}^{L}\right)^{2}}$ and $r_{ij}^{R}=\sqrt{\left(\bar{u}_{ij}^{R}-u_{0}^{R}\right)^{2}+\left(\bar{v}_{ij}^{R}-v_{0}^{R}\right)^{2}}$ are distortion radii of image points under the left and right image coordinate systems, respectively; $s_{ij}^{L}$ and $s_{ij}^{R}$ are the depth information of the corner points in the $i$-th row and $j$-th column in the checkerboard images under the left and right image coordinate systems, respectively.

$$\left\{\!\!\!\begin{aligned} u_{ij}^{L}&\!\!=\!\!\bar{u}_{ij}^{L}+\delta_{u_{ij}}^{L}\\ v_{ij}^{L}&\!\!=\!\!\bar{v}_{ij}^{L}+\delta_{v_{ij}}^{L}\\ u_{ij}^{R}&\!\!=\!\!\bar{u}_{ij}^{R}+\delta_{u_{ij}}^{R}\\ v_{ij}^{R}&\!\!=\!\!\bar{v}_{ij}^{R}+\delta_{v_{ij}}^{R}\\ \delta_{u_{ij}}^{L}&\!\!=\!\!\bar{u}_{ij}^{L}\!\!\cdot\!\!\left[1\!\!+\!\!\left(A_{1}^{L}\!\!-\!\!B_{1}^{L}\!\!\cdot\!\!s_{ij}^{L}\right)\!\!\cdot\!\!r_{ij}^{L2}\!\!+\!\!\left(A_{2}^{L}\!\!-\!\!B_{2}^{L}\!\!\cdot\!\!s_{ij}^{L}\right)\!\!\cdot\!\!r_{ij}^{L4}\!\!+\!\!...\!\right]\\ &+\!\!\left[\frac{1}{C_{1}^{L}\!\!+\!\!\frac{D_{1}^{L}}{s_{ij}^{L}}}\!\!\cdot\!\!\left(r_{ij}^{L2}\!\!+\!\!2\!\!\cdot\!\!\bar{u}_{ij}^{L2}\right)\!\!+\!\!2\!\!\cdot\!\!\frac{1}{C_{2}^{L}\!\!+\!\!\frac{D_{2}^{L}}{s_{ij}^{L}}}\!\!\cdot\!\!\bar{u}_{ij}^{L}\!\!\cdot\!\!\bar{v}_{ij}^{L}\right]\\ \delta_{v_{ij}}^{L}&\!\!=\!\!\bar{v}_{ij}^{L}\!\!\cdot\!\!\left[1\!\!+\!\!\left(A_{1}^{L}\!\!-\!\!B_{1}^{L}\!\!\cdot\!\!s_{ij}^{L}\right)\!\!\cdot\!\!r_{ij}^{L2}\!\!+\!\!\left(A_{2}^{L}\!\!-\!\!B_{2}^{L}\!\!\cdot\!\!s_{ij}^{L}\right)\!\!\cdot\!\!r_{ij}^{L4}\!\!+\!\!...\!\right]\\ &+\!\!\left[\frac{1}{C_{2}^{L}\!\!+\!\!\frac{D_{2}^{L}}{s_{ij}^{L}}}\!\!\cdot\!\!\left(r_{ij}^{L2}\!\!+\!\!2\!\!\cdot\!\!\bar{v}_{ij}^{L2}\right)\!\!+\!\!2\!\!\cdot\!\!\frac{1}{C_{1}^{L}\!\!+\!\!\frac{D_{1}^{L}}{s_{ij}^{L}}}\!\!\cdot\!\!\bar{u}_{ij}^{L}\!\!\cdot\!\!\bar{v}_{ij}^{L}\right]\\ \delta_{u_{ij}}^{R}&\!\!=\!\!\bar{u}_{ij}^{R}\!\!\cdot\!\!\left[1\!\!+\!\!\left(A_{1}^{R}\!\!-\!\!B_{1}^{R}\!\!\cdot\!\!s_{ij}^{R}\right)\!\!\cdot\!\!r_{ij}^{R2}\!\!+\!\!\left(A_{2}^{R}\!\!-\!\!B_{2}^{R}\!\!\cdot\!\!s_{ij}^{R}\right)\!\!\cdot\!\!r_{ij}^{R4}\!\!+\!\!...\!\right]\\ &+\!\!\left[\frac{1}{C_{1}^{R}\!\!+\!\!\frac{D_{1}^{R}}{s_{ij}^{R}}}\!\!\cdot\!\!\left(r_{ij}^{R2}\!\!+\!\!2\!\!\cdot\!\!\bar{u}_{ij}^{R2}\right)\!\!+\!\!2\!\!\cdot\!\!\frac{1}{C_{2}^{R}\!\!+\!\!\frac{D_{2}^{R}}{s_{ij}^{R}}}\!\!\cdot\!\!\bar{u}_{ij}^{R}\!\!\cdot\!\!\bar{v}_{ij}^{R}\right]\\ \delta_{v_{ij}}^{R}&\!\!=\!\!\bar{v}_{ij}^{R}\!\!\cdot\!\!\left[1\!\!+\!\!\left(A_{1}^{R}\!\!-\!\!B_{1}^{R}\!\!\cdot\!\!s_{ij}^{R}\right)\!\!\cdot\!\!r_{ij}^{R2}\!\!+\!\!\left(A_{2}^{R}\!\!-\!\!B_{2}^{R}\!\!\cdot\!\!s_{ij}^{R}\right)\!\!\cdot\!\!r_{ij}^{R4}\!\!+\!\!...\!\right]\\ &+\!\!\left[\frac{1}{C_{2}^{R}\!\!+\!\!\frac{D_{2}^{R}}{s_{ij}^{R}}}\!\!\cdot\!\!\left(r_{ij}^{R2}\!\!+\!\!2\!\!\cdot\!\!\bar{v}_{ij}^{R2}\right)\!\!+\!\!2\!\!\cdot\!\!\frac{1}{C_{1}^{R}\!\!+\!\!\frac{D_{1}^{R}}{s_{ij}^{R}}}\!\!\cdot\!\!\bar{u}_{ij}^{R}\!\!\cdot\!\!\bar{v}_{ij}^{R}\right]\end{aligned}\right.$$ (14)

As shown in Fig. 2, when the corner points on the same straight lines in the left and right checkerboard images are known, the line equations determined by the corner points under the left and right image coordinate systems are described as:

$$\left\{\begin{array}[]{l}\alpha_{i}^{L}u^{L}-v^{L}+\beta_{i}^{L}=0\\ \alpha_{i}^{R}u^{R}-v^{R}+\beta_{i}^{R}=0\end{array}\right.$$ (15)

where $\alpha_{i}^{L}$ and $\beta_{i}^{L}$ are the line parameters of the $i$-th line in the left checkerboard image; $\alpha_{i}^{R}$ and $\beta_{i}^{R}$ are the line parameters of the $i$-th line in the right checkerboard image; Then, (18) is obtained by substituting $\left(\bar{u}_{ij}^{L},\bar{v}_{ij}^{L}\right)$ and $\left(\bar{u}_{ij}^{R},\bar{v}_{ij}^{R}\right)$ into $\left(u^{L},v^{L}\right)$ and $\left(u^{R},v^{R}\right)$ in (17), respectively.

$$\left\{\begin{array}[]{l}F_{L}\left(\alpha_{i}^{L},\beta_{i}^{L},\boldsymbol{K}_{1}^{L},\boldsymbol{K}_{2}^{L},\boldsymbol{P}_{1}^{L},\boldsymbol{P}_{2}^{L}\right)=0\\ F_{R}\left(\alpha_{i}^{R},\beta_{i}^{R},\boldsymbol{K}_{1}^{R},\boldsymbol{K}_{2}^{R},\boldsymbol{P}_{1}^{R},\boldsymbol{P}_{2}^{R}\right)=0\end{array}\right.$$ (16)

$\boldsymbol{K}_{1}^{L}=\left[A_{1}^{L},B_{1}^{L}\right]$ and $\boldsymbol{K}_{2}^{L}=\left[A_{2}^{L},B_{2}^{L}\right]$ are the first and second-order parameters of the radial distortion under the left image coordinate system, respectively; $\boldsymbol{P}_{1}^{L}=\left[C_{1}^{L},D_{1}^{L}\right]$ and $\boldsymbol{P}_{2}^{L}=\left[C_{2}^{L},D_{2}^{L}\right]$ are the first and second-order parameters of the decentering distortion under the left image coordinate system, respectively; $\boldsymbol{K}_{1}^{R}=\left[A_{1}^{R},B_{1}^{R}\right]$ and $\boldsymbol{K}_{2}^{R}=\left[A_{2}^{R},B_{2}^{R}\right]$ are the first and second-order parameters of the radial distortion under the right image coordinate system, respectively; $\boldsymbol{P}_{1}^{R}=\left[C_{1}^{R},D_{1}^{R}\right]$ and $\boldsymbol{P}_{2}^{R}=\left[C_{2}^{R},D_{2}^{R}\right]$ are the first and second-order parameters of the decentering distortion under the right image coordinate system, respectively.

If there are $i$ lines in an image (left or right image) and $j$ observation points are extracted from each line, we obtain $i\times j$ equations. In these equations, there are $2\times i+8$ variables ($2\times i$ line parameters and 8 distortion parameters). If $i\times j>2\times i+8$, the optimal solution of the distortion parameters is obtained.

Figure 2: Linear constraints on the left and right images

$\boldsymbol{M}_{L}^{R}=\left[\boldsymbol{R}_{L}^{R},\boldsymbol{T}_{L}^{R}\right]$ is the 4-by-3 projection matrix, expressing the transformation relationship of the left camera coordinate system relative to the right camera coordinate system, where $\boldsymbol{R}_{L}^{R}$ and $\boldsymbol{T}_{L}^{R}$ are the rotation and translation matrices of the left camera coordinate system relative to the right camera coordinate system, respectively; $\boldsymbol{M}_{R}^{L}=\left[\boldsymbol{R}_{R}^{L},\boldsymbol{T}_{R}^{L}\right]$ is the 4-by-3 projection matrix, expressing the transformation relationship of the right camera coordinate system relative to the left camera coordinate system, where $\boldsymbol{R}_{R}^{L}$ and $\boldsymbol{T}_{R}^{L}$ are the rotation and translation matrices of the right camera coordinate system relative to the left camera coordinate system, respectively; $t_{ij}^{L}$ is the element in the $i$-th row and $j$-th column of $\boldsymbol{M}_{L}^{R}$; $t_{ij}^{R}$ is the element in the $i$-th row and $j$-th column of $\boldsymbol{M}_{R}^{L}$; $f_{L}$ and $f_{R}$ are the focal lengths of the left and right cameras, respectively; $s_{ij}^{L}=\frac{f_{L}\cdot\left(f_{R}\cdot t_{14}^{L}-\bar{u}_{ij}^{R}\cdot t_{34}^{L}\right)}{\bar{u}_{ij}^{R}\cdot\left(t_{31}^{L}\cdot\bar{u}_{ij}^{L}+t_{32}^{L}\cdot\bar{v}_{ij}^{L}+f_{L}\cdot t_{33}^{L}\right)-f_{R}\cdot\left(t_{11}^{L}\cdot\bar{u}_{ij}^{L}+t_{12}^{L}\cdot\bar{v}_{ij}^{L}+f_{L}\cdot t_{13}^{L}\right)}$ and $s_{ij}^{R}=\frac{f_{R}\cdot\left(f_{L}\cdot t_{14}^{R}-\bar{u}_{ij}^{L}\cdot t_{34}^{R}\right)}{\bar{u}_{ij}^{L}\cdot\left(t_{31}^{R}\cdot\bar{u}_{ij}^{R}+t_{32}^{R}\cdot\bar{v}_{ij}^{R}+f_{R}\cdot t_{33}^{R}\right)-f_{L}\cdot\left(t_{11}^{R}\cdot\bar{u}_{ij}^{R}+t_{12}^{R}\cdot\bar{v}_{ij}^{R}+f_{R}\cdot t_{13}^{R}\right)}$ are roughly estimated by substituting the calibrated intrinsic and extrinsic camera parameters, $\left(\bar{u}_{ij}^{L},\bar{v}_{ij}^{L}\right)$ and $\left(\bar{u}_{ij}^{R},\bar{v}_{ij}^{R}\right)$ into the equations.

III-C Overall Optimization of Intrinsic and Extrinsic Camera Parameters and MDM Parameters

In this section, we optimize the intrinsic and extrinsic camera parameters along with the MDM parameters based on the known distances between adjacent points on the checkerboard. First, according to (15), by eliminating $z_{L}$ and $z_{R}$, we obtain (19):

$${\left[\!\!\!\begin{array}[]{ccc}u_{ij}^{L}\!\!\cdot\!\!m_{31}^{L}\!\!-\!\!m_{11}^{L}\!\!&\!\!u_{ij}^{L}\!\!\cdot\!\!m_{32}^{L}\!\!-\!\!m_{12}^{L}\!\!&\!\!u_{ij}^{L}\!\!\cdot\!\!m_{33}^{L}\!\!-\!\!m_{13}^{L}\\ v_{ij}^{L}\!\!\cdot\!\!m_{31}^{L}\!\!-\!\!m_{21}^{L}\!\!&\!\!v_{ij}^{L}\!\!\cdot\!\!m_{32}^{L}\!\!-\!\!m_{22}^{L}\!\!&\!\!v_{ij}^{L}\!\!\cdot\!\!m_{33}^{L}\!\!-\!\!m_{23}^{L}\\ u_{ij}^{R}\!\!\cdot\!\!m_{31}^{R}\!\!-\!\!m_{11}^{R}\!\!&\!\!u_{ij}^{R}\!\!\cdot\!\!m_{32}^{R}\!\!-\!\!m_{12}^{R}\!\!&\!\!u_{ij}^{R}\!\!\cdot\!\!m_{33}^{R}\!\!-\!\!m_{13}^{R}\\ v_{ij}^{R}\!\!\cdot\!\!m_{31}^{R}\!\!-\!\!m_{21}^{R}\!\!&\!\!v_{ij}^{R}\!\!\cdot\!\!m_{32}^{R}\!\!-\!\!m_{22}^{R}\!\!&\!\!v_{ij}^{R}\!\!\cdot\!\!m_{33}^{R}\!\!-\!\!m_{23}^{R}\end{array}\!\!\!\right]\!\!\!\left[\begin{array}[]{c}X_{ij}\\ Y_{ij}\\ Z_{ij}\end{array}\right]}$$ (17)

$$=\left[\begin{array}[]{l}m_{14}^{L}\!\!-\!\!u_{ij}^{L}\!\!\cdot\!\!m_{34}^{L}\\ m_{24}^{L}\!\!-\!\!v_{ij}^{L}\!\!\cdot\!\!m_{34}^{L}\\ m_{14}^{R}\!\!-\!\!u_{ij}^{R}\!\!\cdot\!\!m_{34}^{R}\\ m_{24}^{R}\!\!-\!\!v_{ij}^{R}\!\!\cdot\!\!m_{34}^{R}\end{array}\right]$$

To solve $\left[X_{ij},Y_{ij},Z_{ij}\right]^{T}$, (19) is rewritten as follows:

$$\left\{\!\!\!\!\!\begin{array}[]{c}\boldsymbol{A}\!=\!\left[\!\!\!\!\begin{array}[]{ccc}u_{ij}^{L}\!\cdot\!m_{31}^{L}\!\!-\!\!m_{11}^{L}\!\!&\!\!u_{ij}^{L}\!\cdot\!m_{32}^{L}\!\!-\!\!m_{12}^{L}\!\!&\!\!u_{ij}^{L}\!\cdot\!m_{33}^{L}\!\!-\!\!m_{13}^{L}\\ v_{ij}^{L}\!\cdot\!m_{31}^{L}\!\!-\!\!m_{21}^{L}\!\!&\!\!v_{ij}^{L}\!\cdot\!m_{32}^{L}\!\!-\!\!m_{22}^{L}\!\!&\!\!v_{ij}^{L}\!\cdot\!m_{33}^{L}\!\!-\!\!m_{23}^{L}\\ u_{ij}^{R}\!\cdot\!m_{31}^{R}\!\!-\!\!m_{11}^{R}\!\!&\!\!u_{ij}^{R}\!\cdot\!m_{32}^{R}\!\!-\!\!m_{12}^{R}\!\!&\!\!u_{ij}^{R}\!\cdot\!m_{33}^{R}\!\!-\!\!m_{13}^{R}\\ v_{ij}^{L}\!\cdot\!m_{31}^{R}\!\!-\!\!m_{21}^{R}\!\!&\!\!v_{ij}^{R}\!\cdot\!m_{32}^{R}\!\!-\!\!m_{22}^{R}\!\!&\!\!v_{ij}^{R}\!\cdot\!m_{33}^{R}\!\!-\!\!m_{23}^{R}\end{array}\!\!\!\!\right]\\ \boldsymbol{B}=\left[\begin{array}[]{l}m_{14}^{L}-u_{ij}^{L}\cdot m_{34}^{L}\\ m_{24}^{L}-v_{ij}^{L}\cdot m_{34}^{L}\\ m_{14}^{R}-u_{ij}^{R}\cdot m_{34}^{R}\\ m_{24}^{R}-v_{ij}^{R}\cdot m_{34}^{R}\end{array}\right]\\ \boldsymbol{H}_{i,j}^{l}=\left[\begin{array}[]{l}X_{ij}\\ Y_{ij}\\ Z_{ij}\end{array}\right]=\left(\boldsymbol{A}^{T}\boldsymbol{A}\right)^{-1}\boldsymbol{A}^{T}\boldsymbol{B}\end{array}\right.$$ (18)

Here, $\boldsymbol{A}^{T}$ is the transposition of matrix $\boldsymbol{A}$; $\left(\boldsymbol{A}^{T}\boldsymbol{A}\right)^{-1}$ is the inverse of matrix $\left(\boldsymbol{A}^{T}\boldsymbol{A}\right)$. Then, the objective function is described by (21):

$$E\!\!=\!\!\min\!\!\!\sum_{l=1}^{w}\!\!\sum_{i=1}^{m\!-\!1}\!\!\sum_{j=1}^{n\!-\!1}\!\left(||\!|\boldsymbol{H}_{i,j}^{l}\!-\!\boldsymbol{H}_{i,j\!+\!1}^{l}\|\!\!-\!\!a|\!\!+\!\!|\|\boldsymbol{H}_{i,j}^{l}\!\!-\!\!\boldsymbol{H}_{i\!+\!1,j}^{l}\|\!-\!b\!\mid\!\right)$$ (19)

Here, $\boldsymbol{H}_{i,j}^{l}$ presents the 3D information of corner points in the $i$-th row and $j$-th column in the $l$-th checkerboard image related to the intrinsic and extrinsic camera parameters and the MDM parameters; $a$ and $b$ represent the actual length and width of the rectangle on the checkerboard plane, respectively. Finally, the Levenberg-Marquardt optimization method is employed to optimize the intrinsic and extrinsic camera parameters and the MDM parameters. The optimized initial values are the intrinsic and extrinsic camera parameters calibrated by using the Zhang’s calibration method and the MDM parameters calibrated using linear constraints. Based on the optimization variables, optimized initial values, and optimization objective function, the optimized values of the intrinsic and extrinsic camera parameters and the MDM parameters are obtained.

III-D Iteration-Based 3D Information Reconstruction Method

For the reconstruction of the 3D information of the target, the depth information of the target in the MDM or MDM-R is unknown, which can be roughly estimated using the linear pinhole model. However, the accuracy of the estimated depth information of the target is limited, because the linear pinhole model does not consider the lens distortion. Thus, in this section, an iteration-based 3D information reconstruction method is proposed.

Algorithm 1 : Iteration-based 3D Reconstruction Method

1:Intrinsic and extrinsic stereo camera parameters, MDM/MDM-R parameters, sufficiently small $\epsilon$

2:3D reconstructed information of the target $\left[X_{ij}^{k},Y_{ij}^{k},Z_{ij}^{k}\right]^{\text{T}}$

3:$k\Leftarrow 1$

4:$\!\boldsymbol{S}\!\!\Leftarrow\!\!\left(\!\!\boldsymbol{K}\!\!_{L}\!,\!\boldsymbol{K}\!\!_{R}\!,\!\boldsymbol{R}\!_{L}^{W}\!\!,\!\boldsymbol{T}\!_{L}^{W}\!\!,\!\boldsymbol{R}\!_{R}^{W}\!,\!\boldsymbol{T}\!_{R}^{W}\!,\!\boldsymbol{K}\!_{1}^{L}\!,\!\boldsymbol{K}\!_{2}^{L}\!,\!\boldsymbol{P}\!_{1}^{L}\!,\!\boldsymbol{P}\!_{2}^{L}\!,\!\boldsymbol{K}\!_{1}^{R}\!,\!\boldsymbol{K}\!_{2}^{R}\!,\!\boldsymbol{P}\!_{1}^{R}\!,\!\boldsymbol{P}\!_{2}^{R}\!\!\right)$

5:${}^{k}\!s_{ij}^{L}\!\!\Leftarrow\!\!\frac{f_{L}\cdot\left(f_{R}\cdot t_{14}^{L}-\bar{u}_{ij}^{R}\cdot t_{34}^{L}\right)}{\bar{u}_{ij}^{R}\cdot\left(t_{31}^{L}\cdot\bar{u}_{ij}^{L}+t_{32}^{L}\cdot\bar{v}_{ij}^{L}+f_{L}\cdot t_{33}^{L}\right)-f_{R}\cdot\left(t_{11}^{L}\cdot\bar{u}_{ij}^{L}+t_{12}^{L}\cdot\bar{v}_{ij}^{L}+f_{L}\cdot t_{13}^{L}\right)}$

6:${}^{k}\!s_{ij}^{R}\!\!\Leftarrow\!\!\frac{f_{R}\cdot\left(f_{L}\cdot t_{14}^{R}-\bar{u}_{ij}^{L}\cdot t_{34}^{R}\right)}{\bar{u}_{ij}^{L}\cdot\left(t_{31}^{R}\cdot\bar{u}_{ij}^{R}+t_{32}^{R}\cdot\bar{v}_{ij}^{R}+f_{R}\cdot t_{33}^{R}\right)-f_{L}\cdot\left(t_{11}^{R}\cdot\bar{u}_{ij}^{R}+t_{12}^{R}\cdot\bar{v}_{ij}^{R}+f_{R}\cdot t_{13}^{R}\right)}$

7:while $\delta\geq\epsilon$ do

8:     $[X_{ij}^{k},Y_{ij}^{k},Z_{ij}^{k}]^{T}\Leftarrow\boldsymbol{H}_{i,j}^{l}(\boldsymbol{S},^{k}\!s_{ij}^{L},^{k}\!s_{ij}^{R})$

9:     ${}^{k+1}\!s_{ij}^{L}\Leftarrow\left[0,0,1\right]\left[\boldsymbol{R}_{L}^{W},\boldsymbol{T}_{L}^{W}\right]\left[X_{ij}^{k},Y_{ij}^{k},Z_{ij}^{k},1\right]^{T}$

10:     ${}^{k+1}\!s_{ij}^{R}\Leftarrow\left[0,0,1\right]\left[\boldsymbol{R}_{R}^{W},\boldsymbol{T}_{R}^{W}\right]\left[X_{ij}^{k},Y_{ij}^{k},Z_{ij}^{k},1\right]^{T}$

11:     $[X_{ij}^{k+1},Y_{ij}^{k+1},Z_{ij}^{k+1}]^{T}\Leftarrow\boldsymbol{H}_{i,j}^{l}(\boldsymbol{S},^{k+1}\!s_{ij}^{L},^{k+1}\!s_{ij}^{R})$

12:     $\delta\Leftarrow\left\|\left[X_{ij}^{k+1},Y_{ij}^{k+1},Z_{ij}^{k+1}\right]^{T}-\left[X_{ij}^{k},Y_{ij}^{k},Z_{ij}^{k}\right]^{T}\right\|$

13:     $k\Leftarrow k+1$

14:end while

15:return $\left[X_{ij}^{k},Y_{ij}^{k},Z_{ij}^{k}\right]^{\text{T}}$

Using the proposed iteration-based 3D information reconstruction method, the depth information of the target in the MDM or MDM-R is updated iteratively, which improves the overall reconstruction accuracy. The detailed implementation is shown in Algorithm 1. In the above iteration-based 3D information reconstruction method, the inputs are the intrinsic and extrinsic stereo camera parameters, the MDM/MDM-R parameters, and a sufficiently small $\varepsilon$. First, the initial depth information of the target is estimated using the pinhole model; consequently, the iteration algorithm runs, and the depth information of the target updates accordingly. When $\delta<\varepsilon$, the process of iteration is finished, and more accurate 3D information of the target is reconstructed.

IV Experiments and Results

IV-A Experimental Setup

As shown in Fig. 3, the setup of the stereo vision system comprised two cameras (Nano-M2420, DALSA, Canada), two lenses (GX-5M-1216, GXTECH, China), and a computer. The resolution of the camera was $\text{2464}\times\text{2056}$ pixels, and the pixel size was $\text{3.45}\mu\mathrm{m}\times\text{3.45}\mu\mathrm{m}$. The focal length of each lens was 12 mm. The images were transmitted from the camera to the computer using the TCP/IP protocol, and the frequency of each camera was 20 Hz. The cameras were mounted on a beam-like holder with a quick-release buckle, and the holder was mounted on the optical platform.

Figure 3: Experimental setup

As shown in Fig. 4, the checkerboard plane contains 12×9 (150×105 mm) squares. There are 8 horizontal and 11 vertical lines distributed on the checkerboard, and the lines intersect to form a total of 88 corner points. The distances between adjacent corner points are 15 mm with a positioning accuracy of 0.005 mm.

Figure 4: Schematic diagram of the checkerboard layout

Figure 5: Relationship between the calibration accuracy and the number of calibration image pairs. (a) Relationship between the number of calibration images and $E_{1}^{i,j}$ of the MDM-R. (b) Relationship between the number of calibration images and $E_{1}^{i,j}$ of the MDM.

IV-B Relationship Between the Number of Calibration Image Pairs and the Calibration Accuracy of the MDM

We conducted several experiments to demonstrate the relationship between the number of calibration image pairs and the calibration accuracy of the MDM. The stereo vision system was used to acquire 8, 9, 10, 11, 12, 13, and 14 pairs of calibration images with different checkerboard orientations, respectively. Then, the proposed method was used to calibrate the MDM-R and the MDM using different pairs of calibration images. Finally, another 5 pairs of images with different checkerboard orientations were acquired to validate the accuracy of the MDM-R and the MDM, respectively.

The calibration error was defined as:

$$E_{1}^{i,j}=\left|\|\boldsymbol{H}_{i,j}^{l}-\boldsymbol{H}_{i,j+1}^{l}\|-a\right|$$ (20)

where $i=1,2,\ldots,8$ ; $j=1,2,\ldots,10$; $l$ represents different calibration images. Fig. 5(a) shows the relationship between the number of calibration image pairs and $E_{1}^{i,j}$ for a total of 5 tested calibration image pairs of the MDM-R. From Fig. 5(a), the $E_{1}^{i,j}$ of the MDM-R tended to decrease as the number of calibration image pairs increased. In particular, the $E_{1}^{i,j}$ decreased rapidly when the number of calibration image pairs ranged from 8 to 12. However, it decreased slowly when the number of calibration image pairs exceeded 13. Fig. 5(b) shows the relationship between the number of calibration image pairs and the $E_{1}^{i,j}$ of the MDM. The results indicated that $E_{1}^{i,j}$ decreased slowly when the number of calibration image pairs ranged from 8 to 14, but the maximum error decreased rapidly. Based on the results shown in Fig. 5, the calibration accuracy was satisfied for the MDM-R and the MDM with 13 pairs of calibration images.

TABLE II: calibration errors and calibration time with different lens distortion models.

Calibration Errors with Different Distortion Models ($\mu$m)						Calibration Time (s)
Brown’s radial distortion model	Li’s radial distortion model	MDM-R	Brown’s model	Li’s model	MDM	Brown’s model	Li’s model	MDM
167.5	79.0	35.3	73.1	43.5	18.9	342	1328	351

IV-C Calibration Accuracy Comparison

In this section, the calibration accuracy of the MDM-R and MDM was verified. Besides, the efficiency of the proposed calibration method was confirmed.

In the experiment, firstly, the checkerboard was placed within a depth range of 500 mm to 900 mm from the cameras. Secondly, according to Section II.B, 13 pairs of calibration images with different checkerboard orientations were acquired. Thirdly, our proposed calibration method was used to calibrate the MDM-R and MDM, while Zhang’s calibration method was adopted for calibrating Brown’s radial distortion model and Brown’s distortion model. In addition, according to Li’s calibration method, Li’s radial distortion model and Li’s distortion model were calibrated. During the calibration of these lens distortion models, the calibration time was recorded simultaneously. The calibration errors and corresponding calibration time with different distortion models are shown in Table II.

From the results in Table II, the calibration error of the MDM-R decreased by 55.32% and 78.93% compared to that of Li’s radial distortion model and Brown’s radial distortion model, respectively. Additionally, the calibration error of the MDM decreased by 56.55% and 74.15% compared to that of Li’s distortion model and Brown’s distortion model, respectively. Without a specific requirement of the checkerboard’s orientation compared with Li’s calibration method, the calibration time of the MDM and Brown’s distortion model was roughly four times shorter than that of Li’s distortion model. This demonstrated that our proposed calibration method had higher calibration efficiency. Moreover, depth information was considered in the MDM and Li’s distortion model, resulting in higher calibration accuracy compared to Brown’s distortion model. Due to more parameters needing to be calibrated in Li’s distortion model compared with the MDM, the calibration accuracy of Li’s distortion model was lower than that of the MDM.

Figure 6: The one-dimensional (1D) target

IV-D Measurement Accuracy Comparison

In this section, the measurement accuracy of the MDM and MDM-R was verified using a 1D target, and the proposed iteration-based reconstruction method was also validated. As shown in Fig. 6, the distance between the corner points $Q_{1}$ and $Q_{2}$ on the 1D target was precisely known as 150 mm. The positioning accuracy of the 1D target was 0.005 mm.

In the experiment, the 1D target was placed within the depth range of 680 mm to 800 mm sequentially, and the image pairs of the 1D target were acquired accordingly for measurement accuracy verification. Then, the distance between the corner points $Q_{1}$ and $Q_{2}$ was measured at different depths using the stereo vision system with different lens distortion models and 3D reconstruction methods.

Brown’s radial distortion model and Li’s radial distortion model were calibrated using Zhang’s calibration method and Li’s calibration method, respectively. Brown’s and Li’s distortion models were calibrated using Zhang’s calibration method and Li’s calibration method, respectively. The MDM-R and MDM were calibrated by the proposed calibration method. The non-iteration reconstruction method was employed for Brown’s radial distortion model, as well as Brown’s distortion model. For conducting 3D reconstruction, the depth information in Li’s radial distortion model and Li’s distortion model was roughly estimated. However, the iteration-based reconstruction method was adopted to address the same problem in MDM-R and MDM. The initial depth information was obtained by pin-hole model, and Algorithm 1 in Section III.B was used to iteratively calculate more accurate depth information. The 3D reconstruction error using the iteration-based reconstruction method was compared with that using the non-iteration reconstruction method. The measurement errors with these lens distortion models and 3D reconstruction methods at different depths are shown in Table III.

From the results in Table III, it was obvious that the measurement errors using Li’s radial distortion model and Li’s distortion model were larger than those using the MDM-R and MDM. Besides, the measurement errors using Brown’s radial distortion model and Brown’s distortion model were larger than those using Li’s radial distortion model and Li’s distortion model. Due to more accurate depth information calculated with each iteration, the measurement accuracy by using the iteration-based reconstruction method was higher. Compared with the non-iteration reconstruction method for the MDM, the measurement errors using the iteration-based reconstruction method were decreased by 9.08% (on average). The results in Table III not only verified the measurement accuracy of our proposed distortion model calibrated by the proposed calibration method but also demonstrated the validity of the iteration-based reconstruction method.

TABLE III: Measurement errors with different distortion models at different depths.

$\left\|Q_{1}-Q_{2}\right\|$ with different depths (mm)	Measurement Errors with Different Distortion Models ($\mu$m)
	Brown’s radial distortion model	Li’s radial distortion model	MDM-R		Brown’s model	Li’s model	MDM
			Non-iteration reconstruction	Iteration-based reconstruction			Non-iteration reconstruction	Iteration-based reconstruction
680	384.2	97.1	41.2	39.1	96.0	56.9	24.8	22.9
700	397.9	99.5	46.6	44.1	99.5	58.4	26.6	24.1
720	414.1	102.3	51.1	48.2	103.0	60.6	27.5	25.3
740	422.8	105.6	56.7	52.9	105.6	61.9	29.3	26.4
760	459.1	110.2	61.0	58.7	110.0	64.5	32.5	29.6
780	474.4	115.1	67.2	63.6	113.8	67.3	34.8	31.9
800	497.2	119.6	71.4	67.3	115.4	69.9	38.2	34.1

TABLE IV: Calibration results of the kinematic parameters of the robotic arm.

Link	$a_{i}(\mathrm{~{}mm})$				$\alpha_{i}\left({}^{\circ}\right)$				$d_{i}(\mathrm{~{}mm})$				$\theta_{i}\left({}^{\circ}\right)$
	Zhang’s method	MDM	Li’s method	Nominal value	Zhang’s method	MDM	Li’s method	Nominal value	Zhang’s method	MDM	Li’s method	Nominal value	Zhang’s method	MDM	Li’s method	Nominal value
1	0.80	0.35	0.81	0.00	0.87	1.42	0.24	0.00	365.88	366.29	365.19	365.00	0.00	0.00	0.00	0.00
2	1.06	1.82	0.46	0.00	91.07	91.53	90.31	90.00	65.51	66.39	65.18	65.00	0.00	0.00	0.00	0.00
3	0.84	0.54	0.67	0.00	-91.27	-90.54	-90.56	-90.00	396.49	395.19	396.13	395.00	0.00	0.00	0.00	0.00
4	21.46	20.38	20.52	220.00	91.03	90.59	90.16	90.00	-53.98	-53.52	-54.24	-55.00	0.00	0.00	0.00	0.00
5	-18.94	-19.13	-19.90	-20.00	-90.24	-91.92	-91.17	-90.00	385.27	386.95	385.71	385.00	0.00	0.00	0.00	0.00
6	1.45	1.82	0.31	0.00	90.21	90.43	91.23	90.00	100.35	101.51	100.19	100.00	91.46	90.73	91.37	90.00
7	111.35	110.38	110.16	110.00	90.52	90.76	90.64	90.00	135.10	134.41	135.11	135.00	0.00	0.00	0.00	0.00

IV-E Robotic Arm Kinematic Parameters Calibration

In this section, we applied our proposed stereo vision system to the kinematic parameters calibration of a robotic arm. As shown in Fig. 7, the experimental setup comprised a seven-degree-of-freedom robotic arm (Flexiv Robotics, Rizon 4), a stereo vision system, and a spherical marker. The stereo vision system consisted of two cameras, two lenses, and two LED lights. The marker had a diameter of 20 mm and was attached to the end-effector of the robotic arm.

Figure 7: Experimental setup for kinematic parameters calibration of the robotic arm

In the experiment, firstly, Brown’s distortion model, Li’s distortion model, and MDM were used to calibrate the stereo vision system. From the results in Table III, the measurement accuracy of the stereo vision system calibrated by MDM was higher than calibrated by Brown’s distortion model and Li’s distortion model. Secondly, the stereo vision system with different distortion models was used to calibrate the kinematic parameters of the robotic arm. The method in [20] was employed to calibrate the kinematic parameters of the robotic arm. Thirdly, to validate the performance of the kinematic parameters calibration, the end-effector of the robotic arm was moved to the 50 desired positions. The position of the end-effector of the robotic arm was measured by the stereo vision system.

Figure 8: Robotic arm end-effector position errors by using stereo vision system calibrated by different distortion models

The results are shown in Fig. 8. The position errors were calculated by the distance between the desired position and the measurement position of the end-effector of the robotic arm. The RMS position error before the kinematic parameters calibration of the robotic arm was 2833.6 $\mu$m. By using the stereo vision system with Zhang’s distortion model, MDM, and Li’s distortion model for calibrating the robotic arm, the RMS position errors were 1456.7 $\mu$m, 780.6 $\mu$m, and 1020.9 $\mu$m, respectively. The above results demonstrated that the stereo vision system with higher measurement accuracy for the kinematic parameters calibration of the robotic arm will result in higher motion accuracy of the robotic arm, which will improve the performance of the robotic arm with some tasks that require high accuracy.

V Conclusion

In this paper, we proposed an MDM, which considers both the radial and decentering lens distortions. Besides, a flexible three-step calibration method for the MDM was presented. The proposed flexible calibration method requires the cameras to observe a planar pattern in different orientations. Unlike the existing calibration methods for depth-dependent distortion models, the lenses are not required to be perpendicular to the planar pattern during our calibration procedures. In the experimental section, we demonstrated that the MDM improved the calibration accuracy by 56.55% and 74.15% compared with Li’s distortion model and the traditional Brown’s distortion model, respectively. Besides, the proposed calibration method was more efficient compared with Li’s calibration method. In addition, the results demonstrated that the measurement accuracy using the iteration-based reconstruction method was improved by 9.08%, on average, compared with using the non-iteration reconstruction method. In the future, we will apply our proposed stereo vision system to various scenarios, such as registration between different systems.