-
Hyperpixels: Pixel Filter Arrays of Multivariate Optical Elements for Optimized Spectral Imaging
Authors:
Calum Williams,
Richard Cousins,
Christopher J. Mellor,
Sarah E. Bohndiek,
George S. D. Gordon
Abstract:
We introduce the concept of `hyperpixels' in which each element of a pixel filter array (suitable for CMOS image sensor integration) has a spectral transmission tailored to a target spectral component expected in application-specific scenes. These are analogous to arrays of multivariate optical elements that could be used for sensing specific analytes. Spectral tailoring is achieved by engineering…
▽ More
We introduce the concept of `hyperpixels' in which each element of a pixel filter array (suitable for CMOS image sensor integration) has a spectral transmission tailored to a target spectral component expected in application-specific scenes. These are analogous to arrays of multivariate optical elements that could be used for sensing specific analytes. Spectral tailoring is achieved by engineering the heights of multiple sub-pixel Fabry-Perot resonators that cover each pixel area. We first present a design approach for hyperpixels, based on a matched filter concept and, as an exemplar, design a set of 4 hyperpixels tailored to optimally discriminate between 4 spectral reflectance targets. Next, we fabricate repeating 2x2 pixel filter arrays of these designs, alongside repeating 2x2 arrays of an optimal bandpass filters, perform both spectral and imaging characterization. Experimentally measured hyperpixel transmission spectra show a 2.4x reduction in unmixing matrix condition number (p=0.031) compared to the optimal band-pass set. Imaging experiments using the filter arrays with a monochrome sensor achieve a 3.47x reduction in unmixing matrix condition number (p=0.020) compared to the optimal band-pass set. This demonstrates the utility of the hyperpixel approach and shows its superiority even over the optimal bandpass case. We expect that with further improvements in design and fabrication processes increased performance may be obtained. Because the hyperpixels are straightforward to customize, fabricate and can be placed atop monochrome sensors, this approach is highly versatile and could be adapted to a wide range of real-time imaging applications which are limited by low SNR including micro-endoscopy, capsule endoscopy, industrial inspection and machine vision.
△ Less
Submitted 25 March, 2024;
originally announced March 2024.
-
Distribution-informed and wavelength-flexible data-driven photoacoustic oximetry
Authors:
Janek Gröhl,
Kylie Yeung,
Kevin Gu,
Thomas R. Else,
Monika Golinska,
Ellie V. Bunce,
Lina Hacker,
Sarah E. Bohndiek
Abstract:
Significance: Photoacoustic imaging (PAI) promises to measure spatially-resolved blood oxygen saturation, but suffers from a lack of accurate and robust spectral unmixing methods to deliver on this promise. Accurate blood oxygenation estimation could have important clinical applications, from cancer detection to quantifying inflammation.
Aim: This study addresses the inflexibility of existing da…
▽ More
Significance: Photoacoustic imaging (PAI) promises to measure spatially-resolved blood oxygen saturation, but suffers from a lack of accurate and robust spectral unmixing methods to deliver on this promise. Accurate blood oxygenation estimation could have important clinical applications, from cancer detection to quantifying inflammation.
Aim: This study addresses the inflexibility of existing data-driven methods for estimating blood oxygenation in PAI by introducing a recurrent neural network architecture.
Approach: We created 25 simulated training dataset variations to assess neural network performance. We used a long short-term memory network to implement a wavelength-flexible network architecture and proposed the Jensen-Shannon divergence to predict the most suitable training dataset.
Results: The network architecture can handle arbitrary input wavelengths and outperforms linear unmixing and the previously proposed learned spectral decolouring method. Small changes in the training data significantly affect the accuracy of our method, but we find that the Jensen-Shannon divergence correlates with the estimation error and is thus suitable for predicting the most appropriate training datasets for any given application.
Conclusions: A flexible data-driven network architecture combined with the Jensen-Shannon Divergence to predict the best training data set provides a promising direction that might enable robust data-driven photoacoustic oximetry for clinical use cases.
△ Less
Submitted 21 March, 2024;
originally announced March 2024.
-
Targeted Multispectral Filter Array Design for Endoscopic Cancer Detection in the Gastrointestinal Tract
Authors:
Michaela Taylor-Williams,
Ran Tao,
Travis W Sawyer,
Dale J Waterhouse,
Jonghee Yoon,
Sarah E Bohndiek
Abstract:
Colour differences between healthy and diseased tissue in the gastrointestinal tract are detected visually by clinicians during white light endoscopy (WLE); however, the earliest signs of disease are often just a slightly different shade of pink compared to healthy tissue. Here, we propose to target alternative colours for imaging to improve contrast using custom multispectral filter arrays (MSFAs…
▽ More
Colour differences between healthy and diseased tissue in the gastrointestinal tract are detected visually by clinicians during white light endoscopy (WLE); however, the earliest signs of disease are often just a slightly different shade of pink compared to healthy tissue. Here, we propose to target alternative colours for imaging to improve contrast using custom multispectral filter arrays (MSFAs) that could be deployed in an endoscopic chip-on-tip configuration. Using an open-source toolbox, Opti-MSFA, we examined the optimal design of MSFAs for early cancer detection in the gastrointestinal tract. The toolbox was first extended to use additional classification models (k-Nearest Neighbour, Support Vector Machine, and Spectral Angle Mapper). Using input spectral data from published clinical trials examining the oesophagus and colon, we optimised the design of MSFAs with 3 to 9 different bands. We examined the variation of the spectral and spatial classification accuracy as a function of number of bands. The MSFA designs have high classification accuracies, suggesting that future implementation in endoscopy hardware could potentially enable improved early detection of disease in the gastrointestinal tract during routine screening and surveillance. Optimal MSFA configurations can achieve similar classification accuracies as the full spectral data in an implementation that could be realised in far simpler hardware. The reduced number of spectral bands could enable future deployment of multispectral imaging in an endoscopic chip-on-tip configuration.
△ Less
Submitted 15 August, 2023;
originally announced August 2023.
-
Moving beyond simulation: data-driven quantitative photoacoustic imaging using tissue-mimicking phantoms
Authors:
Janek Gröhl,
Thomas R. Else,
Lina Hacker,
Ellie V. Bunce,
Paul W. Sweeney,
Sarah E. Bohndiek
Abstract:
Accurate measurement of optical absorption coefficients from photoacoustic imaging (PAI) data would enable direct map** of molecular concentrations, providing vital clinical insight. The ill-posed nature of the problem of absorption coefficient recovery has prohibited PAI from achieving this goal in living systems due to the domain gap between simulation and experiment. To bridge this gap, we in…
▽ More
Accurate measurement of optical absorption coefficients from photoacoustic imaging (PAI) data would enable direct map** of molecular concentrations, providing vital clinical insight. The ill-posed nature of the problem of absorption coefficient recovery has prohibited PAI from achieving this goal in living systems due to the domain gap between simulation and experiment. To bridge this gap, we introduce a collection of experimentally well-characterised imaging phantoms and their digital twins. This first-of-a-kind phantom data set enables supervised training of a U-Net on experimental data for pixel-wise estimation of absorption coefficients. We show that training on simulated data results in artefacts and biases in the estimates, reinforcing the existence of a domain gap between simulation and experiment. Training on experimentally acquired data, however, yielded more accurate and robust estimates of optical absorption coefficients. We compare the results to fluence correction with a Monte Carlo model from reference optical properties of the materials, which yields a quantification error of approximately 20%. Application of the trained U-Nets to a blood flow phantom demonstrated spectral biases when training on simulated data, while application to a mouse model highlighted the ability of both learning-based approaches to recover the depth-dependent loss of signal intensity. We demonstrate that training on experimental phantoms can restore the correlation of signal amplitudes measured in depth. While the absolute quantification error remains high and further improvements are needed, our results highlight the promise of deep learning to advance quantitative PAI.
△ Less
Submitted 11 June, 2023;
originally announced June 2023.
-
Mechanical-scan-free and multi-color super-resolution imaging with diffractive spot array illumination
Authors:
Ning Xu,
Sarah E. Bohndiek,
Zexing Li,
Cilong Zhang,
Qiaofeng Tan
Abstract:
Point-scanning microscopy approaches are transforming super-resolution imaging. Despite achieving parallel high-speed imaging using multifocal techniques, efficient multi-color capability with high-quality illumination is currently lacking. In this paper, we present for the first time Mechanical-scan-free and multi-Color Super-resolution Microscopy (MCoSM) by spot array illumination, which enables…
▽ More
Point-scanning microscopy approaches are transforming super-resolution imaging. Despite achieving parallel high-speed imaging using multifocal techniques, efficient multi-color capability with high-quality illumination is currently lacking. In this paper, we present for the first time Mechanical-scan-free and multi-Color Super-resolution Microscopy (MCoSM) by spot array illumination, which enables mechanical-scan-free super-resolution imaging with adjustable resolution and field of view (FoV) based on spatial light modulators (SLMs). Through 100s-10,000s super-resolution spot illumination with different FoV for imaging, we demonstrate the adjustable capacity of MCoSM. MCoSM extends current spectral imaging capabilities through a time-sharing process of different color illumination with phase-shifting scanning, while retaining the spatial flexibility of super-resolution imaging with diffractive spot array illumination. To showcase the prospects for further combining MCoSM with multi-color imaging, we also perform spectral unmixing (four-colors) on images of fluorescent beads at high resolution. MCoSM provides a versatile platform for studying molecular interactions in complex samples at the nanoscale level.
△ Less
Submitted 13 March, 2023;
originally announced March 2023.
-
Optimizing achromaticity in metalenses, and development of a layered thin-film metalens
Authors:
Calvin M. Hooper,
Sarah E. Bohndiek,
Calum Williams
Abstract:
Metalenses are ultrathin optical devices designed to replicate behavior of conventional refractive lenses, or lens arrays, utilizing nanoscale resonant structures to redirect incident light. These are often comprised of discrete meta-atoms such as nanoscale dielectric pillars. Achromatic focusing - associated with traditional multi-element refractive counterparts - is frequently attempted with sin…
▽ More
Metalenses are ultrathin optical devices designed to replicate behavior of conventional refractive lenses, or lens arrays, utilizing nanoscale resonant structures to redirect incident light. These are often comprised of discrete meta-atoms such as nanoscale dielectric pillars. Achromatic focusing - associated with traditional multi-element refractive counterparts - is frequently attempted with single-layer metalens designs, which has proven difficult to achieve with bounded refractive indices and total lens thickness. A recent study (F.Presutti and F.Monticone, 2020) formalized this, applying optical delay-line limitations to metalenses, resulting in a generalized trade-off in achromaticity for focusing systems. In this work, we (1) theoretically explore achromaticity in metalens design, and (2) propose a thin-film multilayer design as an alternative to the discrete meta-atom approach for large numerical aperture (NA) achromatic metalenses. It is shown that wavefront modulation can also be achieved with spectrally-varying transmission magnitudes, rather than purely matching a phase profile. In fact, even with a bounded refractive index, perfect achromatic operation over a given spectral range can be offset by imperfect operation elsewhere, resulting in a NA limited by the smallest general spectral feature controlled. These considerations lead to a generalized phase-matching optimization routine, and a thin-film metalens is simulated, utilizing layered TiO2/MgF2 with total thicknesses under 1 μm (20 layers), focusing across 6 simultaneous wavelengths (350-740 nm, Δλ~65 nm). A significant proportion (>40% spectral average) of the reflected light is focused for moderate NA (~0.35). With the maturity of the optical coating industry, the conformal thin-film approach reduces manufacturing complexity from its discrete nanoscale meta-atom equivalents.
△ Less
Submitted 8 February, 2022; v1 submitted 5 February, 2022;
originally announced February 2022.
-
Characterising optical fibre transmission matrices using metasurface reflector stacks for lensless imaging without distal access
Authors:
George S. D. Gordon,
Milana Gataric,
Alberto Gil C. P. Ramos,
Ralf Mouthaan,
Calum Williams,
Jonghee Yoon,
Timothy D. Wilkinson,
Sarah E. Bohndiek
Abstract:
The ability to form images through hair-thin optical fibres promises to open up new applications from biomedical imaging to industrial inspection. Unfortunately, deployment has been limited because small changes in mechanical deformation (e.g. bending) and temperature can completely scramble optical information, distorting images. Since such changes are dynamic, correcting them requires measuremen…
▽ More
The ability to form images through hair-thin optical fibres promises to open up new applications from biomedical imaging to industrial inspection. Unfortunately, deployment has been limited because small changes in mechanical deformation (e.g. bending) and temperature can completely scramble optical information, distorting images. Since such changes are dynamic, correcting them requires measurement of the fibre transmission matrix (TM) in situ immediately before imaging. TM calibration typically requires access to both the proximal and distal facets of the fibre simultaneously, which is not feasible during most realistic usage scenarios without compromising the thin form factor with bulky distal optics. Here, we introduce a new approach to determine the TM of multi-mode fibre (MMF) or multi-core fibre (MCF) in a reflection-mode configuration without access to the distal facet. A thin stack of structured metasurface reflectors is used at the distal facet to introduce wavelength-dependent, spatially heterogeneous reflectance profiles. We derive a first-order fibre model that compensates these wavelength-dependent changes in the TM and show that, consequently, the reflected data at 3 wavelengths can be used to unambiguously reconstruct the full TM by an iterative optimisation algorithm. We then present a method for sample illumination and imaging following TM reconstruction. Unlike previous approaches, our method does not require the TM to be unitary making it applicable to physically realistic fibre systems. We demonstrate TM reconstruction and imaging first using simulated non-unitary fibres and noisy reflection matrices, then using much larger experimentally-measured TMs of a densely-packed MCF, and finally on an experimentally-measured multi-wavelength set of TMs recorded from a MMF. Our findings pave the way for online transmission matrix calibration in situ in hair-thin optical fibres
△ Less
Submitted 5 April, 2019; v1 submitted 4 April, 2019;
originally announced April 2019.
-
Grayscale-to-color: Single-step fabrication of bespoke multispectral filter arrays
Authors:
Calum Williams,
George S. D. Gordon,
Sophia Gruber,
Timothy D. Wilkinson,
Sarah E. Bohndiek
Abstract:
Conventional cameras, such as in smartphones, capture wideband red, green and blue (RGB) spectral components, replicating human vision. Multispectral imaging (MSI) captures spatial and spectral information beyond our vision but typically requires bulky optical components and is expensive. Snapshot multispectral image sensors have been proposed as a key enabler for a plethora of MSI applications, f…
▽ More
Conventional cameras, such as in smartphones, capture wideband red, green and blue (RGB) spectral components, replicating human vision. Multispectral imaging (MSI) captures spatial and spectral information beyond our vision but typically requires bulky optical components and is expensive. Snapshot multispectral image sensors have been proposed as a key enabler for a plethora of MSI applications, from diagnostic medical imaging to remote sensing. To achieve low-cost and compact designs, spatially variant multispectral filter arrays (MSFAs) based on thin-film optical components are deposited atop image sensors. Conventional MSFAs achieve spectral filtering through either multi-layer stacks or pigment, requiring: complex mixtures of materials; additional lithographic steps for each additional wavelength; and large thicknesses to achieve high transmission efficiency. By contrast, we show here for the first time a single-step grayscale lithographic process that enables fabrication of bespoke MSFAs based on the Fabry-Perot resonances of spatially variant metal-insulator-metal (MIM) cavities, where the exposure dose controls insulator (cavity) thickness. We demonstrate customizable MSFAs scalable up to N-wavelength bands spanning the visible and near-infrared with high transmission efficiency (~75%) and narrow linewidths (~50 nm). Using this technique, we achieve multispectral imaging of several spectrally distinct target using our bespoke MIM-MSFAs fitted to a monochrome CMOS image sensor. Our unique framework provides an attractive alternative to conventional MSFA manufacture, by reducing both fabrication complexity and cost of these intricate optical devices, while increasing customizability.
△ Less
Submitted 27 January, 2019;
originally announced January 2019.
-
Quantitative phase and polarisation endoscopy applied to detection of early oesophageal tumourigenesis
Authors:
George S. D. Gordon,
James Joseph,
Maria P. Alcolea,
Travis Sawyer,
Alexander J. Macfaden,
Calum Williams,
Catherine R. M. Fitzpatrick,
Philip H. Jones,
Massimiliano di Pietro,
Rebecca C. Fitzgerald,
Timothy D. Wilkinson,
Sarah E. Bohndiek
Abstract:
Phase and polarisation of coherent light are highly perturbed by interaction with microstructural changes in pre-malignant tissue, holding promise for label-free early cancer detection in endoscopically accessible tissues such as the gastrointestinal tract. Flexible optical fibres used in conventional diagnostic endoscopy scramble phase and polarisation, restricting clinicians instead to low-contr…
▽ More
Phase and polarisation of coherent light are highly perturbed by interaction with microstructural changes in pre-malignant tissue, holding promise for label-free early cancer detection in endoscopically accessible tissues such as the gastrointestinal tract. Flexible optical fibres used in conventional diagnostic endoscopy scramble phase and polarisation, restricting clinicians instead to low-contrast amplitude-only imaging. Here, we unscramble phase and polarisation images by exploiting the near-diagonal multi-core fibre (MCF) transmission matrix to create a novel parallelised fibre characterisation architecture, scalable to arbitrary MCFs without additional experimental overhead. Our flexible MCF holographic endoscope produces full-field en-face images of amplitude, quantitative phase and resolved polarimetric properties using a low-cost laser diode and camera. We demonstrate that recovered phase enables computational re-focusing at working distances up to 1mm over a field-of-view up to 750$\times$750 $μm^2$. Furthermore, we demonstrate that the spatial distribution of phase and polarisation information enables label-free visualisation of early tumours in oesophageal mouse issue that are not identifiable using conventional amplitude-only information, a milestone towards future application for early cancer detection in endoscopy.
△ Less
Submitted 9 November, 2018;
originally announced November 2018.
-
Reconstruction of optical vector-fields with applications in endoscopic imaging
Authors:
Milana Gataric,
George S. D. Gordon,
Francesco Renna,
Alberto Gil C. P. Ramos,
Maria P. Alcolea,
Sarah E. Bohndiek
Abstract:
We introduce a framework for the reconstruction of the amplitude, phase and polarisation of an optical vector-field using calibration measurements acquired by an imaging device with an unknown linear transformation. By incorporating effective regularisation terms, this new approach is able to recover an optical vector-field with respect to an arbitrary representation system, which may be different…
▽ More
We introduce a framework for the reconstruction of the amplitude, phase and polarisation of an optical vector-field using calibration measurements acquired by an imaging device with an unknown linear transformation. By incorporating effective regularisation terms, this new approach is able to recover an optical vector-field with respect to an arbitrary representation system, which may be different from the one used in calibration. In particular, it enables the recovery of an optical vector-field with respect to a Fourier basis, which is shown to yield indicative features of increased scattering associated with tissue abnormalities. We demonstrate the effectiveness of our approach using synthetic holographic images as well as biological tissue samples in an experimental setting where measurements of an optical vector-field are acquired by a fibre endoscope, and observe that indeed the recovered Fourier coefficients are useful in distinguishing healthy tissues from lesions in early stages of oesophageal cancer.
△ Less
Submitted 18 July, 2018; v1 submitted 27 April, 2018;
originally announced April 2018.
