Showing 1–18 of 18 results for author: Bin, M

Nanocrystallites Modulate Intermolecular Interactions in Cryoprotected Protein Solutions

Authors: Mariia Filianina, Maddalena Bin, Sharon Berkowicz, Mario Reiser, Hailong Li, Sonja Timmermann, Malte Blankenburg, Katrin Amann-Winkel, Christian Gutt, Fivos Perakis

Abstract: Studying protein interactions at low temperatures has important implications for optimizing cryostorage processes of biological tissue, food, and protein-based drugs. One of the major challenges is related to the formation of ice nanocrystals which can occur even in the presence of cryoprotectants and can lead to protein denaturation. Here, using a combination of small- and wide-angle X-ray scatte… ▽ More Studying protein interactions at low temperatures has important implications for optimizing cryostorage processes of biological tissue, food, and protein-based drugs. One of the major challenges is related to the formation of ice nanocrystals which can occur even in the presence of cryoprotectants and can lead to protein denaturation. Here, using a combination of small- and wide-angle X-ray scattering (SAXS and WAXS), we investigate the structural evolution of concentrated Lysozyme solutions in a cryoprotected glycerol-water mixture upon cooling from room temperature (T=300 K) down to cryogenic temperatures (T=195 K). Upon cooling, we observe a transition near the melting temperature of the solution (T~245 K), which manifests both in the temperature dependence of the scattering intensity peak position reflecting protein-protein length scales (SAXS) and the interatomic distances within the solvent (WAXS). Upon thermal cycling, a hysteresis is observed in the scattering intensity, which is attributed to the formation of nanocrystallites in the order of 10 nm. The experimental data is well described by the two-Yukawa model, which indicates temperature-dependent changes in the short-range attraction of the protein-protein interaction potential. Our results demonstrate that the nanocrystal growth yields effectively stronger protein-protein attraction and influences the protein pair distribution function beyond the first coordination shell. △ Less

Submitted 12 April, 2023; originally announced April 2023.

Coherent X-ray Scattering Reveals Nanoscale Fluctuations in Hydrated Proteins

Authors: Maddalena Bin, Mario Reiser, Mariia Filianina, Sharon Berkowicz, Sudipta Das, Sonja Timmermann, Wojciech Roseker, Robert Bauer, Jonatan Öström, Aigerim Karina, Katrin Amann-Winkel, Marjorie Ladd-Parada, Fabian Westermeier, Michael Sprung, Johannes Möller, Felix Lehmkühler, Christian Gutt, Fivos Perakis

Abstract: Hydrated proteins undergo a transition in the deeply supercooled regime, which is attributed to rapid changes in hydration water and protein structural dynamics. Here, we investigate the nanoscale stress relaxation in hydrated lysozyme proteins stimulated and probed by X-ray Photon Correlation Spectroscopy (XPCS). This approach allows us to access the nanoscale dynamic response in the deeply super… ▽ More Hydrated proteins undergo a transition in the deeply supercooled regime, which is attributed to rapid changes in hydration water and protein structural dynamics. Here, we investigate the nanoscale stress relaxation in hydrated lysozyme proteins stimulated and probed by X-ray Photon Correlation Spectroscopy (XPCS). This approach allows us to access the nanoscale dynamic response in the deeply supercooled regime (T = 180 K) which is typically not accessible through equilibrium methods. The relaxation time constants exhibit Arrhenius temperature dependence upon cooling with a minimum in the Kohlrausch-Williams-Watts exponent at T = 227 K. The observed minimum is attributed to an increase in dynamical heterogeneity, which coincides with enhanced fluctuations observed in the two-time correlation functions and a maximum in the dynamic susceptibility quantified by the normalised variance $χ_T$. Our study provides new insights into X-ray stimulated stress relaxation and the underlying mechanisms behind spatio-temporal fluctuations in biological granular materials. △ Less

Submitted 26 January, 2023; originally announced January 2023.

Data-driven Output Regulation via Gaussian Processes and Luenberger Internal Models

Authors: Lorenzo Gentilini, Michelangelo Bin, Lorenzo Marconi

Abstract: This paper deals with the problem of adaptive output regulation for multivariable nonlinear systems by presenting a learning-based adaptive internal model-based design strategy. The approach builds on the recently proposed adaptive internal model design techniques based on the theory of nonlinear Luenberger observers, and the adaptation side is approached as a probabilistic regression problem. In… ▽ More This paper deals with the problem of adaptive output regulation for multivariable nonlinear systems by presenting a learning-based adaptive internal model-based design strategy. The approach builds on the recently proposed adaptive internal model design techniques based on the theory of nonlinear Luenberger observers, and the adaptation side is approached as a probabilistic regression problem. In particular, Gaussian process priors are employed to cope with the learning problem. Unlike the previous approaches in the field, here only coarse assumptions about the friend structure are required, making the proposed approach suitable for applications where the exosystem is highly uncertain. The paper presents performance bounds on the attained regulation error and numerical simulations showing how the proposed method outperforms previous approaches. △ Less

Submitted 28 October, 2022; originally announced October 2022.

Comments: arXiv admin note: text overlap with arXiv:2206.12225

Adaptive Nonlinear Regulation via Gaussian Process

Authors: Lorenzo Gentilini, Michelangelo Bin, Lorenzo Marconi

Abstract: The paper deals with the problem of output regulation of nonlinear systems by presenting a learning-based adaptive internal model-based design strategy. We borrow from the adaptive internal model design technique recently proposed in [1] and extend it by means of a Gaussian process regressor. The learning-based adaptation is performed by following an "event-triggered" logic so that hybrid tools ar… ▽ More The paper deals with the problem of output regulation of nonlinear systems by presenting a learning-based adaptive internal model-based design strategy. We borrow from the adaptive internal model design technique recently proposed in [1] and extend it by means of a Gaussian process regressor. The learning-based adaptation is performed by following an "event-triggered" logic so that hybrid tools are used to analyse the resulting closed-loop system. Unlike the approach proposed in [1] where the friend is supposed to belong to a specific finite-dimensional model set, here we only require smoothness of the ideal steady-state control action. The paper also presents numerical simulations showing how the proposed method outperforms previous approaches. △ Less

Submitted 24 June, 2022; originally announced June 2022.

Comments: Submitted to CDC2022

Stability, Linear Convergence, and Robustness of the Wang-Elia Algorithm for Distributed Consensus Optimization

Authors: Michelangelo Bin, Ivano Notarnicola, Thomas Parisini

Abstract: We revisit an algorithm for distributed consensus optimization proposed in 2010 by J. Wang and N. Elia. By means of a Lyapunov-based analysis, we prove input-to-state stability of the algorithm relative to a closed invariant set composed of optimal equilibria and with respect to perturbations affecting the algorithm's dynamics. In the absence of perturbations, this result implies linear convergenc… ▽ More We revisit an algorithm for distributed consensus optimization proposed in 2010 by J. Wang and N. Elia. By means of a Lyapunov-based analysis, we prove input-to-state stability of the algorithm relative to a closed invariant set composed of optimal equilibria and with respect to perturbations affecting the algorithm's dynamics. In the absence of perturbations, this result implies linear convergence of the local estimates and Lyapunov stability of the optimal steady state. Moreover, we unveil fundamental connections with the well-known Gradient Tracking and with distributed integral control. Overall, our results suggest that a control theoretic approach can have a considerable impact on (distributed) optimization, especially when robustness is considered. △ Less

Submitted 21 April, 2022; originally announced April 2022.

Resolving molecular diffusion and aggregation of antibody proteins with megahertz X-ray free-electron laser pulses

Authors: Mario Reiser, Anita Girelli, Anastasia Ragulskaya, Sudipta Das, Sharon Berkowicz, Maddalena Bin, Marjorie Ladd-Parada, Mariia Filianina, Hanna-Friederike Poggemann, Nafisa Begam, Mohammad Sayed Akhundzadeh, Sonja Timmermann, Lisa Randolph, Yuriy Chushkin, Tilo Seydel, Ulrike Boesenberg, Jörg Hallmann, Johannes Möller, Angel Rodriguez-Fernandez, Robert Rosca, Robert Schaffer, Markus Scholz, Roman Shayduk, Alexey Zozulya, Anders Madsen , et al. (4 additional authors not shown)

Abstract: X-ray free-electron lasers (XFELs) with megahertz repetition rate can provide novel insights into structural dynamics of biological macromolecule solutions. However, very high dose rates can lead to beam-induced dynamics and structural changes due to radiation damage. Here, we probe the dynamics of dense antibody protein (Ig-PEG) solutions using megahertz X-ray photon correlation spectroscopy (MHz… ▽ More X-ray free-electron lasers (XFELs) with megahertz repetition rate can provide novel insights into structural dynamics of biological macromolecule solutions. However, very high dose rates can lead to beam-induced dynamics and structural changes due to radiation damage. Here, we probe the dynamics of dense antibody protein (Ig-PEG) solutions using megahertz X-ray photon correlation spectroscopy (MHz-XPCS) at the European XFEL. By varying the total dose and dose rate, we identify a regime for measuring the motion of proteins in their first coordination shell, quantify XFEL-induced effects such as driven motion, and map out the extent of agglomeration dynamics. The results indicate that for average dose rates below $1.06\,\mathrm{kGy}\mathrm{μs}^{-1}$ in a time window up to $10\,\mathrm{μs}$, it is possible to capture the protein dynamics before the onset of beam induced aggregation. We refer to this approach as correlation before aggregation and demonstrate that MHz-XPCS bridges an important spatio-temporal gap in measurement techniques for biological samples. △ Less

Submitted 5 October, 2022; v1 submitted 22 February, 2022; originally announced February 2022.

Comments: 22 pages, 6 figures

Journal ref: Nat Commun 13, 5528 (2022)

Output Regulation by Postprocessing Internal Models for a Class of Multivariable Nonlinear Systems

Authors: Michelangelo Bin, Lorenzo Marconi

Abstract: In this paper we propose a new design paradigm, which employing a postprocessing internal model unit, to approach the problem of output regulation for a class of multivariable minimum-phase nonlinear systems possessing a partial normal form. Contrary to previous approaches, the proposed regulator handles control inputs of dimension larger than the number of regulated variables, provided that a con… ▽ More In this paper we propose a new design paradigm, which employing a postprocessing internal model unit, to approach the problem of output regulation for a class of multivariable minimum-phase nonlinear systems possessing a partial normal form. Contrary to previous approaches, the proposed regulator handles control inputs of dimension larger than the number of regulated variables, provided that a controllability assumption holds, and can employ additional measurements that need not to vanish at the ideal error-zeroing steady state, but that can be useful for stabilization purposes or to fulfil the minimum-phase requirement. Conditions for practical and asymptotic output regulation are given, underlying how in postprocessing schemes the design of internal models is necessarily intertwined with that of the stabilizer. △ Less

Submitted 24 November, 2021; originally announced November 2021.

Comments: The published version contains a few small rendering issues in some formulae. Here, these are corrected

Journal ref: International Journal of Robust and Nonlinear Control, vol. 30, pp. 1115-1140, 2020

On node ranking in graphs

Authors: Ekaterina Dudkina, Michelangelo Bin, Jane Breen, Emanuele Crisostomi, Pietro Ferraro, Steve Kirkland, Jakub Marecek, Roderick Murray-Smith, Thomas Parisini, Lewi Stone, Serife Yilmaz, Robert Shorten

Abstract: The ranking of nodes in a network according to their ``importance'' is a classic problem that has attracted the interest of different scientific communities in the last decades. The current COVID-19 pandemic has recently rejuvenated the interest in this problem, as it is related to the selection of which individuals should be tested in a population of asymptomatic individuals, or which individuals… ▽ More The ranking of nodes in a network according to their ``importance'' is a classic problem that has attracted the interest of different scientific communities in the last decades. The current COVID-19 pandemic has recently rejuvenated the interest in this problem, as it is related to the selection of which individuals should be tested in a population of asymptomatic individuals, or which individuals should be vaccinated first. Motivated by the COVID-19 spreading dynamics, in this paper we review the most popular methods for node ranking in undirected unweighted graphs, and compare their performance in a benchmark realistic network, that takes into account the community-based structure of society. Also, we generalize a classic benchmark network originally proposed by Newman for ranking nodes in unweighted graphs, to show how ranks change in the weighted case. △ Less

Submitted 20 July, 2021; originally announced July 2021.

Journal ref: International Journal of Control, 2023

A Distributed Methodology for Approximate Uniform Global Minimum Sharing

Authors: Michelangelo Bin, Thomas Parisini

Abstract: The paper deals with the distributed minimum sharing problem: a set of decision-makers compute the minimum of some local quantities of interest in a distributed and decentralized way by exchanging information through a communication network. We propose an adjustable approximate solution which enjoys several properties of crucial importance in applications. In particular, the proposed solution has… ▽ More The paper deals with the distributed minimum sharing problem: a set of decision-makers compute the minimum of some local quantities of interest in a distributed and decentralized way by exchanging information through a communication network. We propose an adjustable approximate solution which enjoys several properties of crucial importance in applications. In particular, the proposed solution has good decentralization properties and it is scalable in that the number of local variables does not grow with the size or topology of the communication network. Moreover, a global and uniform (both in the initial time and in the initial conditions) asymptotic stability result is provided towards a steady state which can be made arbitrarily close to the sought minimum. Exact asymptotic convergence can be recovered at the price of losing uniformity with respect to the initial time. △ Less

Submitted 25 June, 2021; v1 submitted 16 December, 2020; originally announced December 2020.

Journal ref: Automatica, vol. 131, pp. 109777, 2021

About Robustness of Control Systems Embedding an Internal Model

Authors: Michelangelo Bin, Daniele Astolfi, Lorenzo Marconi

Abstract: Robustness is a basic property of any control system. In the context of linear output regulation, it was proved that embedding an internal model of the exogenous signals is necessary and sufficient to achieve tracking of the desired reference signals in spite of external disturbances and parametric uncertainties. This result is commonly known as "internal model principle". A complete extension of… ▽ More Robustness is a basic property of any control system. In the context of linear output regulation, it was proved that embedding an internal model of the exogenous signals is necessary and sufficient to achieve tracking of the desired reference signals in spite of external disturbances and parametric uncertainties. This result is commonly known as "internal model principle". A complete extension of such linear result to general nonlinear systems is still an open problem, which is exacerbated by the large number of alternative definitions of uncertainty and desired control goals that are possible in a nonlinear setting. In this paper, we develop a general framework in which all these different notions can be formally characterized in a unifying way. Classical results are reinterpreted in the proposed setting, and new results and insights are presented with a focus on robust rejection/tracking of arbitrary harmonic content. Moreover, we show by counter-example that, in the relevant case of continuous unstructured uncertainties, there are problems for which no smooth finite-dimensional robust regulator exists ensuring exact regulation. △ Less

Submitted 22 April, 2021; v1 submitted 17 October, 2020; originally announced October 2020.

Comments: updated version 04.2021

Model Identification and Adaptive State Observation for a Class of Nonlinear Systems

Authors: Michelangelo Bin, Lorenzo Marconi

Abstract: In this paper we consider the joint problems of state estimation and model identification for a class of continuous-time nonlinear systems in output-feedback canonical form. An adaptive observer is proposed that combines an extended high-gain observer and a discrete-time identifier. The extended observer provides the identifier with a data set permitting the identification of the system model and… ▽ More In this paper we consider the joint problems of state estimation and model identification for a class of continuous-time nonlinear systems in output-feedback canonical form. An adaptive observer is proposed that combines an extended high-gain observer and a discrete-time identifier. The extended observer provides the identifier with a data set permitting the identification of the system model and the identifier adapts the extended observer according to the new estimated model. The design of the identifier is approached as a system identification problem and sufficient conditions are presented that, if satisfied, allow different identification algorithms to be used for the adaptation phase. The cases of recursive least-squares and multiresolution black-box identification via wavelet-based identifiers are specifically addressed. Stability results are provided relating the asymptotic estimation error to the prediction capabilities of the identifier. Robustness with respect to additive disturbances affecting the system equations and measurements is also established in terms of an input-to-state stability property relative to the noiseless estimates. △ Less

Submitted 30 November, 2020; v1 submitted 11 October, 2020; originally announced October 2020.

Comments: This is the accepted version of https://ieeexplore.ieee.org/document/9272829

Kemeny-based testing for COVID-19

Authors: Serife Yilmaz, Ekaterina Dudkina, Michelangelo Bin, Emanuele Crisostomi, Pietro Ferraro, Roderick Murray-Smith, Thomas Parisini, Lewi Stone, Robert Shorten

Abstract: Testing, tracking and tracing abilities have been identified as pivotal in hel** countries to safely reopen activities after the first wave of the COVID-19 virus. Contact tracing apps give the unprecedented possibility to reconstruct graphs of daily contacts, so the question is who should be tested? As human contact networks are known to exhibit community structure, in this paper we show that th… ▽ More Testing, tracking and tracing abilities have been identified as pivotal in hel** countries to safely reopen activities after the first wave of the COVID-19 virus. Contact tracing apps give the unprecedented possibility to reconstruct graphs of daily contacts, so the question is who should be tested? As human contact networks are known to exhibit community structure, in this paper we show that the Kemeny constant of a graph can be used to identify and analyze bridges between communities in a graph. Our "Kemeny indicator" is the change in Kemeny constant when a node or edge is removed from the graph. We show that testing individuals who are associated with large values of the Kemeny indicator can help in efficiently intercepting new virus outbreaks, when they are still in their early stage. Extensive simulations provide promising results in early identification and in blocking possible "super-spreaders" links that transmit disease between different communities. △ Less

Submitted 24 July, 2020; v1 submitted 15 June, 2020; originally announced June 2020.

Post-Lockdown Abatement of COVID-19 by Fast Periodic Switching

Authors: M. Bin, P. Cheung, E. Crisostomi, P. Ferraro, H. Lhachemi, R. Murray-Smith, C. Myant, T. Parisini, R. Shorten, S. Stein, L. Stone

Abstract: COVID-19 abatement strategies have risks and uncertainties which could lead to repeating waves of infection. We show -- as proof of concept grounded on rigorous mathematical evidence -- that periodic, high-frequency alternation of into, and out-of, lockdown effectively mitigates second-wave effects, while allowing continued, albeit reduced, economic activity. Periodicity confers (i) predictability… ▽ More COVID-19 abatement strategies have risks and uncertainties which could lead to repeating waves of infection. We show -- as proof of concept grounded on rigorous mathematical evidence -- that periodic, high-frequency alternation of into, and out-of, lockdown effectively mitigates second-wave effects, while allowing continued, albeit reduced, economic activity. Periodicity confers (i) predictability, which is essential for economic sustainability, and (ii) robustness, since lockdown periods are not activated by uncertain measurements over short time scales. In turn -- while not eliminating the virus -- this fast switching policy is sustainable over time, and it mitigates the infection until a vaccine or treatment becomes available, while alleviating the social costs associated with long lockdowns. Typically, the policy might be in the form of 1-day of work followed by 6-days of lockdown every week (or perhaps 2 days working, 5 days off) and it can be modified at a slow-rate based on measurements filtered over longer time scales. Our results highlight the potential efficacy of high frequency switching interventions in post lockdown mitigation. All code is available on Github (https://github.com/V4p1d/FPSP_Covid19). A software tool has also been developed so that interested parties can explore the proof-of-concept system. △ Less

Submitted 27 October, 2020; v1 submitted 22 March, 2020; originally announced March 2020.

Comments: Version 6: title slightly changed and extended mathematical results underpinning the mitigation strategy provided, revised sensitivity analysis and discussion given

Stable Neural Flows

Authors: Stefano Massaroli, Michael Poli, Michelangelo Bin, **kyoo Park, Atsushi Yamashita, Hajime Asama

Abstract: We introduce a provably stable variant of neural ordinary differential equations (neural ODEs) whose trajectories evolve on an energy functional parametrised by a neural network. Stable neural flows provide an implicit guarantee on asymptotic stability of the depth-flows, leading to robustness against input perturbations and low computational burden for the numerical solver. The learning procedure… ▽ More We introduce a provably stable variant of neural ordinary differential equations (neural ODEs) whose trajectories evolve on an energy functional parametrised by a neural network. Stable neural flows provide an implicit guarantee on asymptotic stability of the depth-flows, leading to robustness against input perturbations and low computational burden for the numerical solver. The learning procedure is cast as an optimal control problem, and an approximate solution is proposed based on adjoint sensivity analysis. We further introduce novel regularizers designed to ease the optimization process and speed up convergence. The proposed model class is evaluated on non-linear classification and function approximation tasks. △ Less

Submitted 18 March, 2020; originally announced March 2020.

A System Theoretical Perspective to Gradient-Tracking Algorithms for Distributed Quadratic Optimization

Authors: Michelangelo Bin, Ivano Notarnicola, Lorenzo Marconi, Giuseppe Notarstefano

Abstract: In this paper we consider a recently developed distributed optimization algorithm based on gradient tracking. We propose a system theory framework to analyze its structural properties on a preliminary, quadratic optimization set-up. Specifically, we focus on a scenario in which agents in a static network want to cooperatively minimize the sum of quadratic cost functions. We show that the gradient… ▽ More In this paper we consider a recently developed distributed optimization algorithm based on gradient tracking. We propose a system theory framework to analyze its structural properties on a preliminary, quadratic optimization set-up. Specifically, we focus on a scenario in which agents in a static network want to cooperatively minimize the sum of quadratic cost functions. We show that the gradient tracking distributed algorithm for the investigated program can be viewed as a sparse closed-loop linear system in which the dynamic state-feedback controller includes consensus matrices and optimization (stepsize) parameters. The closed-loop system turns out to be not completely reachable and asymptotic stability can be shown restricted to a proper invariant set. Convergence to the global minimum, in turn, can be obtained only by means of a proper initialization. The proposed system interpretation of the distributed algorithm provides also additional insights on other structural properties and possible design choices that are discussed in the last part of the paper as a starting point for future developments. △ Less

Submitted 15 November, 2019; originally announced November 2019.

Approximate Nonlinear Regulation via Identification-Based Adaptive Internal Models

Authors: Michelangelo Bin, Pauline Bernard, Lorenzo Marconi

Abstract: This paper concerns the problem of adaptive output regulation for multivariable nonlinear systems in normal form. We present a regulator employing an adaptive internal model of the exogenous signals based on the theory of nonlinear Luenberger observers. Adaptation is performed by means of discrete-time system identification schemes, in which every algorithm fulfilling some optimality and stability… ▽ More This paper concerns the problem of adaptive output regulation for multivariable nonlinear systems in normal form. We present a regulator employing an adaptive internal model of the exogenous signals based on the theory of nonlinear Luenberger observers. Adaptation is performed by means of discrete-time system identification schemes, in which every algorithm fulfilling some optimality and stability conditions can be used. Practical and approximate regulation results are given relating the prediction capabilities of the identified model to the asymptotic bound on the regulated variables, which become asymptotic whenever a "right" internal model exists in the identifier's model set. The proposed approach, moreover, does not require "high-gain" stabilization actions. △ Less

Submitted 15 September, 2020; v1 submitted 11 July, 2019; originally announced July 2019.

Comments: This new version contains minor corrections and modifications in the text

"Class-Type" Identification-Based Internal Models in Multivariable Nonlinear Output Regulation

Authors: Michelangelo Bin, Lorenzo Marconi

Abstract: The paper deals with the problem of output regulation in a "non-equilibrium" context for a special class of multivariable nonlinear systems stabilizable by high-gain feedback. A post-processing internal model design suitable for the multivariable nature of the system, which might have more inputs than regulation errors, is proposed. Uncertainties in the system and exosystem are dealt with by assum… ▽ More The paper deals with the problem of output regulation in a "non-equilibrium" context for a special class of multivariable nonlinear systems stabilizable by high-gain feedback. A post-processing internal model design suitable for the multivariable nature of the system, which might have more inputs than regulation errors, is proposed. Uncertainties in the system and exosystem are dealt with by assuming that the ideal steady state input belongs to a certain "class of signals" by which an appropriate model set for the internal model can be derived. The adaptation mechanism for the internal model is then cast as an identification problem and a least square solution is specifically developed. In line with recent developments in the field, the vision that emerges from the paper is that approximate, possibly asymptotic, regulation is the appropriate way of approaching the problem in a multivariable and uncertain context. New insights about the use of identification tools in the design of adaptive internal models are also presented. △ Less

Submitted 21 April, 2020; v1 submitted 15 May, 2018; originally announced May 2018.

Comments: This version is the peer-reviewed accepted post-print of the paper, and it substitutes the previous two versions. The published version is available at https://ieeexplore.ieee.org/document/8911463

Multi-Modal Hybrid Deep Neural Network for Speech Enhancement

Authors: Zhenzhou Wu, Sunil Sivadas, Yong Kiam Tan, Ma Bin, Rick Siow Mong Goh

Abstract: Deep Neural Networks (DNN) have been successful in en- hancing noisy speech signals. Enhancement is achieved by learning a nonlinear map** function from the features of the corrupted speech signal to that of the reference clean speech signal. The quality of predicted features can be improved by providing additional side channel information that is robust to noise, such as visual cues. In this pa… ▽ More Deep Neural Networks (DNN) have been successful in en- hancing noisy speech signals. Enhancement is achieved by learning a nonlinear map** function from the features of the corrupted speech signal to that of the reference clean speech signal. The quality of predicted features can be improved by providing additional side channel information that is robust to noise, such as visual cues. In this paper we propose a novel deep learning model inspired by insights from human audio visual perception. In the proposed unified hybrid architecture, features from a Convolution Neural Network (CNN) that processes the visual cues and features from a fully connected DNN that processes the audio signal are integrated using a Bidirectional Long Short-Term Memory (BiLSTM) network. The parameters of the hybrid model are jointly learned using backpropagation. We compare the quality of enhanced speech from the hybrid models with those from traditional DNN and BiLSTM models. △ Less

Submitted 15 June, 2016; originally announced June 2016.