Showing 1–50 of 197 results for author: Eckstein, M

Convolutional Neural Network Model Observers Discount Signal-like Anatomical Structures During Search in Virtual Digital Breast Tomosynthesis Phantoms

Authors: Aditya Jonnalagadda, Bruno B. Barufaldi, Andrew D. A. Maidment, Susan P. Weinstein, Craig K. Abbey, Miguel P. Eckstein

Abstract: Model observers are computational tools to evaluate and optimize task-based medical image quality. Linear model observers, such as the Channelized Hotelling Observer (CHO), predict human accuracy in detection tasks with a few possible signal locations in clinical phantoms or real anatomic backgrounds. In recent years, Convolutional Neural Networks (CNNs) have been proposed as a new type of model o… ▽ More Model observers are computational tools to evaluate and optimize task-based medical image quality. Linear model observers, such as the Channelized Hotelling Observer (CHO), predict human accuracy in detection tasks with a few possible signal locations in clinical phantoms or real anatomic backgrounds. In recent years, Convolutional Neural Networks (CNNs) have been proposed as a new type of model observer. What is not well understood is what CNNs add over the more common linear model observer approaches. We compare the CHO and CNN detection accuracy to the radiologist's accuracy in searching for two types of signals (mass and microcalcification) embedded in 2D/3D breast tomosynthesis phantoms (DBT). We show that the CHO model's accuracy is comparable to the CNN's performance for a location-known-exactly detection task. However, for the search task with 2D/3D DBT phantoms, the CHO's detection accuracy was significantly lower than the CNN accuracy. A comparison to the radiologist's accuracy showed that the CNN but not the CHO could match or exceed the radiologist's accuracy in the 2D microcalcification and 3D mass search conditions. An analysis of the eye position showed that radiologists fixated more often and longer at the locations corresponding to CNN false positives. Most CHO false positives were the phantom's normal anatomy and were not fixated by radiologists. In conclusion, we show that CNNs can be used as an anthropomorphic model observer for the search task for which traditional linear model observers fail due to their inability to discount false positives arising from the anatomical backgrounds. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Comments: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

From Text to Pixel: Advancing Long-Context Understanding in MLLMs

Authors: Yujie Lu, Xiujun Li, Tsu-Jui Fu, Miguel Eckstein, William Yang Wang

Abstract: The rapid progress in Multimodal Large Language Models (MLLMs) has significantly advanced their ability to process and understand complex visual and textual information. However, the integration of multiple images and extensive textual contexts remains a challenge due to the inherent limitation of the models' capacity to handle long input sequences efficiently. In this paper, we introduce SEEKER,… ▽ More The rapid progress in Multimodal Large Language Models (MLLMs) has significantly advanced their ability to process and understand complex visual and textual information. However, the integration of multiple images and extensive textual contexts remains a challenge due to the inherent limitation of the models' capacity to handle long input sequences efficiently. In this paper, we introduce SEEKER, a multimodal large language model designed to tackle this issue. SEEKER aims to optimize the compact encoding of long text by compressing the text sequence into the visual pixel space via images, enabling the model to handle long text within a fixed token-length budget efficiently. Our empirical experiments on six long-context multimodal tasks demonstrate that SEEKER can leverage fewer image tokens to convey the same amount of textual information compared with the OCR-based approach, and is more efficient in understanding long-form multimodal input and generating long-form textual output, outperforming all existing proprietary and open-source MLLMs by large margins. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Greater benefits of deep learning-based computer-aided detection systems for finding small signals in 3D volumetric medical images

Authors: Devi Klein, Srijita Karmakar, Aditya Jonnalagadda, Craig K. Abbey, Miguel P. Eckstein

Abstract: Purpose: Radiologists are tasked with visually scrutinizing large amounts of data produced by 3D volumetric imaging modalities. Small signals can go unnoticed during the 3d search because they are hard to detect in the visual periphery. Recent advances in machine learning and computer vision have led to effective computer-aided detection (CADe) support systems with the potential to mitigate percep… ▽ More Purpose: Radiologists are tasked with visually scrutinizing large amounts of data produced by 3D volumetric imaging modalities. Small signals can go unnoticed during the 3d search because they are hard to detect in the visual periphery. Recent advances in machine learning and computer vision have led to effective computer-aided detection (CADe) support systems with the potential to mitigate perceptual errors. Approach: Sixteen non-expert observers searched through digital breast tomosynthesis (DBT) phantoms and single cross-sectional slices of the DBT phantoms. The 3D/2D searches occurred with and without a convolutional neural network (CNN)-based CADe support system. The model provided observers with bounding boxes superimposed on the image stimuli while they looked for a small microcalcification signal and a large mass signal. Eye gaze positions were recorded and correlated with changes in the area under the ROC curve (AUC). Results: The CNN-CADe improved the 3D search for the small microcalcification signal (delta AUC = 0.098, p = 0.0002) and the 2D search for the large mass signal (delta AUC = 0.076, p = 0.002). The CNN-CADe benefit in 3D for the small signal was markedly greater than in 2D (delta delta AUC = 0.066, p = 0.035). Analysis of individual differences suggests that those who explored the least with eye movements benefited the most from the CNN-CADe (r = -0.528, p = 0.036). However, for the large signal, the 2D benefit was not significantly greater than the 3D benefit (delta delta AUC = 0.033, p = 0.133). Conclusion: The CNN-CADe brings unique performance benefits to the 3D (vs. 2D) search of small signals by reducing errors caused by the under-exploration of the volumetric data. △ Less

Submitted 30 April, 2024; originally announced May 2024.

Controlling radiative heat flow through cavity electrodynamics

Authors: Francesca Fassioli, Jerome Faist, Martin Eckstein, Daniele Fausti

Abstract: Cavity electrodynamics is emerging as a promising tool to control chemical processes and quantum material properties. In this work we develop a formalism to describe the cavity mediated energy exchange between a material and its electromagnetic environment. We show that coplanar cavities can significantly affect the heat load on the sample if the cavity resonance lies within the frequency region w… ▽ More Cavity electrodynamics is emerging as a promising tool to control chemical processes and quantum material properties. In this work we develop a formalism to describe the cavity mediated energy exchange between a material and its electromagnetic environment. We show that coplanar cavities can significantly affect the heat load on the sample if the cavity resonance lies within the frequency region where free-space radiative heat dominates, typically the mid-IR at ambient temperature, while spectral filtering is necessary for having an effect with lower frequency cavities. △ Less

Submitted 29 February, 2024; originally announced March 2024.

Theory of Quantum Light-Matter Interaction in Cavities: Extended Systems and the Long Wavelength Approximation

Authors: Mark Kamper Svendsen, Michael Ruggenthaler, Hannes Hübener, Christian Schäfer, Martin Eckstein, Angel Rubio, Simone Latini

Abstract: When light and matter interact strongly, the coupled system inherits properties from both constituents. It is consequently possible to alter the properties of either by engineering the other. This intriguing possibility has lead to the emergence of the cavity-materials-engineering paradigm which seeks to tailor material properties by engineering the fluctuations of a dark electromagnetic environme… ▽ More When light and matter interact strongly, the coupled system inherits properties from both constituents. It is consequently possible to alter the properties of either by engineering the other. This intriguing possibility has lead to the emergence of the cavity-materials-engineering paradigm which seeks to tailor material properties by engineering the fluctuations of a dark electromagnetic environment. The theoretical description of hybrid light-matter systems is complicated by the combined complexity of a realistic description of the extended electronic and quantum electromagnetic fields. Here we derive an effective, non-perturbative theory for low dimensional crystals embedded in a paradigmatic Fabry-Pérot resonator in the long-wavelength limit. The theory encodes the multi-mode nature of the electromagnetic field into an effective single-mode scheme and it naturally follows from requiring a negligible momentum transfer from the photonic system to the matter. Crucially, in the effective theory the single light mode is characterized by a finite effective mode volume even in the limit of bulk cavity-matter systems and can be directly determined by realistic cavity parameters. As a consequence, the coupling of the effective mode to matter remains finite for bulk materials. By leveraging on the realistic description of the cavity system we make our effective theory free from the double counting of the coupling of matter to the electromagnetic vacuum fluctuations of free space. Our results provide a substantial step towards the realistic description of interacting cavity-matter systems at the level of the fundamental Hamiltonian, by effectively including the electromagnetic environment and going beyond the perfect mirrors approximation. △ Less

Submitted 28 December, 2023; originally announced December 2023.

A new class of distances on complex projective spaces

Authors: Rafał Bistroń, Michał Eckstein, Shmuel Friedland, Tomasz Miller, Karol Życzkowski

Abstract: The complex projective space $\mathbb{P}(\mathbb{C}^n)$ can be interpreted as the space of all quantum pure states of size $n$. A distance on this space, interesting from the perspective of quantum physics, can be induced from a classical distance defined on the $n$-point probability simplex by the `earth mover problem'. We show that this construction leads to a quantity satisfying the triangle in… ▽ More The complex projective space $\mathbb{P}(\mathbb{C}^n)$ can be interpreted as the space of all quantum pure states of size $n$. A distance on this space, interesting from the perspective of quantum physics, can be induced from a classical distance defined on the $n$-point probability simplex by the `earth mover problem'. We show that this construction leads to a quantity satisfying the triangle inequality, which yields a true distance on complex projective space belonging to the family of quantum $2$-Wasserstein distances. △ Less

Submitted 5 December, 2023; originally announced December 2023.

Comments: 31 pages

Numerically exact simulation of photo-doped Mott insulators

Authors: Fabian Künzel, André Erpenbeck, Daniel Werner, Enrico Arrigoni, Emanuel Gull, Guy Cohen, Martin Eckstein

Abstract: A description of long-lived photo-doped states in Mott insulators is challenging, as it needs to address exponentially separated timescales. We demonstrate how properties of such states can be computed using numerically exact steady state techniques, in particular Quantum Monte Carlo, by using a time-local ansatz for the distribution function with separate Fermi functions for the electron and hole… ▽ More A description of long-lived photo-doped states in Mott insulators is challenging, as it needs to address exponentially separated timescales. We demonstrate how properties of such states can be computed using numerically exact steady state techniques, in particular Quantum Monte Carlo, by using a time-local ansatz for the distribution function with separate Fermi functions for the electron and hole quasiparticles. The simulations show that the Mott gap remains robust to large photo-do**, and the photo-doped state has hole and electron quasiparticles with strongly renormalized properties. △ Less

Submitted 23 November, 2023; originally announced November 2023.

Comments: 9 pages, 9 figures

Engineering Photon-mediated Long-Range Spin Interactions in Mott Insulators

Authors: Paul Fadler, Jiajun Li, Kai Phillip Schmidt, Martin Eckstein

Abstract: We investigate the potential to induce long-range spin interactions in a Mott insulator via the quantum electromagnetic field of a cavity. The coupling between light and spins is inherently non-linear, and occurs via multi-photon processes like Raman scattering and two-photon absorption/emission with electronically excited intermediate states. Based on this, two pathways are elucidated: (i) In the… ▽ More We investigate the potential to induce long-range spin interactions in a Mott insulator via the quantum electromagnetic field of a cavity. The coupling between light and spins is inherently non-linear, and occurs via multi-photon processes like Raman scattering and two-photon absorption/emission with electronically excited intermediate states. Based on this, two pathways are elucidated: (i) In the absence of external driving, long-range interactions are mediated by the exchange of at least two virtual cavity photons. We show that these vacuum-mediated interactions can surpass local Heisenberg interactions in mesoscopic setups such as sufficiently small split-ring resonators. (ii) In a laser-driven cavity, interactions can be tailored through a hybrid scheme involving both external laser photons and cavity photons. This offers a versatile pathway for Floquet engineering of long-range interactions in macroscopic systems. In general, the derivation of these interactions requires careful consideration: Notably, we demonstrate that a simple phenomenological approach, based on a spin-photon Hamiltonian that captures Raman and two-photon processes with effective matrix elements, can be used only if the cavity is resonantly driven. Outside of these narrow resonant regimes as well as for the undriven case, a fourth-order series expansion within the underlying electronic model is necessary, which we perform to obtain long-range four-spin interactions in the half-filled Hubbard model. △ Less

Submitted 2 November, 2023; originally announced November 2023.

Comments: 25 pages, 9 figures

Journal ref: Phys. Rev. B 109, 085149 (2024)

Photo-induced nonequilibrium states in Mott insulators

Authors: Yuta Murakami, Denis Golež, Martin Eckstein, Philipp Werner

Abstract: The study of nonequilibrium phenomena in interacting lattice systems can provide new perspectives on correlation effects, and information on metastable states of matter. Mott insulators are a promising class of systems for nonequilibrium studies, since they exhibit exotic phenomena and complex phase diagrams upon do**, and because a large Mott gap provides protection against fast thermalization… ▽ More The study of nonequilibrium phenomena in interacting lattice systems can provide new perspectives on correlation effects, and information on metastable states of matter. Mott insulators are a promising class of systems for nonequilibrium studies, since they exhibit exotic phenomena and complex phase diagrams upon do**, and because a large Mott gap provides protection against fast thermalization and heating after photo-excitations. We can thus expect the emergence of interesting transient states and photo-induced phases in Mott systems. This review presents the current understanding of the mechanisms which control the time evolution of photo-doped charge carriers and the properties of photo-induced metastable states. We focus on recent theoretical progress, identify the relevant underlying concepts, and link them to experimental observations. The review starts with a general discussion of field-induced nonequilibrium setups and an overview of key experiments which revealed characteristic properties of photo-excited Mott states, proceeds with a compact overview of the theoretical tools which have been developed to investigate these strongly correlated nonequilibrium states, and then analyzes Mott insulators driven out of equilibrium by static electric fields, periodic fields, and short laser pulses. We also discuss the appearance of nonthermal electronic orders in photo-excited Mott systems, including nonthermal spin and orbital orders, $η$ pairing states, and novel types of excitonic orders. △ Less

Submitted 8 October, 2023; originally announced October 2023.

Comments: Review article

Domain generalization across tumor types, laboratories, and species -- insights from the 2022 edition of the Mitosis Domain Generalization Challenge

Authors: Marc Aubreville, Nikolas Stathonikos, Taryn A. Donovan, Robert Klopfleisch, Jonathan Ganz, Jonas Ammeling, Frauke Wilm, Mitko Veta, Samir Jabari, Markus Eckstein, Jonas Annuscheit, Christian Krumnow, Engin Bozaba, Sercan Cayir, Hongyan Gu, Xiang 'Anthony' Chen, Mostafa Jahanifar, Adam Shephard, Satoshi Kondo, Satoshi Kasai, Sujatha Kotte, VG Saipradeep, Maxime W. Lafarge, Viktor H. Koelzer, Ziyue Wang , et al. (5 additional authors not shown)

Abstract: Recognition of mitotic figures in histologic tumor specimens is highly relevant to patient outcome assessment. This task is challenging for algorithms and human experts alike, with deterioration of algorithmic performance under shifts in image representations. Considerable covariate shifts occur when assessment is performed on different tumor types, images are acquired using different digitization… ▽ More Recognition of mitotic figures in histologic tumor specimens is highly relevant to patient outcome assessment. This task is challenging for algorithms and human experts alike, with deterioration of algorithmic performance under shifts in image representations. Considerable covariate shifts occur when assessment is performed on different tumor types, images are acquired using different digitization devices, or specimens are produced in different laboratories. This observation motivated the inception of the 2022 challenge on MItosis Domain Generalization (MIDOG 2022). The challenge provided annotated histologic tumor images from six different domains and evaluated the algorithmic approaches for mitotic figure detection provided by nine challenge participants on ten independent domains. Ground truth for mitotic figure detection was established in two ways: a three-expert consensus and an independent, immunohistochemistry-assisted set of labels. This work represents an overview of the challenge tasks, the algorithmic strategies employed by the participants, and potential factors contributing to their success. With an $F_1$ score of 0.764 for the top-performing team, we summarize that domain generalization across various tumor domains is possible with today's deep learning-based recognition pipelines. However, we also found that domain characteristics not present in the training set (feline as new species, spindle cell shape as new morphology and a new scanner) led to small but significant decreases in performance. When assessed against the immunohistochemistry-assisted reference standard, all methods resulted in reduced recall scores, but with only minor changes in the order of participants in the ranking. △ Less

Submitted 31 January, 2024; v1 submitted 27 September, 2023; originally announced September 2023.

Journal ref: Medical Image Analysis Volume 94, May 2024, 103155

Nonequilibrium DMFT approach to time-resolved Raman spectroscopy

Authors: Philipp Werner, Martin Eckstein, Naoto Tsuji

Abstract: Raman spectroscopy uses light scattering to extract information on low-energy excitations of solids. The Raman process is described by diagrams which are fourth order in the light-matter interaction, and in particular the resonant contribution, which involves four different space-time arguments, is difficult to evaluate. If one instead simulates explicitly the incoming (classical) light pulse, the… ▽ More Raman spectroscopy uses light scattering to extract information on low-energy excitations of solids. The Raman process is described by diagrams which are fourth order in the light-matter interaction, and in particular the resonant contribution, which involves four different space-time arguments, is difficult to evaluate. If one instead simulates explicitly the incoming (classical) light pulse, the Raman signal is given by the outgoing photon flux and can be determined from a two-point correlation function. Such a formalism can be used to compute the time-resolved Raman spectrum of non-equilibrium systems, as well as nonlinear signals which are higher order in the incoming field, such as hyper Raman scattering. Here we explain how to implement this time-dependent formalism within the dynamical mean field theory framework. The method is illustrated with applications to the Holstein-Hubbard model in the strong electron-phonon coupling regime. We demonstrate hyper Raman scattering in measurements with strong probe fields and frequency mixing signals in the presence of a pump field, and simulate the evolution of Stokes and anti-Stokes features after photo-excitations of metallic and Mott insulating systems. △ Less

Submitted 6 September, 2023; originally announced September 2023.

Journal ref: Phys. Rev. B 108, 245157 (2023)

Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation

Authors: Wanrong Zhu, Xinyi Wang, Yujie Lu, Tsu-Jui Fu, Xin Eric Wang, Miguel Eckstein, William Yang Wang

Abstract: The field of text-to-image (T2I) generation has garnered significant attention both within the research community and among everyday users. Despite the advancements of T2I models, a common issue encountered by users is the need for repetitive editing of input prompts in order to receive a satisfactory image, which is time-consuming and labor-intensive. Given the demonstrated text generation power… ▽ More The field of text-to-image (T2I) generation has garnered significant attention both within the research community and among everyday users. Despite the advancements of T2I models, a common issue encountered by users is the need for repetitive editing of input prompts in order to receive a satisfactory image, which is time-consuming and labor-intensive. Given the demonstrated text generation power of large-scale language models, such as GPT-k, we investigate the potential of utilizing such models to improve the prompt editing process for T2I generation. We conduct a series of experiments to compare the common edits made by humans and GPT-k, evaluate the performance of GPT-k in prompting T2I, and examine factors that may influence this process. We found that GPT-k models focus more on inserting modifiers while humans tend to replace words and phrases, which includes changes to the subject matter. Experimental results show that GPT-k are more effective in adjusting modifiers rather than predicting spontaneous changes in the primary subject matters. Adopting the edit suggested by GPT-k models may reduce the percentage of remaining edits by 20-30%. △ Less

Submitted 28 October, 2023; v1 submitted 18 May, 2023; originally announced May 2023.

Comments: EMNLP 2023

Photo-induced charge-transfer renormalization in NiO

Authors: Tobias Lojewski, Denis Golez, Katharina Ollefs, Loïc Le Guyader, Lea Kämmerer, Nico Rothenbach, Robin Y. Engel, Piter S. Miedema, Martin Beye, Gheorghe S. Chiuzbăian, Robert Carley, Rafael Gort, Benjamin E. Van Kuiken, Giuseppe Mercurio, Justina Schlappa, Alexander Yaroslavtsev, Andreas Scherz, Florian Döring, Christian David, Heiko Wende, Uwe Bovensiepen, Martin Eckstein, Philipp Werner, Andrea Eschenlohr

Abstract: Photo-doped states in strongly correlated charge transfer insulators are characterized by $d$-$d$ and $d$-$p$ interactions and the resulting intertwined dynamics of charge excitations and local multiplets. Here we use femtosecond x-ray absorption spectroscopy in combination with dynamical mean-field theory to disentangle these contributions in NiO. Upon resonant optical excitation across the charg… ▽ More Photo-doped states in strongly correlated charge transfer insulators are characterized by $d$-$d$ and $d$-$p$ interactions and the resulting intertwined dynamics of charge excitations and local multiplets. Here we use femtosecond x-ray absorption spectroscopy in combination with dynamical mean-field theory to disentangle these contributions in NiO. Upon resonant optical excitation across the charge transfer gap, the Ni $L_3$ and O $K$ absorption edges red-shift for $>10$ ps, associated with photo-induced changes in the screening environment. An additional signature below the Ni $L_3$ edge is identified for $<1$ ps, reflecting a transient nonthermal population of local many-body multiplets. We employ a nonthermal generalization of the multiplet ligand field theory to show that the feature originates from $d$-$d$ transitions. Overall, the photo-doped state differs significantly from a chemically doped state. Our results demonstrate the ability to reveal excitation pathways in correlated materials by x-ray spectroscopies, which is relevant for ultrafast materials design. △ Less

Submitted 24 May, 2024; v1 submitted 17 May, 2023; originally announced May 2023.

Light-induced insulator-metal transition in Sr$_2$IrO$_4$ reveals the nature of the insulating ground state

Authors: Dongsung Choi, Changming Yue, Doron Azoury, Zachary Porter, Jiyu Chen, Francesco Petocchi, Edoardo Baldini, Baiqing Lv, Masataka Mogi, Yifan Su, Stephen D. Wilson, Martin Eckstein, Philipp Werner, Nuh Gedik

Abstract: Sr$_2$IrO$_4$ has attracted a lot of attention due to its structural and electronic similarities to La$_2$CuO$_4$ which is the parent compound of high-T$_c$ superconducting cuprates. It was proposed to be a strong spin-orbit coupled J$_{eff}$ = 1/2 Mott insulator, but the Mott nature of its insulating ground state and the origin of the gap have not been conclusively established. Here, we use ultra… ▽ More Sr$_2$IrO$_4$ has attracted a lot of attention due to its structural and electronic similarities to La$_2$CuO$_4$ which is the parent compound of high-T$_c$ superconducting cuprates. It was proposed to be a strong spin-orbit coupled J$_{eff}$ = 1/2 Mott insulator, but the Mott nature of its insulating ground state and the origin of the gap have not been conclusively established. Here, we use ultrafast laser pulses to realize an insulator-metal transition in Sr$_2$IrO$_4$ and probe the resulting dynamics using time- and angle-resolved photoemission spectroscopy. We observe a closing of the gap and the formation of weakly-renormalized electronic bands in the gap region. Comparing these observations to the expected temperature and do** evolution of Mott gaps and Hubbard bands provides clear evidence that the insulating state does not originate from Mott correlations. We instead propose a correlated band insulator picture, where antiferromagnetic correlations play a key role in the opening of the gap. More broadly, our results demonstrate that energy-momentum resolved nonequilibrium dynamics can be used to clarify the nature of equilibrium states in correlated materials. △ Less

Submitted 12 May, 2023; originally announced May 2023.

Photo-induced charge dynamics in 1$T$-TaS$_2$

Authors: Francesco Petocchi, Jiyu Chen, Jiajun Li, Martin Eckstein, Philipp Werner

Abstract: Recent theoretical studies showed that the electronic structure of 1$T$-TaS$_2$ in the low-temperature commensurate charge density wave phase exhibits a nontrivial interplay between band-insulating and Mott insulating behavior. This has important implications for the interpretation of photo-do** experiments. Here we use nonequilibrium dynamical mean-field theory simulations of a realistic multi-… ▽ More Recent theoretical studies showed that the electronic structure of 1$T$-TaS$_2$ in the low-temperature commensurate charge density wave phase exhibits a nontrivial interplay between band-insulating and Mott insulating behavior. This has important implications for the interpretation of photo-do** experiments. Here we use nonequilibrium dynamical mean-field theory simulations of a realistic multi-layer structure to clarify the charge carrier dynamics induced by a laser pulse. The solution is propagated up to the picosecond timescale by employing a memory-truncation scheme. While long-lived doublons and holons only exist in the surface state of a specific structure, the disturbance of bonding states in the bilayers which make up the bulk of the system explain the almost instantaneous appearance of in-gap states. Our simulations consistently explain the coexistence of a doublon feature with a prominent ``background" signal in previous time-resolved photoemission experiments, and they suggest strategies for the selective population of the ingap and doublon states by exploiting the sensitivity to the pump polarization and pump frequency. △ Less

Submitted 19 December, 2022; originally announced December 2022.

Comments: 12 pages, 9 figures

Visualize Before You Write: Imagination-Guided Open-Ended Text Generation

Authors: Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, William Yang Wang

Abstract: Recent advances in text-to-image synthesis make it possible to visualize machine imaginations for a given context. On the other hand, when generating text, human writers are gifted at creative visualization, which enhances their writings by forming imaginations as blueprints before putting down the stories in words. Inspired by such a cognitive process, we ask the natural question of whether we ca… ▽ More Recent advances in text-to-image synthesis make it possible to visualize machine imaginations for a given context. On the other hand, when generating text, human writers are gifted at creative visualization, which enhances their writings by forming imaginations as blueprints before putting down the stories in words. Inspired by such a cognitive process, we ask the natural question of whether we can endow machines with the same ability to utilize visual information and construct a general picture of the context to guide text generation. In this work, we propose iNLG that uses machine-generated images to guide language models in open-ended text generation. The experiments and analyses demonstrate the effectiveness of iNLG on open-ended text generation tasks, including text completion, story generation, and concept-to-text generation in both few-shot and full-data scenarios. Both automatic metrics and human evaluations verify that the text snippets generated by our iNLG are coherent and informative while displaying minor degeneration. △ Less

Submitted 14 February, 2023; v1 submitted 7 October, 2022; originally announced October 2022.

Comments: EACL 2023

Cavity-mediated thermal control of metal-to-insulator transition in 1T-TaS$_{2}$

Authors: Giacomo Jarc, Shahla Yasmin Mathengattil, Angela Montanaro, Francesca Giusti, Enrico Maria Rigoni, Rudi Sergo, Francesca Fassioli, Stephan Winnerl, Simone Dal Zilio, Dragan Mihailovic, Peter Prelovšek, Martin Eckstein, Daniele Fausti

Abstract: Placing quantum materials into optical cavities provides a unique platform for controlling quantum cooperative properties of matter, via both weak and strong light-matter coupling. Here we report the experimental evidence of reversible cavity control of a metal-to-insulator phase transition in a correlated solid-state material. We embed the charge density wave material 1T-TaS$_{2}$ into cryogenic… ▽ More Placing quantum materials into optical cavities provides a unique platform for controlling quantum cooperative properties of matter, via both weak and strong light-matter coupling. Here we report the experimental evidence of reversible cavity control of a metal-to-insulator phase transition in a correlated solid-state material. We embed the charge density wave material 1T-TaS$_{2}$ into cryogenic tunable terahertz cavities and show that a switch between conductive and insulating behaviors, associated with a large change in the sample temperature, is obtained by mechanically tuning the distance between the cavity mirrors and their alignment. The large thermal modification observed is indicative of a Purcell-like scenario in which the spectral profile of the cavity modifies the energy exchange between the material and the external electromagnetic field. Our findings provide opportunities for controlling the thermodynamics and macroscopic transport properties of quantum materials by engineering their electromagnetic environment. △ Less

Submitted 20 October, 2023; v1 submitted 5 October, 2022; originally announced October 2022.

Comments: 52 pages, 36 figures (including Supplementary Information)

Journal ref: Nature 622, 487-492 (2023)

Time-resolved photoemission and RIXS study of a site-selective Mott insulator

Authors: Philipp Werner, Francesco Petocchi, Martin Eckstein

Abstract: Inspired by the physics of rare earth nickelates, we study the photoemission (PES) and resonant inelastic X-ray scattering (RIXS) spectra of a correlated electron system with two types of insulating sublattices. Sublattice A is characterized by a hybridization gap and a low-spin state, while sublattice B features a Mott gap and a local magnetic moment. We show how the coupling of these two qualita… ▽ More Inspired by the physics of rare earth nickelates, we study the photoemission (PES) and resonant inelastic X-ray scattering (RIXS) spectra of a correlated electron system with two types of insulating sublattices. Sublattice A is characterized by a hybridization gap and a low-spin state, while sublattice B features a Mott gap and a local magnetic moment. We show how the coupling of these two qualitatively different insulating states affects the dynamics of photo-induced charge carriers and how the nonequilibrium states manifest themselves in the PES and RIXS signals. In particular, we find that charge carriers created on the B sublattice migrate to the A sublattice, where they contribute to the creation of in-gap states in the PES signal, and to characteristic peaks in the nonequilibrium RIXS spectrum. While the contributions from the two sublattices cannot be easily distinguished in the local photoemission spectrum, the weights of the RIXS signals in the two-dimensional $ω_\text{in}$-$ω_\text{out}$ space provide information on the local state evolution on both sublattices. △ Less

Submitted 15 September, 2022; originally announced September 2022.

Dynamical mean-field study of a photon-mediated ferroelectric phase transition

Authors: Katharina Lenk, Jiajun Li, Philipp Werner, Martin Eckstein

Abstract: The interplay of light and matter gives rise to intriguing cooperative effects in quantum many-body systems. This is even true in thermal equilibrium, where the electromagnetic field can hybridize with collective modes of matter, and virtual photons can induce interactions in the solid. Here, we show how these light-mediated interactions can be treated using the dynamical mean-field theory formali… ▽ More The interplay of light and matter gives rise to intriguing cooperative effects in quantum many-body systems. This is even true in thermal equilibrium, where the electromagnetic field can hybridize with collective modes of matter, and virtual photons can induce interactions in the solid. Here, we show how these light-mediated interactions can be treated using the dynamical mean-field theory formalism. We consider a minimal model of a two-dimensional material that couples to a surface plasmon polariton mode of a metal-dielectric interface. Within the mean-field approximation, the system exhibits a ferroelectric phase transition that is unaffected by the light-matter coupling. Bosonic dynamical mean-field theory provides a more accurate description and reveals that the photon-mediated interactions enhance the ferroelectric order and stabilize the ferroelectric phase. △ Less

Submitted 16 December, 2022; v1 submitted 9 September, 2022; originally announced September 2022.

Journal ref: Phys. Rev. B 106, 245124 (2022)

Stochastic semiclassical theory for non-equilibrium electron-phonon coupled systems

Authors: Antonio Picano, Francesco Grandi, Philipp Werner, Martin Eckstein

Abstract: We discuss a semiclassical approach to solve the quantum impurity model within non-equilibrium dynamical mean-field theory for electron-lattice models. The effect of electronic fluctuations on the phonon is kept beyond Ehrenfest dynamics, leading to a stochastic phonon evolution with dam** and noise terms that are self-consistently determined by the electronic correlation functions in the fluctu… ▽ More We discuss a semiclassical approach to solve the quantum impurity model within non-equilibrium dynamical mean-field theory for electron-lattice models. The effect of electronic fluctuations on the phonon is kept beyond Ehrenfest dynamics, leading to a stochastic phonon evolution with dam** and noise terms that are self-consistently determined by the electronic correlation functions in the fluctuating phonon field. Together with a solution of the electronic model based on a non-perturbative quantum Boltzmann equation, the approach can be used to address the coupled dynamics of the electrons and the lattice during photo-induced phase transitions. Results for the Anderson-Holstein model are benchmarked against numerically exact quantum Monte Carlo data. We find good agreement for the phonon distribution function at temperatures comparable to the charge ordering temperature. The general formulation can be extended to models with electron-electron interactions or multi-orbital systems. △ Less

Submitted 10 July, 2023; v1 submitted 1 September, 2022; originally announced September 2022.

Comments: 17 pages, 4 figures

Journal ref: Phys. Rev. B 108, 035115 (2023)

Two instances of random access code in the quantum regime

Authors: Nitica Sakharwade, Michał Studziński, Michał Eckstein, Paweł Horodecki

Abstract: We consider two classes of quantum generalisations of Random Access Code (RAC) and study lower bounds for probabilities of success for such tasks. It provides a useful framework for the study of certain information processing tasks with constrained resources. The first class is based on a random access code with quantum inputs and output known as No-Signalling Quantum RAC (NS-QRAC) [A. Grudka et a… ▽ More We consider two classes of quantum generalisations of Random Access Code (RAC) and study lower bounds for probabilities of success for such tasks. It provides a useful framework for the study of certain information processing tasks with constrained resources. The first class is based on a random access code with quantum inputs and output known as No-Signalling Quantum RAC (NS-QRAC) [A. Grudka et al. Phys. Rev. A 92, 052312 (2015)], where unbounded entanglement and constrained classical communication are allowed, which can be seen as quantum teleportation with constrained classical communication, for which we provide a quantum lower bound. We consider two modifications to the NS-QRAC scenario, first where unbounded entanglement and constrained quantum communication is allowed and, second where bounded entanglement and unconstrained classical communication are allowed, where we find a monogamy relation for the transmission fidelities, which -- in contrast to the usual communication schemes -- involves multiple senders and a single receiver. We provide lower bounds for these scenarios. The second class is based on a random access code with a quantum channel and shared entanglement [A. Tavakoli et al. PRX Quantum 2 (4) 040357 (2021)]. We study the set of tasks where two inputs made of two digits of $d$-base are encoded over a qudit and a maximally entangled state, which can be seen as quantum dense coding with constrained quantum communication, for which we provide quantum lower bounds for $d=2,3,4$. The encoding employed utilises Gray codes. △ Less

Submitted 14 June, 2023; v1 submitted 30 August, 2022; originally announced August 2022.

Comments: Version after Referee report corrections, 16 Pages, 11 Figures. Paper is published New Journal of Physics (2023). Version 3 and Version 2 are identical other than corrected reference number

Journal ref: New J. Phys. 25 053038 (2023)

Sub-cycle multidimensional spectroscopy of strongly correlated materials

Authors: V. Valmispild, E. Gorelov, M. Eckstein, A. Lichtenstein, H. Aoki, M. Katsnelson, M. Ivanov, O. Smirnova

Abstract: Strongly correlated solids are extremely complex and fascinating quantum systems, where new states continue to emerge, especially when interaction with light triggers interplay between them. In this interplay, sub-laser-cycle electron response is particularly attractive as a tool for ultrafast manipulation of matter at PHz scale. Here we introduce a new type of non-linear multidimensional spectros… ▽ More Strongly correlated solids are extremely complex and fascinating quantum systems, where new states continue to emerge, especially when interaction with light triggers interplay between them. In this interplay, sub-laser-cycle electron response is particularly attractive as a tool for ultrafast manipulation of matter at PHz scale. Here we introduce a new type of non-linear multidimensional spectroscopy, which allows us to unravel the sub-cycle dynamics of strongly correlated systems interacting with few-cycle infrared pulses and the complex interplay between different correlated states evolving on the sub-femtosecond time-scale. We demonstrate that single particle sub-cycle electronic response is extremely sensitive to correlated many-body dynamics and provides direct access to many body response functions. For the two-dimensional Hubbard model under the influence of ultra-short, intense electric field transients, we demonstrate that our approach can resolve pathways of charge and energy flow between localized and delocalized many-body states on the sub-cycle time scale and follow the creation of a highly correlated state surviving after the end of the laser pulse. Our findings open a way towards a regime of imaging and manipulating strongly correlated materials at optical rates, beyond the multi-cycle approach employed in Floquet engineering, with the sub-cycle response being a key tool for accessing many body phenomena. △ Less

Submitted 10 March, 2023; v1 submitted 9 August, 2022; originally announced August 2022.

Comments: 10 pages, 4, figures, Methods (5 pages), Supplementary information (4 figures, 4 pages)

Control of Yu-Shiba-Rusinov States through a Bosonic Mode

Authors: Helene Müller, Martin Eckstein, Silvia Viola Kusminskiy

Abstract: We investigate the impact of a bosonic degree of freedom on Yu-Shiba-Rusinov (YSR) states emerging from a magnetic impurity in a conventional superconductor. Starting from the Anderson impurity model, we predict that an additional p-wave conduction band channel opens up if a bosonic mode is coupled to the tunnelling between impurity and host, which implies an additional pair of odd-parity YSR stat… ▽ More We investigate the impact of a bosonic degree of freedom on Yu-Shiba-Rusinov (YSR) states emerging from a magnetic impurity in a conventional superconductor. Starting from the Anderson impurity model, we predict that an additional p-wave conduction band channel opens up if a bosonic mode is coupled to the tunnelling between impurity and host, which implies an additional pair of odd-parity YSR states. The bosonic mode can be a vibrational mode or the electromagnetic field in a cavity. The exchange couplings in the two channels depend sensitively on the state of the bosonic mode (ground state, few quanta or classically driven Floquet state), which opens possibilities for phononics or photonics control of such systems, with a rich variety of ground and excited states. △ Less

Submitted 19 March, 2023; v1 submitted 28 July, 2022; originally announced July 2022.

Journal ref: Phys. Rev. Lett. 130, 106905 (2023)

Neuro-Symbolic Procedural Planning with Commonsense Prompting

Authors: Yujie Lu, Weixi Feng, Wanrong Zhu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, William Yang Wang

Abstract: Procedural planning aims to implement complex high-level goals by decomposition into sequential simpler low-level steps. Although procedural planning is a basic skill set for humans in daily life, it remains a challenge for large language models (LLMs) that lack a deep understanding of the cause-effect relations in procedures. Previous methods require manual exemplars to acquire procedural plannin… ▽ More Procedural planning aims to implement complex high-level goals by decomposition into sequential simpler low-level steps. Although procedural planning is a basic skill set for humans in daily life, it remains a challenge for large language models (LLMs) that lack a deep understanding of the cause-effect relations in procedures. Previous methods require manual exemplars to acquire procedural planning knowledge from LLMs in the zero-shot setting. However, such elicited pre-trained knowledge in LLMs induces spurious correlations between goals and steps, which impair the model generalization to unseen tasks. In contrast, this paper proposes a neuro-symbolic procedural PLANner (PLAN) that elicits procedural planning knowledge from the LLMs with commonsense-infused prompting. To mitigate spurious goal-step correlations, we use symbolic program executors on the latent procedural representations to formalize prompts from commonsense knowledge bases as a causal intervention toward the Structural Causal Model. Both automatic and human evaluations on WikiHow and RobotHow show the superiority of PLAN on procedural planning without further training or manual exemplars. △ Less

Submitted 16 February, 2023; v1 submitted 6 June, 2022; originally announced June 2022.

Comments: ICLR 2023 Spotlight

Collective theory for an interacting solid in a single-mode cavity

Authors: Katharina Lenk, Jiajun Li, Philipp Werner, Martin Eckstein

Abstract: We investigate the control of interacting matter through strong coupling to a single electromagnetic mode, such as the photon mode in a Fabry-Perot or split-ring cavity. For this purpose, we analyze the exact effective theory for the collective light-matter hybrid modes of a generic system of $N$ transition dipoles within an interacting solid. The approach allows to predict properties of the coupl… ▽ More We investigate the control of interacting matter through strong coupling to a single electromagnetic mode, such as the photon mode in a Fabry-Perot or split-ring cavity. For this purpose, we analyze the exact effective theory for the collective light-matter hybrid modes of a generic system of $N$ transition dipoles within an interacting solid. The approach allows to predict properties of the coupled light-matter system from the nonlinear response functions of the uncoupled matter ``outside the cavity''. The limit of large $N$ corresponds to a conventional macroscopic description based on the polarizability of matter. In this limit, the cavity does not affect the static ferroelectric response. Corrections, which are needed to understand finite size systems and to obtain the nonlinear light-matter response, can be obtained from the non-linear susceptibilities of the matter outside the cavity. The theory is benchmarked for the Dicke model, and for a quantum Ising model which serves as a minimal mean-field model for a quantum paraelectric material like SrTiO3. △ Less

Submitted 11 May, 2022; originally announced May 2022.

Comments: 20 pages, 6 figures

Pseudoparticle vertex solver for quantum impurity models

Authors: Aaram J. Kim, Jiajun Li, Martin Eckstein, Philipp Werner

Abstract: We present a quantum impurity solver based on a pseudo-particle framework, which combines diagrammatic resummations for a three-point vertex with diagrammatic Monte Carlo sampling of a four-point vertex. This recently proposed approach [A. J. Kim et al., arXiv:2112.15549] is generalized here to fermionic impurity problems and we discuss the technical details of the implementation, including the ti… ▽ More We present a quantum impurity solver based on a pseudo-particle framework, which combines diagrammatic resummations for a three-point vertex with diagrammatic Monte Carlo sampling of a four-point vertex. This recently proposed approach [A. J. Kim et al., ar** approach, the Monte Carlo updates, and the routines for checking the two-particle irreducibility of the four-point vertex. We also explain how the vertex information can be efficiently stored using a Dubiner basis representation. The convergence properties of the algorithm are demonstrated with applications to exactly solvable impurity models and dynamical mean field theory simulations of the single-orbital Hubbard model. It is furthermore shown that the algorithm can handle a two-orbital problem with off-diagonal hybridizations, which would cause a severe sign problem in standard hybridization-expansion Monte Carlo simulations. Since the vertex-based algorithm successfully handles sign oscillating integrals in equilibrium and samples only connected diagrams, it may be a promising approach for real-time simulations. △ Less

Submitted 3 September, 2022; v1 submitted 28 April, 2022; originally announced April 2022.

Comments: 16 pages, 21 figures

Journal ref: Phys. Rev. B 106, 085124 (2022)

Imagination-Augmented Natural Language Understanding

Authors: Yujie Lu, Wanrong Zhu, Xin Eric Wang, Miguel Eckstein, William Yang Wang

Abstract: Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations. Such abilities enable us to construct new abstract concepts or concrete objects, and are essential in involving practical knowledge to solve problems in low-resource scenarios. However, most existing methods for Natural Language Understan… ▽ More Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations. Such abilities enable us to construct new abstract concepts or concrete objects, and are essential in involving practical knowledge to solve problems in low-resource scenarios. However, most existing methods for Natural Language Understanding (NLU) are mainly focused on textual signals. They do not simulate human visual imagination ability, which hinders models from inferring and learning efficiently from limited data samples. Therefore, we introduce an Imagination-Augmented Cross-modal Encoder (iACE) to solve natural language understanding tasks from a novel learning perspective -- imagination-augmented cross-modal understanding. iACE enables visual imagination with external knowledge transferred from the powerful generative and pre-trained vision-and-language models. Extensive experiments on GLUE and SWAG show that iACE achieves consistent improvement over visually-supervised pre-trained models. More importantly, results in extreme and normal few-shot settings validate the effectiveness of iACE in low-resource natural language understanding circumstances. △ Less

Submitted 3 May, 2022; v1 submitted 18 April, 2022; originally announced April 2022.

Comments: NAACL 2022 Main Conference

Monotonicity of the quantum 2-Wasserstein distance

Authors: Rafał Bistroń, Michał Eckstein, Karol Życzkowski

Abstract: We study a quantum analogue of the 2-Wasserstein distance as a measure of proximity on the set $Ω_N$ of density matrices of dimension $N$. We show that such (semi-)distances do not induce Riemannian metrics on the tangent bundle of $Ω_N$ and are typically not unitary invariant. Nevertheless, we prove that for $N=2$ dimensional Hilbert space the quantum 2-Wasserstein distance (unique up to rescalin… ▽ More We study a quantum analogue of the 2-Wasserstein distance as a measure of proximity on the set $Ω_N$ of density matrices of dimension $N$. We show that such (semi-)distances do not induce Riemannian metrics on the tangent bundle of $Ω_N$ and are typically not unitary invariant. Nevertheless, we prove that for $N=2$ dimensional Hilbert space the quantum 2-Wasserstein distance (unique up to rescaling) is monotonous with respect to any single-qubit quantum operation and the solution of the quantum transport problem is essentially unique. Furthermore, for any $N \geq 3$ and the quantum cost matrix proportional to a projector we demonstrate the monotonicity under arbitrary mixed unitary channels. Finally, we provide numerical evidence which allows us to conjecture that the unitary invariant quantum 2-Wasserstein semi-distance is monotonous with respect to all CPTP maps in any dimension $N$. △ Less

Submitted 9 September, 2022; v1 submitted 15 April, 2022; originally announced April 2022.

Comments: 30 pages, 5 figures

Local interpretation of time-resolved X-ray absorption in Mott insulators: Insights from nonequilibrium dynamical mean-field theory

Authors: Philipp Werner, Denis Golez, Martin Eckstein

Abstract: We present a formalism based on nonequilibrium dynamical mean field theory (DMFT) which allows to compute the time-resolved X-ray absorption spectrum (XAS) of photo-excited solids. By applying this formalism to the photo-doped half-filled and quarter-filled two-orbital Hubbard models in the Mott insulating regime we clarify how the time-resolved XAS signal reflects the nonequilibrium population of… ▽ More We present a formalism based on nonequilibrium dynamical mean field theory (DMFT) which allows to compute the time-resolved X-ray absorption spectrum (XAS) of photo-excited solids. By applying this formalism to the photo-doped half-filled and quarter-filled two-orbital Hubbard models in the Mott insulating regime we clarify how the time-resolved XAS signal reflects the nonequilibrium population of different local states. Apart from the missing broadening associated with continuum excitations, the atomic XAS spectrum computed with the nonthermal state populations provides a good approximation to the full nonequilibrium DMFT result. This suggest a route to combine the accurate DMFT description of nonequilibrum states of solids with cluster calculations of the XAS signal. △ Less

Submitted 19 April, 2022; v1 submitted 14 April, 2022; originally announced April 2022.

Report number: Phys. Rev. B 106, 165106 (2022)

Causality and time order -- relativistic and probabilistic aspects

Authors: Michał Eckstein, Michael Heller

Abstract: We investigate temporal and causal threads in the fabric of contemporary physical theories with an emphasis on empirical and operationalistic aspects. Building on the axiomatization of general relativity proposed by J. Ehlers, F. Pirani and A. Schild and the global space-time structure elaborated by R. Penrose, S.W. Hawking, B. Carter and others, we argue that the current way of doing relativistic… ▽ More We investigate temporal and causal threads in the fabric of contemporary physical theories with an emphasis on empirical and operationalistic aspects. Building on the axiomatization of general relativity proposed by J. Ehlers, F. Pirani and A. Schild and the global space-time structure elaborated by R. Penrose, S.W. Hawking, B. Carter and others, we argue that the current way of doing relativistic physics presupposes treating time and causality as primitive concepts, neither of them being `more primitive' than the other. The decision regarding which concepts to assume as primitive and which statements to regard as axioms depends on the choice of the angle at which we contemplate the whole. This standard approach is based on the presupposition that the concept of a point-like particle is a viable approximation. However, this assumption is not supported by a realistic approach to doing physics and, in particular, by quantum theory. We remove this assumption by analysing the recent works by M. Eckstein and T. Miller. They consider the space $P(M)$ of probability measures on space-time $M$ such that, for an element $μ\in P(M)$, the number $μ(K)$ specifies the probability of the occurrence of some event associated with the space-time region $K$ and the measure $μ$. In this way, $M$ is not to be regarded as a collection of space-time events, but rather as a support for corresponding probability measures. As shown by Eckstein and Miller, the space $P(M)$ inherits the causal order from the underlying space-time and facilitates a rigorous notion of a `causal evolution of probability measures'. We look at the deductive chains creating temporal and causal structures analysed in these works, in order to highlight their operational (or quasi-operational) aspect. This is impossible without taking into account the relative frequencies and correlations observed in relevant experiments. △ Less

Submitted 15 February, 2022; originally announced February 2022.

Comments: 22 pages, 2 figures. Comments are welcome!

MSC Class: 83-03; 81P05; 60A10

A deep Q-learning method for optimizing visual search strategies in backgrounds of dynamic noise

Authors: Weimin Zhou, Miguel P. Eckstein

Abstract: Humans process visual information with varying resolution (foveated visual system) and explore images by orienting through eye movements the high-resolution fovea to points of interest. The Bayesian ideal searcher (IS) that employs complete knowledge of task-relevant information optimizes eye movement strategy and achieves the optimal search performance. The IS can be employed as an important tool… ▽ More Humans process visual information with varying resolution (foveated visual system) and explore images by orienting through eye movements the high-resolution fovea to points of interest. The Bayesian ideal searcher (IS) that employs complete knowledge of task-relevant information optimizes eye movement strategy and achieves the optimal search performance. The IS can be employed as an important tool to evaluate the optimality of human eye movements, and potentially provide guidance to improve human observer visual search strategies. Najemnik and Geisler (2005) derived an IS for backgrounds of spatial 1/f noise. The corresponding template responses follow Gaussian distributions and the optimal search strategy can be analytically determined. However, the computation of the IS can be intractable when considering more realistic and complex backgrounds such as medical images. Modern reinforcement learning methods, successfully applied to obtain optimal policy for a variety of tasks, do not require complete knowledge of the background generating functions and can be potentially applied to anatomical backgrounds. An important first step is to validate the optimality of the reinforcement learning method. In this study, we investigate the ability of a reinforcement learning method that employs Q-network to approximate the IS. We demonstrate that the search strategy corresponding to the Q-network is consistent with the IS search strategy. The findings show the potential of the reinforcement learning with Q-network approach to estimate optimal eye movement planning with real anatomical backgrounds. △ Less

Submitted 28 January, 2022; originally announced January 2022.

Comments: SPIE Medical Imaging 2022

Dynamical phase transitions in the collisionless pre-thermal states of isolated quantum systems: theory and experiments

Authors: Jamir Marino, Martin Eckstein, Matthew S. Foster, Ana Maria Rey

Abstract: We overview the concept of dynamical phase transitions in isolated quantum systems quenched out of equilibrium. We focus on non-equilibrium transitions characterized by an order parameter, which features qualitatively distinct temporal behaviour on the two sides of a certain dynamical critical point. Dynamical phase transitions are currently mostly understood as long-lived prethermal phenomena in… ▽ More We overview the concept of dynamical phase transitions in isolated quantum systems quenched out of equilibrium. We focus on non-equilibrium transitions characterized by an order parameter, which features qualitatively distinct temporal behaviour on the two sides of a certain dynamical critical point. Dynamical phase transitions are currently mostly understood as long-lived prethermal phenomena in a regime where inelastic collisions are incapable to thermalize the system. The latter enables the dynamics to substain phases that explicitly break detailed balance and therefore cannot be encompassed by traditional thermodynamics. Our presentation covers both cold atoms as well as condensed matter systems. We revisit a broad plethora of platforms exhibiting pre-thermal DPTs, which become theoretically tractable in a certain limit, such as for a large number of particles, large number of order parameter components, or large spatial dimension. The systems we explore include, among others, quantum magnets with collective interactions, $φ^4$ quantum field theories, and Fermi-Hubbard models. A section dedicated to experimental explorations of DPTs in condensed matter and AMO systems connects this large variety of theoretical models. △ Less

Submitted 24 January, 2022; originally announced January 2022.

Comments: 37 pages; 10 figures; review article

Journal ref: Rep. Prog. Phys. 85 116001 (2022)

Vertex-based Diagrammatic Treatment of Light-Matter-Coupled Systems

Authors: Aaram J. Kim, Katharina Lenk, Jiajun Li, Philipp Werner, Martin Eckstein

Abstract: We propose a diagrammatic Monte Carlo approach for general spin-boson models, which can be regarded as a generalization of the strong-coupling expansion for fermionic impurity models. The algorithm is based on a self-consistently computed three-point vertex and a stochastically sampled four-point vertex, and achieves convergence to the numerically exact result in a wide parameter regime. The perfo… ▽ More We propose a diagrammatic Monte Carlo approach for general spin-boson models, which can be regarded as a generalization of the strong-coupling expansion for fermionic impurity models. The algorithm is based on a self-consistently computed three-point vertex and a stochastically sampled four-point vertex, and achieves convergence to the numerically exact result in a wide parameter regime. The performance of the algorithm is demonstrated with applications to a spin-boson model representing an emitter in a waveguide. As a function of the coupling strength, the spin exhibits a delocalization-localization crossover at low temperatures, signaling a qualitative change in the real-time relaxation. In certain parameter regimes, the response functions of the emitter coupled to the electromagnetic continuum can be described by an effective Rabi model with appropriately defined parameters. We also discuss the spatial distribution of the photon density around the emitter. △ Less

Submitted 31 December, 2021; originally announced December 2021.

Comments: 11 pages, 7 figures

Inhomogeneous disordering at a photo-induced charge density wave transition

Authors: Antonio Picano, Francesco Grandi, Martin Eckstein

Abstract: Using ultrashort laser pulses, it has become possible to probe the dynamics of long-range order in solids on microscopic timescales. In the conventional description of symmetry-broken phases within time-dependent Ginzburg-Landau theory, the order parameter evolves coherently, with small fluctuations along an average trajectory. Recent experiments, however, indicate that some systems can support a… ▽ More Using ultrashort laser pulses, it has become possible to probe the dynamics of long-range order in solids on microscopic timescales. In the conventional description of symmetry-broken phases within time-dependent Ginzburg-Landau theory, the order parameter evolves coherently, with small fluctuations along an average trajectory. Recent experiments, however, indicate that some systems can support a different scenario, named ultrafast inhomogeneous disordering, where the average order parameter is no longer representative of the state on the atomic scale. Here we theoretically show that ultrafast disordering can occur in a minimal, yet paradigmatic, model for a Peierls instability if atomic scale inhomogeneities of both the electronic structure and the charge density wave order parameter are taken into account. The latter is achieved using a non-equilibrium generalization of statistical dynamical mean-field theory, coupled to stochastic differential equations for the order parameter. △ Less

Submitted 7 June, 2023; v1 submitted 31 December, 2021; originally announced December 2021.

Comments: 13 pages, 7 figures

Journal ref: Physical Review B 107, 245112 (2023)

Memory truncated Kadanoff-Baym equations

Authors: Christopher Stahl, Nagamalleswararao Dasari, Jiajun Li, Antonio Picano, Philipp Werner, Martin Eckstein

Abstract: The Keldysh formalism for nonequilibrium Green's functions is a powerful theoretical framework for the description of the electronic structure, spectroscopy, and dynamics of strongly correlated systems. However, the underlying Kadanoff-Baym equations (KBE) for the two-time Keldysh Green's functions involve a memory kernel which results in a high computational cost for long simulation times… ▽ More The Keldysh formalism for nonequilibrium Green's functions is a powerful theoretical framework for the description of the electronic structure, spectroscopy, and dynamics of strongly correlated systems. However, the underlying Kadanoff-Baym equations (KBE) for the two-time Keldysh Green's functions involve a memory kernel which results in a high computational cost for long simulation times $t_\text{max}$, with a cubic scaling of the computation time with $t_\text{max}$. Truncation of the memory kernel can reduce the computational cost to linear scaling with $t_\text{max}$, but the required memory times will depend on the model and the diagrammatic approximation to the self-energy. We explain how a truncation of the memory kernel can be incorporated into the time-propagation algorithm to solve the KBE, and investigate the systematic truncation of the memory kernel for the Hubbard model in different parameter regimes, and for different diagrammatic approximations. The truncation is easier to control within dynamical mean-field solutions, where it is applied to a momentum-independent self-energy. Here, simulation times up to two orders of magnitude longer are accessible both in the weak and strong coupling regime, allowing for a study of long-time phenomena such as the crossover between pre-thermalization and thermalization dynamics. △ Less

Submitted 10 December, 2021; originally announced December 2021.

Comments: 14 pages, 11 figures

Ultrafast control of spin-orbital separation probed with time-resolved RIXS

Authors: Aaron Müller, Francesco Grandi, Martin Eckstein

Abstract: Quasi-one-dimensional systems exhibit many-body effects elusive in higher dimensions. A prime example is spin-orbital separation, which has been measured by resonant inelastic X-ray scattering (RIXS) in Sr$_2$CuO$_3$. Here, we theoretically analyze the time-resolved RIXS spectrum of Sr$_2$CuO$_3$ under the action of a time-dependent electric field. We show that the external field can reversibly mo… ▽ More Quasi-one-dimensional systems exhibit many-body effects elusive in higher dimensions. A prime example is spin-orbital separation, which has been measured by resonant inelastic X-ray scattering (RIXS) in Sr$_2$CuO$_3$. Here, we theoretically analyze the time-resolved RIXS spectrum of Sr$_2$CuO$_3$ under the action of a time-dependent electric field. We show that the external field can reversibly modify the parameters in the effective $t-J$ model used to describe spinon and orbiton dynamics in the material. For strong driving amplitudes, we find that the spectrum changes qualitatively as a result of reversing the relative spinon to orbiton velocity. The analysis shows that in general, the spin-orbital dynamics in Mott insulators in combination with time-resolved RIXS should provide a suitable platform to explore the reversible control of many-body physics in the solid with strong laser fields. △ Less

Submitted 30 September, 2022; v1 submitted 24 November, 2021; originally announced November 2021.

Comments: 13 pages, 4 figures

Non-equilibrium evolution of the optical conductivity of the weakly interacting Hubbard model: Drude response and $π$-ton type vertex corrections

Authors: Olivier Simard, Martin Eckstein, Philipp Werner

Abstract: The optical conductivity contains information about energy absorption and the underlying physical processes. In finite-dimensional systems, vertex corrections to the bare bubble need to be considered, which is a computationally challenging task. Recent numerical studies showed that in the weak coupling limit, near an ordering instability with wave vector $π$, $π$-tons (or Maki-Thompson diagrams) y… ▽ More The optical conductivity contains information about energy absorption and the underlying physical processes. In finite-dimensional systems, vertex corrections to the bare bubble need to be considered, which is a computationally challenging task. Recent numerical studies showed that in the weak coupling limit, near an ordering instability with wave vector $π$, $π$-tons (or Maki-Thompson diagrams) yield the most relevant vertex corrections. This provides a route for including vertex corrections into, for example, dynamical mean field theory estimates of the optical conductivity. By implementing calculations on the Kadanoff-Baym contour, we reveal the characteristic spectral signatures of the $π$-tons and their evolution under non-equilibrium conditions. We consider interaction quenches of the weakly-correlated Hubbard model near the antiferromagnetic phase boundary, and analyze the evolution of the Drude and $π$-ton features. While the bubble contribution to the optical conductivity is found to thermalize rapidly, after some oscillations with frequencies related to the local spectral function, the $π$-ton contribution exhibits a slower evolution. We link this observation to the prethermalization phenomenon which has been previously studied in weakly interacting, quenched Hubbard models. △ Less

Submitted 21 December, 2021; v1 submitted 15 October, 2021; originally announced October 2021.

Comments: 15 pages, 12 figures

Fast whole-slide cartography in colon cancer histology using superpixels and CNN classification

Authors: Frauke Wilm, Michaela Benz, Volker Bruns, Serop Baghdadlian, Jakob Dexl, David Hartmann, Petr Kuritcyn, Martin Weidenfeller, Thomas Wittenberg, Susanne Merkel, Arndt Hartmann, Markus Eckstein, Carol I. Geppert

Abstract: Automatic outlining of different tissue types in digitized histological specimen provides a basis for follow-up analyses and can potentially guide subsequent medical decisions. The immense size of whole-slide-images (WSI), however, poses a challenge in terms of computation time. In this regard, the analysis of non-overlap** patches outperforms pixelwise segmentation approaches, but still leaves… ▽ More Automatic outlining of different tissue types in digitized histological specimen provides a basis for follow-up analyses and can potentially guide subsequent medical decisions. The immense size of whole-slide-images (WSI), however, poses a challenge in terms of computation time. In this regard, the analysis of non-overlap** patches outperforms pixelwise segmentation approaches, but still leaves room for optimization. Furthermore, the division into patches, regardless of the biological structures they contain, is a drawback due to the loss of local dependencies. We propose to subdivide the WSI into coherent regions prior to classification by grou** visually similar adjacent pixels into superpixels. Afterwards, only a random subset of patches per superpixel is classified and patch labels are combined into a superpixel label. We propose a metric for identifying superpixels with an uncertain classification and evaluate two medical applications, namely tumor area and invasive margin estimation and tumor composition analysis. The algorithm has been developed on 159 hand-annotated WSIs of colon resections and its performance is compared to an analysis without prior segmentation. The algorithm shows an average speed-up of 41% and an increase in accuracy from 93.8% to 95.7%. By assigning a rejection label to uncertain superpixels, we further increase the accuracy by 0.4%. Whilst tumor area estimation shows high concordance to the annotated area, the analysis of tumor composition highlights limitations of our approach. By combining superpixel segmentation and patch classification, we designed a fast and accurate framework for whole-slide cartography that is AI-model agnostic and provides the basis for various medical endpoints. △ Less

Submitted 15 March, 2022; v1 submitted 30 June, 2021; originally announced June 2021.

Comments: 28 pages, 17 figures, 5 tables, published in SPIE Journal of Medical Imaging

Journal ref: J. Med. Imag. 9(2), 027501 (2022)

ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation

Authors: Wanrong Zhu, Xin Eric Wang, An Yan, Miguel Eckstein, William Yang Wang

Abstract: Automatic evaluations for natural language generation (NLG) conventionally rely on token-level or embedding-level comparisons with text references. This differs from human language processing, for which visual imagination often improves comprehension. In this work, we propose ImaginE, an imagination-based automatic evaluation metric for natural language generation. With the help of StableDiffusion… ▽ More Automatic evaluations for natural language generation (NLG) conventionally rely on token-level or embedding-level comparisons with text references. This differs from human language processing, for which visual imagination often improves comprehension. In this work, we propose ImaginE, an imagination-based automatic evaluation metric for natural language generation. With the help of StableDiffusion, a state-of-the-art text-to-image generator, we automatically generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings. Experiments spanning several text generation tasks demonstrate that adding machine-generated images with our ImaginE displays great potential in introducing multi-modal information into NLG evaluation, and improves existing automatic metrics' correlations with human similarity judgments in both reference-based and reference-free evaluation scenarios. △ Less

Submitted 14 February, 2023; v1 submitted 10 June, 2021; originally announced June 2021.

Comments: EACL 2023

FoveaTer: Foveated Transformer for Image Classification

Authors: Aditya Jonnalagadda, William Yang Wang, B. S. Manjunath, Miguel P. Eckstein

Abstract: Many animals and humans process the visual field with a varying spatial resolution (foveated vision) and use peripheral processing to make eye movements and point the fovea to acquire high-resolution information about objects of interest. This architecture results in computationally efficient rapid scene exploration. Recent progress in self-attention-based Vision Transformers, an alternative to th… ▽ More Many animals and humans process the visual field with a varying spatial resolution (foveated vision) and use peripheral processing to make eye movements and point the fovea to acquire high-resolution information about objects of interest. This architecture results in computationally efficient rapid scene exploration. Recent progress in self-attention-based Vision Transformers, an alternative to the traditionally convolution-reliant computer vision systems. However, the Transformer models do not explicitly model the foveated properties of the visual system nor the interaction between eye movements and the classification task. We propose Foveated Transformer (FoveaTer) model, which uses pooling regions and eye movements to perform object classification tasks using a Vision Transformer architecture. Using square pooling regions or biologically-inspired radial-polar pooling regions, our proposed model pools the image features from the convolution backbone and uses the pooled features as an input to transformer layers. It decides on subsequent fixation location based on the attention assigned by the Transformer to various locations from past and present fixations. It dynamically allocates more fixation/computational resources to more challenging images before making the final image category decision. Using five ablation studies, we evaluate the contribution of different components of the Foveated model. We perform a psychophysics scene categorization task and use the experimental data to find a suitable radial-polar pooling region combination. We also show that the Foveated model better explains the human decisions in a scene categorization task than a Baseline model. We demonstrate our model's robustness against PGD adversarial attacks with both types of pooling regions, where we see the Foveated model outperform the Baseline model. △ Less

Submitted 2 October, 2022; v1 submitted 28 May, 2021; originally announced May 2021.

Effective theory of lattice electrons strongly coupled to quantum electromagnetic fields

Authors: Jiajun Li, Lukas Schamriß, Martin Eckstein

Abstract: Recent experiments have revealed the tantalizing possibility of fabricating lattice electronic systems strongly coupled to quantum fluctuations of electromagnetic fields, e.g., by means of geometry confinement from a cavity or artificial gauge fields in quantum simulators. In this work, we develop a high-frequency expansion to construct the effective models for lattice electrons strongly coupled t… ▽ More Recent experiments have revealed the tantalizing possibility of fabricating lattice electronic systems strongly coupled to quantum fluctuations of electromagnetic fields, e.g., by means of geometry confinement from a cavity or artificial gauge fields in quantum simulators. In this work, we develop a high-frequency expansion to construct the effective models for lattice electrons strongly coupled to a continuum of off-resonant photon modes with arbitrary dispersion. The theory is nonperturbative in the light-matter coupling strength, and is therefore particularly suitable for the ultrastrong light-matter coupling regime. Using the effective models, we demonstrate how the dispersion and topology of the electronic energy bands can be tuned by the cavity. In particular, quasi-one-dimensional physics can emerge in a two-dimensional square lattice due to a spatially anisotropic band renormalization, and a topologically nontrivial anomalous quantum Hall state can be induced in a honeycomb lattice when the cavity setup breaks time-reversal symmetry. We also demonstrate that the photon-mediated interaction induces an unconventional superconducting paired phase distinct from the pair-density-wave state discussed in models with truncated light-matter coupling. Finally, we study a realistic setup of a Fabry-Pérot cavity. Our work provides a systematic framework to explore the emergent phenomena due to strong light-matter coupling and points out new directions of engineering orders and topological states in solids. △ Less

Submitted 30 April, 2022; v1 submitted 18 May, 2021; originally announced May 2021.

Quantum Optimal Transport

Authors: Sam Cole, Michał Eckstein, Shmuel Friedland, Karol Życzkowski

Abstract: We analyze a quantum version of the Monge--Kantorovich optimal transport problem. The quantum transport cost related to a Hermitian cost matrix $C$ is minimized over the set of all bipartite coupling states $ρ^{AB}$ with fixed reduced density matrices $ρ^A$ and $ρ^B$ of size $m$ and $n$. The minimum quantum optimal transport cost $\rT^Q_{C}(ρ^A,ρ^B)$ can be efficiently computed using semidefinite… ▽ More We analyze a quantum version of the Monge--Kantorovich optimal transport problem. The quantum transport cost related to a Hermitian cost matrix $C$ is minimized over the set of all bipartite coupling states $ρ^{AB}$ with fixed reduced density matrices $ρ^A$ and $ρ^B$ of size $m$ and $n$. The minimum quantum optimal transport cost $\rT^Q_{C}(ρ^A,ρ^B)$ can be efficiently computed using semidefinite programming. In the case $m=n$ the cost $\rT^Q_{C}$ gives a semidistance if and only if $C$ is positive semidefinite and vanishes exactly on the subspace of symmetric matrices. Furthermore, if $C$ satisfies the above conditions, then $\sqrt{\rT^Q_{C}}$ induces a quantum analogue of the Wasserstein-2 distance. Taking the quantum cost matrix $C^Q$ to be the projector on the antisymmetric subspace, we provide a semi-analytic expression for $\rT^Q_{C^Q}$ for any pair of single-qubit states and show that its square root yields a transport distance on the Bloch ball. Numerical simulations suggest that this property holds also in higher dimensions. Assuming that the cost matrix suffers decoherence and that the density matrices become diagonal, we study the quantum-to-classical transition of the Earth mover's distance, propose a continuous family of interpolating distances, and demonstrate that the quantum transport is cheaper than the classical one. Furthermore, we introduce a related quantity -- the SWAP-fidelity -- and compare its properties with the standard Uhlmann--Jozsa fidelity. We also discuss the quantum optimal transport for general $d$-partite systems. △ Less

Submitted 12 July, 2022; v1 submitted 14 May, 2021; originally announced May 2021.

Comments: A detailed and expanded version of most mathematical results in arXiv:2102.07787: "Quantum Monge-Kantorovich problem and transport distance between density matrices". 61 pages

MSC Class: 81P40; 90C22; 15A69

Journal ref: Math Phys Anal Geom 26, 14 (2023)

Nonequilibrium RIXS study of an electron-phonon model

Authors: Philipp Werner, Martin Eckstein

Abstract: We use the nonequilibrium dynamical mean field theory formalism to compute the equilibrium and nonequilibrium resonant inelastic X-ray scattering (RIXS) signal of a strongly interacting fermionic lattice model with a coupling of dispersionless phonons to the total charge on a given site. In the atomic limit, this model produces phonon subbands in the spectral function, but not in the RIXS signal.… ▽ More We use the nonequilibrium dynamical mean field theory formalism to compute the equilibrium and nonequilibrium resonant inelastic X-ray scattering (RIXS) signal of a strongly interacting fermionic lattice model with a coupling of dispersionless phonons to the total charge on a given site. In the atomic limit, this model produces phonon subbands in the spectral function, but not in the RIXS signal. Electron hop** processes however result in phonon-related modifications of the charge excitation peak. We discuss the equilibrium RIXS spectra and the characteristic features of nonequilibrium states induced by photo-do** and by the application of a static electric field. The latter produces features related to Wannier-Stark states, which are dressed with phonon sidebands. Thanks to the effect of field-induced localization, the phonon features can be clearly resolved even in systems with weak electron-phonon coupling. △ Less

Submitted 12 May, 2021; originally announced May 2021.

Journal ref: Phys. Rev. B 104, 085155 (2021)

Comparing Visual Reasoning in Humans and AI

Authors: Shravan Murlidaran, William Yang Wang, Miguel P. Eckstein

Abstract: Recent advances in natural language processing and computer vision have led to AI models that interpret simple scenes at human levels. Yet, we do not have a complete understanding of how humans and AI models differ in their interpretation of more complex scenes. We created a dataset of complex scenes that contained human behaviors and social interactions. AI and humans had to describe the scenes w… ▽ More Recent advances in natural language processing and computer vision have led to AI models that interpret simple scenes at human levels. Yet, we do not have a complete understanding of how humans and AI models differ in their interpretation of more complex scenes. We created a dataset of complex scenes that contained human behaviors and social interactions. AI and humans had to describe the scenes with a sentence. We used a quantitative metric of similarity between scene descriptions of the AI/human and ground truth of five other human descriptions of each scene. Results show that the machine/human agreement scene descriptions are much lower than human/human agreement for our complex scenes. Using an experimental manipulation that occludes different spatial regions of the scenes, we assessed how machines and humans vary in utilizing regions of images to understand the scenes. Together, our results are a first step toward understanding how machines fall short of human visual reasoning with complex scenes depicting human behaviors. △ Less

Submitted 29 April, 2021; originally announced April 2021.

Gaze Perception in Humans and CNN-Based Model

Authors: Nicole X. Han, William Yang Wang, Miguel P. Eckstein

Abstract: Making accurate inferences about other individuals' locus of attention is essential for human social interactions and will be important for AI to effectively interact with humans. In this study, we compare how a CNN (convolutional neural network) based model of gaze and humans infer the locus of attention in images of real-world scenes with a number of individuals looking at a common location. We… ▽ More Making accurate inferences about other individuals' locus of attention is essential for human social interactions and will be important for AI to effectively interact with humans. In this study, we compare how a CNN (convolutional neural network) based model of gaze and humans infer the locus of attention in images of real-world scenes with a number of individuals looking at a common location. We show that compared to the model, humans' estimates of the locus of attention are more influenced by the context of the scene, such as the presence of the attended target and the number of individuals in the image. △ Less

Submitted 17 April, 2021; originally announced April 2021.

Ultrafast metal-to-insulator switching in a strongly correlated system

Authors: Francesco Grandi, Martin Eckstein

Abstract: Light-manipulation of correlated electronic phases in solids offers the tantalizing prospect of realizing electronic devices operating at the ultrafast time-scale. In this context, the experimental realization of non-equilibrium transitions from a metal to a band or Mott insulator has shown to be particularly elusive. Using dynamical mean-field theory, we study a simple model representing the main… ▽ More Light-manipulation of correlated electronic phases in solids offers the tantalizing prospect of realizing electronic devices operating at the ultrafast time-scale. In this context, the experimental realization of non-equilibrium transitions from a metal to a band or Mott insulator has shown to be particularly elusive. Using dynamical mean-field theory, we study a simple model representing the main physical properties of the oxygen-enriched compound LaTiO$_{3+x}$. By properly optimizing the photo-do** of electrons from a low-energy band into the valence states of the system, we show it is possible to induce a valence transition from a correlated metallic state to a Mott insulator at ultrashort time scales and to contain the heating during this process, with the final non-thermal valence insulator having almost the same effective temperature of the starting metal. △ Less

Submitted 8 April, 2021; originally announced April 2021.

Comments: 9 pages, 7 figures

M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers

Authors: Tsu-Jui Fu, Xin Eric Wang, Scott T. Grafton, Miguel P. Eckstein, William Yang Wang

Abstract: Video editing tools are widely used nowadays for digital design. Although the demand for these tools is high, the prior knowledge required makes it difficult for novices to get started. Systems that could follow natural language instructions to perform automatic editing would significantly improve accessibility. This paper introduces the language-based video editing (LBVE) task, which allows the m… ▽ More Video editing tools are widely used nowadays for digital design. Although the demand for these tools is high, the prior knowledge required makes it difficult for novices to get started. Systems that could follow natural language instructions to perform automatic editing would significantly improve accessibility. This paper introduces the language-based video editing (LBVE) task, which allows the model to edit, guided by text instruction, a source video into a target video. LBVE contains two features: 1) the scenario of the source video is preserved instead of generating a completely different video; 2) the semantic is presented differently in the target video, and all changes are controlled by the given instruction. We propose a Multi-Modal Multi-Level Transformer (M$^3$L) to carry out LBVE. M$^3$L dynamically learns the correspondence between video perception and language semantic at different levels, which benefits both the video understanding and video frame synthesis. We build three new datasets for evaluation, including two diagnostic and one from natural videos with human-labeled text. Extensive experimental results show that M$^3$L is effective for video editing and that LBVE can lead to a new field toward vision-and-language research. △ Less

Submitted 18 March, 2022; v1 submitted 2 April, 2021; originally announced April 2021.

Comments: CVPR'22

Diagnosing Vision-and-Language Navigation: What Really Matters

Authors: Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Eric Wang, Qi Wu, Miguel Eckstein, William Yang Wang

Abstract: Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Multiple setups have been proposed, and researchers apply new model architectures or training techniques to boost navigation performance. However, there still exist non-negligible gaps between machines' performance and human benchmarks. Moreover, the… ▽ More Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Multiple setups have been proposed, and researchers apply new model architectures or training techniques to boost navigation performance. However, there still exist non-negligible gaps between machines' performance and human benchmarks. Moreover, the agents' inner mechanisms for navigation decisions remain unclear. To the best of our knowledge, how the agents perceive the multimodal input is under-studied and needs investigation. In this work, we conduct a series of diagnostic experiments to unveil agents' focus during navigation. Results show that indoor navigation agents refer to both object and direction tokens when making decisions. In contrast, outdoor navigation agents heavily rely on direction tokens and poorly understand the object tokens. Transformer-based agents acquire a better cross-modal understanding of objects and display strong numerical reasoning ability than non-Transformer-based agents. When it comes to vision-and-language alignments, many models claim that they can align object tokens with specific visual targets. We find unbalanced attention on the vision and text input and doubt the reliability of such cross-modal alignments. △ Less

Submitted 4 May, 2022; v1 submitted 30 March, 2021; originally announced March 2021.

Comments: NAACL 2022

Probing the limits of quantum theory with quantum information at subnuclear scales

Authors: Michał Eckstein, Paweł Horodecki

Abstract: Modern quantum engineering techniques enabled successful foundational tests of quantum mechanics. Yet, the universal validity of quantum postulates is an open question. Here we propose a new theoretical framework of Q-data tests, which recognises the established validity of quantum theory, but allows for more general -- 'post-quantum' -- scenarios in certain physical regimes. It can accommodate a… ▽ More Modern quantum engineering techniques enabled successful foundational tests of quantum mechanics. Yet, the universal validity of quantum postulates is an open question. Here we propose a new theoretical framework of Q-data tests, which recognises the established validity of quantum theory, but allows for more general -- 'post-quantum' -- scenarios in certain physical regimes. It can accommodate a large class of models with modified quantum wave dynamics, correlations beyond entanglement or general probabilistic postulates. We discuss its experimental implementation suited to probe the nature of strong nuclear interactions. In contrast to the present accelerator experiments, it shifts the focus from high-luminosity beam physics to individual particle coherent control. △ Less

Submitted 28 June, 2022; v1 submitted 22 March, 2021; originally announced March 2021.

Comments: 28 pages, 11 figures, published version. Comments are welcome!

MSC Class: 81V05; 81P16; 81P18

Journal ref: Proc. R. Soc. A. 478:20210806 (2022)

Fluctuation control of non-thermal orbital order

Authors: Francesco Grandi, Martin Eckstein

Abstract: Orbitally ordered states exhibit unique features which make them a promising platform for exploring the ultrafast dynamics of long-range order in solids: Their free energy typically has multiple discrete minima, and electric laser fields or selectively excited phonons can exert effective forces that may be used to steer the order parameter through these free energy landscapes. Moreover, their free… ▽ More Orbitally ordered states exhibit unique features which make them a promising platform for exploring the ultrafast dynamics of long-range order in solids: Their free energy typically has multiple discrete minima, and electric laser fields or selectively excited phonons can exert effective forces that may be used to steer the order parameter through these free energy landscapes. Moreover, their free energy strongly depends on fluctuations, and in some cases restoring forces close to a minimum are exclusively of entropic origin (order-by-disorder mechanisms). This can open pathways to control the dynamics of the order parameter via non-thermal fluctuations. In this work, we study the laser-induced non-equilibrium dynamics in a $120^\circ$ compass model, using time-dependent Ginzburg-Landau theory. We analyze protocols to switch the order parameter between equivalent configurations, with a focus on the interplay between the external force due to the driving field, and the non-thermal entropic forces. In particular, we find that remanent non-thermal fluctuations after some excitation can stabilize the high-symmetry phase even when the homogeneous potential has retrieved its low-temperature form, which facilitates laser-induced switching. △ Less

Submitted 19 March, 2021; originally announced March 2021.

Comments: 12 pages, 8 figures

Journal ref: Phys. Rev. B 103, 245117 (2021)