-
No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
Authors:
Angéline Pouget,
Lucas Beyer,
Emanuele Bugliarello,
Xiao Wang,
Andreas Peter Steiner,
Xiaohua Zhai,
Ibrahim Alabdulmohsin
Abstract:
We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training data to English image-text pairs disadvantages communities of lower socioeconomic status and negatively impacts cultural understanding. Notably, this pe…
▽ More
We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training data to English image-text pairs disadvantages communities of lower socioeconomic status and negatively impacts cultural understanding. Notably, this performance gap is not captured by - and even at odds with - the currently popular evaluation metrics derived from the Western-centric ImageNet and COCO datasets. Second, pretraining with global, unfiltered data before fine-tuning on English content can improve cultural understanding without sacrificing performance on said popular benchmarks. Third, we introduce the task of geo-localization as a novel evaluation metric to assess cultural diversity in VLMs. Our work underscores the value of using diverse data to create more inclusive multimodal systems and lays the groundwork for develo** VLMs that better represent global perspectives.
△ Less
Submitted 24 May, 2024; v1 submitted 22 May, 2024;
originally announced May 2024.
-
Discovering hidden physics using ML-based multimodal super-resolution measurement and its application to fusion plasmas
Authors:
Azarakhsh Jalalvand,
SangKyeun Kim,
Jaemin Seo,
Qiming Hu,
Max Curie,
Peter Steiner,
Andrew Oakleigh Nelson,
Yong-Su Na,
Egemen Kolemen
Abstract:
A non-linear complex system governed by multi-spatial and multi-temporal physics scales cannot be fully understood with a single diagnostic, as each provides only a partial view and much information is lost during data extraction. Combining multiple diagnostics also results in imperfect projections of the system's physics. By identifying hidden inter-correlations between diagnostics, we can levera…
▽ More
A non-linear complex system governed by multi-spatial and multi-temporal physics scales cannot be fully understood with a single diagnostic, as each provides only a partial view and much information is lost during data extraction. Combining multiple diagnostics also results in imperfect projections of the system's physics. By identifying hidden inter-correlations between diagnostics, we can leverage mutual support to fill in these gaps, but uncovering these inter-correlations analytically is too complex. We introduce a groundbreaking machine learning methodology to address this issue. Our multimodal approach generates super resolution data encompassing multiple physics phenomena, capturing detailed structural evolution and responses to perturbations previously unobservable. This methodology addresses a critical problem in fusion plasmas: the Edge Localized Mode (ELM), a plasma instability that can severely damage reactor walls. One method to stabilize ELM is using resonant magnetic perturbation to trigger magnetic islands. However, low spatial and temporal resolution of measurements limits the analysis of these magnetic islands due to their small size, rapid dynamics, and complex interactions within the plasma. With super-resolution diagnostics, we can experimentally verify theoretical models of magnetic islands for the first time, providing unprecedented insights into their role in ELM stabilization. This advancement aids in develo** effective ELM suppression strategies for future fusion reactors like ITER and has broader applications, potentially revolutionizing diagnostics in fields such as astronomy, astrophysics, and medical imaging.
△ Less
Submitted 27 June, 2024; v1 submitted 9 May, 2024;
originally announced May 2024.
-
Multi-level recording in dual-layer FePt-C granular film for heat-assisted magnetic recording
Authors:
P. Tozman,
S. Isogami,
I. Suzuki,
A. Bolyachkin,
H. Sepehri-Amin,
S. J. Greaves,
H. Suto,
Y. Sasaki,
H. T. Y. Chang,
Y Kubota,
P. Steiner,
P. -W. Huang,
K. Hono,
Y. K. Takahashi
Abstract:
Multi-level magnetic recording is a new concept for increasing the data storage capacity of hard disk drives. However, its implementation has been limited by a lack of suitable media capable of storing information at multiple levels. Herein, we overcome this problem by develo** dual FePt-C nanogranular films separated by a Ru-C breaking layer with a cubic crystal structure. The FePt grains in th…
▽ More
Multi-level magnetic recording is a new concept for increasing the data storage capacity of hard disk drives. However, its implementation has been limited by a lack of suitable media capable of storing information at multiple levels. Herein, we overcome this problem by develo** dual FePt-C nanogranular films separated by a Ru-C breaking layer with a cubic crystal structure. The FePt grains in the bottom and top layers of the developed media exhibited different effective magnetocrystalline anisotropies and Curie temperatures. The former is realized by different degrees of ordering in the L10-FePt grains, whereas the latter was attributed to the diffusion of Ru, thereby enabling separate magnetic recordings at each layer under different magnetic fields and temperatures. Furthermore, the magnetic measurements and heat-assisted magnetic recording simulations showed that these media enabled 3-level recording and could potentially be extended to 4-level recording, as the up-down and down-up states exhibited non-zero magnetization.
△ Less
Submitted 5 July, 2023;
originally announced July 2023.
-
PaLI-X: On Scaling up a Multilingual Vision and Language Model
Authors:
Xi Chen,
Josip Djolonga,
Piotr Padlewski,
Basil Mustafa,
Soravit Changpinyo,
Jialin Wu,
Carlos Riquelme Ruiz,
Sebastian Goodman,
Xiao Wang,
Yi Tay,
Siamak Shakeri,
Mostafa Dehghani,
Daniel Salz,
Mario Lucic,
Michael Tschannen,
Arsha Nagrani,
Hexiang Hu,
Mandar Joshi,
Bo Pang,
Ceslee Montgomery,
Paulina Pietrzyk,
Marvin Ritter,
AJ Piergiovanni,
Matthias Minderer,
Filip Pavetic
, et al. (18 additional authors not shown)
Abstract:
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-sh…
▽ More
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.
△ Less
Submitted 29 May, 2023;
originally announced May 2023.
-
Reversible Ionic Liquid Intercalation for Electrically Controlled Thermal Radiation from Graphene Devices
Authors:
Xiaoxiao Yu,
Gokhan Bakan,
Hengyi Guo,
M. Said Ergoktas,
Pietro Steiner,
Coskun Kocabas
Abstract:
Using graphene as a tuneable optical material enables a series of optical devices such as switchable radar absorbers, variable infrared emissivity surfaces, or visible electrochromic devices. These devices rely on controlling the charge density on graphene with electrostatic gating or intercalation. In this paper, we studied the effect of ionic liquid intercalation on the long-term performance of…
▽ More
Using graphene as a tuneable optical material enables a series of optical devices such as switchable radar absorbers, variable infrared emissivity surfaces, or visible electrochromic devices. These devices rely on controlling the charge density on graphene with electrostatic gating or intercalation. In this paper, we studied the effect of ionic liquid intercalation on the long-term performance of optoelectronic devices operating within a broad infrared wavelength range. Our spectroscopic and thermal characterization results reveal the key limiting factors for the intercalation process and the performance of the infrared devices, such as the electrolyte ion-size asymmetry and charge distribution scheme and the effects of oxygen. Our results provide insight for the limiting mechanism for graphene applications in infrared thermal management and tunable heat signature control.
△ Less
Submitted 15 June, 2023; v1 submitted 20 February, 2023;
originally announced February 2023.
-
Electrically controlled heat transport in multilayer graphene
Authors:
Pietro Steiner,
Saqeeb Adnan,
Muhammed Said Ergoktas,
Julien Barrier,
Xiaoxiao Yu,
Vicente Orts,
Gokhan Bakan,
Jonathan Aze,
Yury Malevich,
Kaiyuan Wang,
Pietro Cataldi,
Mark Bisset,
Sinan Balci,
Sefik Suzer,
Marat Khafizov,
Coskun Kocabas
Abstract:
The ability to control heat transport with electrical signals has been an outstanding challenge due to the lack of efficient electrothermal materials. Previous attempts have mainly concentrated on phase-change and layered materials and encountered various problems such as low thermal conductivities and modest on/off ratios. Here, we demonstrate a graphene-based electrothermal switch enabling elect…
▽ More
The ability to control heat transport with electrical signals has been an outstanding challenge due to the lack of efficient electrothermal materials. Previous attempts have mainly concentrated on phase-change and layered materials and encountered various problems such as low thermal conductivities and modest on/off ratios. Here, we demonstrate a graphene-based electrothermal switch enabling electrically tuneable heat flow. The device uses reversible electro-intercalation of ions to modulate the in-plane thermal conductivity of graphene film by over thirteen-fold via electrically tuneable phonon scattering. We anticipate that our results could provide a realistic pathway for adaptive thermal transport enabling a new class of electrically driven thermal devices which would find a broad spectrum of applications in aerospace and microelectronics.
△ Less
Submitted 21 February, 2022;
originally announced February 2022.
-
PyRCN: A Toolbox for Exploration and Application of Reservoir Computing Networks
Authors:
Peter Steiner,
Azarakhsh Jalalvand,
Simon Stone,
Peter Birkholz
Abstract:
Reservoir Computing Networks (RCNs) belong to a group of machine learning techniques that project the input space non-linearly into a high-dimensional feature space, where the underlying task can be solved linearly. Popular variants of RCNs are capable of solving complex tasks equivalently to widely used deep neural networks, but with a substantially simpler training paradigm based on linear regre…
▽ More
Reservoir Computing Networks (RCNs) belong to a group of machine learning techniques that project the input space non-linearly into a high-dimensional feature space, where the underlying task can be solved linearly. Popular variants of RCNs are capable of solving complex tasks equivalently to widely used deep neural networks, but with a substantially simpler training paradigm based on linear regression. In this paper, we show how to uniformly describe RCNs with small and clearly defined building blocks, and we introduce the Python toolbox PyRCN (Python Reservoir Computing Networks) for optimizing, training and analyzing RCNs on arbitrarily large datasets. The tool is based on widely-used scientific packages and complies with the scikit-learn interface specification. It provides a platform for educational and exploratory analyses of RCNs, as well as a framework to apply RCNs on complex tasks including sequence processing. With a small number of building blocks, the framework allows the implementation of numerous different RCN architectures. We provide code examples on how to set up RCNs for time series prediction and for sequence classification tasks. PyRCN is around ten times faster than reference toolboxes on a benchmark task while requiring substantially less boilerplate code.
△ Less
Submitted 10 May, 2022; v1 submitted 8 March, 2021;
originally announced March 2021.
-
Cluster-based Input Weight Initialization for Echo State Networks
Authors:
Peter Steiner,
Azarakhsh Jalalvand,
Peter Birkholz
Abstract:
Echo State Networks (ESNs) are a special type of recurrent neural networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. Despite the recent success of ESNs in various tasks of audio, image and radar recognition, we postulate that a purely random initialization is not the ideal way of initializing ESNs. The aim of…
▽ More
Echo State Networks (ESNs) are a special type of recurrent neural networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. Despite the recent success of ESNs in various tasks of audio, image and radar recognition, we postulate that a purely random initialization is not the ideal way of initializing ESNs. The aim of this work is to propose an unsupervised initialization of the input connections using the $K$-Means algorithm on the training data. We show that for a large variety of datasets this initialization performs equivalently or superior than a randomly initialized ESN whilst needing significantly less reservoir neurons. Furthermore, we discuss that this approach provides the opportunity to estimate a suitable size of the reservoir based on prior knowledge about the data.
△ Less
Submitted 20 January, 2022; v1 submitted 8 March, 2021;
originally announced March 2021.
-
Multispectral Electro-Optical Surfaces: from Visible to Microwave
Authors:
M. Said Ergoktas,
Gokhan Bakan,
Evgeniya Kovalska,
Lewis W. Le Fevre,
Richard P. Fields,
Pietro Steiner,
Xiaoxiao Yu,
Omer Salihoglu,
Sinan Balci,
Kostya Novoselov,
Robert A. W. Dryfe,
Coskun Kocabas
Abstract:
Optical materials with colour-changing abilities have been explored for display devices, smart windows, or modulation of visual appearance. The efficiency of these materials, however, has strong wavelength dependence, which limits their functionality to a narrow spectral range. Here, we report graphene-based electro-optical devices with unprecedented optical tunability covering the entire electrom…
▽ More
Optical materials with colour-changing abilities have been explored for display devices, smart windows, or modulation of visual appearance. The efficiency of these materials, however, has strong wavelength dependence, which limits their functionality to a narrow spectral range. Here, we report graphene-based electro-optical devices with unprecedented optical tunability covering the entire electromagnetic spectrum from the visible to microwave. We achieve this non-volatile and reversible tunability by electro-intercalation of lithium into graphene layers in an optically accessible device structure. This unique colour-changing capability, together with area-selective intercalation, inspires fabrication of new multispectral devices, including display devices and electro-optical camouflage coating. We anticipate that these results provide realistic approaches for programmable smart optical surfaces with a potential utility in many scientific and engineering fields.
△ Less
Submitted 16 July, 2020;
originally announced July 2020.
-
Video-speed Graphene Modulator Arrays for Terahertz Imaging Applications
Authors:
Yury Malevich,
M. Said Ergoktas,
Gokhan Bakan,
Pietro Steiner,
Coskun Kocabas
Abstract:
Electrically tuneable high mobility charges on graphene yield an efficient electro-optical platform to control and manipulate terahertz (THz) waves. Real-world applications require a multiplex THz device with efficient modulation over a large active area. The trade-off between the efficient gating and switching speed, however, hinders the realization of these applications. Here, we demonstrate a l…
▽ More
Electrically tuneable high mobility charges on graphene yield an efficient electro-optical platform to control and manipulate terahertz (THz) waves. Real-world applications require a multiplex THz device with efficient modulation over a large active area. The trade-off between the efficient gating and switching speed, however, hinders the realization of these applications. Here, we demonstrate a large-format 256-pixel THz modulator which provides high-frame-rate reconfigurable transmission patterns. The time-domain and frequency-domain THz characterizations of graphene devices reveal the relaxation pathways of gate-induced charges and ion packing at graphene-electrolyte interface. The fundamental understanding of these limiting factors enables us to break the trade-off permitting switching frequencies up to 1 kHz. To show the promises of these devices, we demonstrate a single-pixel THz camera which allows spatial and spectroscopic imaging of large-area objects without any moving components. These results provide a significant advancement towards the achievement of non-invasive THz imaging systems using graphene-based platforms.
△ Less
Submitted 1 June, 2020;
originally announced June 2020.
-
Multifunctional Biocomposites based on Polyhydroxyalkanoate and Graphene/Carbon-Nanofiber Hybrids for Electrical and Thermal Applications
Authors:
Pietro Cataldi,
Pietro Steiner,
Thomas Raine,
Kailing Lin,
Coskun Kocabas,
Robert J. Young,
Mark Bissett,
Ian A. Kinloch,
Dimitrios G. Papageorgiou
Abstract:
Most polymers are long-lasting and produced from monomers derived from fossil fuel sources. Bio-based and/or biodegradable plastics have been proposed as a sustainable alternative. Amongst those available, polyhydroxyalkanoate (PHA) shows great potential across a large variety of applications but is currently limited to packaging, cosmetics and tissue engineering due to its relatively poor physica…
▽ More
Most polymers are long-lasting and produced from monomers derived from fossil fuel sources. Bio-based and/or biodegradable plastics have been proposed as a sustainable alternative. Amongst those available, polyhydroxyalkanoate (PHA) shows great potential across a large variety of applications but is currently limited to packaging, cosmetics and tissue engineering due to its relatively poor physical properties. An expansion of its uses can be accomplished by develo** nanocomposites where PHAs are used as the polymer matrix. Herein, a PHA biopolyester was melt blended with graphene nanoplatelets (GNPs) or with a 1:1 hybrid mixture of GNPs and carbon nanofibers (CNFs). The resulting nanocomposites exhibited enhanced thermal stability while their Young's modulus roughly doubled compared to pure PHA. The hybrid nanocomposites percolated electrically at lower nanofiller loadings compared to the GNP-PHA system. The electrical conductivity at 15 wt.% loading was ~ 6 times higher than the GNP-based sample. As a result, the electromagnetic interference shielding performance of the hybrid material was around 50% better than the pure GNPs nanocomposites, exhibiting shielding effectiveness above 20 dB, which is the threshold for common commercial applications. The thermal conductivity increased significantly for both types of bio-nanocomposites and reached values around 5 W K-1 m-1 with the hybrid-based material displaying the best performance. Considering the solvent-free and industrially compatible production method, the proposed multifunctional materials are promising to expand the range of application of PHAs and increase the environmental sustainability of the plastic and plastic electronics industry.
△ Less
Submitted 18 May, 2020;
originally announced May 2020.
-
Hybrid Graphene/Carbon Nanofiber Wax Emulsion for Paper-based Electronics and Thermal Management
Authors:
Xinhui Wu,
Pietro Steiner,
Thomas Raine,
Gergo Pinter,
Andrey Kretinin,
Coskun Kocabas,
Mark Bissett,
Pietro Cataldi
Abstract:
Materials for electronics that function as electrical and/or thermal conductors are often rigid, expensive, difficult to be sourced and sometimes toxic. An electrically and thermally conductive nanocomposite that is lightweight, flexible and eco-friendly could improve the environmental friendliness of the electronics sector and enable new applications. Considering this, we have fabricated electric…
▽ More
Materials for electronics that function as electrical and/or thermal conductors are often rigid, expensive, difficult to be sourced and sometimes toxic. An electrically and thermally conductive nanocomposite that is lightweight, flexible and eco-friendly could improve the environmental friendliness of the electronics sector and enable new applications. Considering this, we have fabricated electrically and thermally conductive flexible materials by functionalizing paper with nanocarbon conductive inks. Carnauba wax is emulsified in isopropyl alcohol and mixed with graphene nanoplatelets (GNPs) or with hybrids of GNPs and carbon nanofibers (CNFs). The percolation threshold of the hybrid samples is lowered compared with the pure GNPs composites, due to their increased filler aspect ratio. The hybrid samples also exhibit superior bending and folding stability. Densification of the coating to decrease their sheet resistance enables them to achieve as low as ~ 50 Ω sq-1 for the GNP-based paper. The densification procedure improves the bending stability, the abrasion resistance, and the electromagnetic interference shielding of the paper-based conductors. Finally, the compressed samples show an impressive enhancement of their thermal diffusivity. The flexible and multifunctional nanocarbon coated paper is a promising electronic conductor and thermally dissipative material and, at the same time, can increase the environmental sustainability of the electronics sector.
△ Less
Submitted 23 April, 2020;
originally announced April 2020.
-
Graphene-enabled adaptive infrared textiles
Authors:
M. Said Ergoktas,
Gokhan Bakan,
Pietro Steiner,
Cian Bartlam,
Yury Malevich,
Elif Ozden Yenigun,
Guanliang He,
Nazmul Karim,
Pietro Cataldi,
Mark Bissett,
Ian A. Kinloch,
Kostya S. Novoselov,
Coskun Kocabas
Abstract:
Interactive clothing requires sensing and display functionalities to be embedded on textiles. Despite the significant progress of electronic textiles, the integration of optoelectronic materials on fabrics still remains as an outstanding challenge. Here, using the electro-optical tunability of graphene, we report adaptive optical textiles with electrically controlled reflectivity and emissivity co…
▽ More
Interactive clothing requires sensing and display functionalities to be embedded on textiles. Despite the significant progress of electronic textiles, the integration of optoelectronic materials on fabrics still remains as an outstanding challenge. Here, using the electro-optical tunability of graphene, we report adaptive optical textiles with electrically controlled reflectivity and emissivity covering the infrared and near-infrared wavelengths. We achieve electro-optical modulation by reversible intercalation of ions into graphene layers laminated on fabrics. We demonstrate a new class of infrared textile devices including display, yarn and stretchable devices using natural and synthetic textiles. To show the promise of our approach, we fabricated an active device directly onto a t-shirt which enables long-wavelength infrared communication via modulation of the thermal radiation from the human body. The results presented here, provide complementary technologies which could leverage the ubiquitous use of functional textiles.
△ Less
Submitted 18 April, 2020;
originally announced April 2020.
-
Coherent exciton dynamics in supramolecular light-harvesting nanotubes revealed by ultrafast quantum process tomography
Authors:
Joel Yuen-Zhou,
Dylan H. Arias,
Dorthe M. Eisele,
Colby P. Steiner,
Jacob J. Krich,
Moungi Bawendi,
Keith A. Nelson,
Alán Aspuru-Guzik
Abstract:
Long-lived exciton coherences have been recently observed in photosynthetic complexes via ultrafast spectroscopy, opening exciting possibilities for the study and design of coherent exciton transport. Yet, ambiguity in the spectroscopic signals has led to arguments for interpreting them in terms of the exciton dynamics, demanding more stringent tests. We propose a novel strategy, Quantum Process T…
▽ More
Long-lived exciton coherences have been recently observed in photosynthetic complexes via ultrafast spectroscopy, opening exciting possibilities for the study and design of coherent exciton transport. Yet, ambiguity in the spectroscopic signals has led to arguments for interpreting them in terms of the exciton dynamics, demanding more stringent tests. We propose a novel strategy, Quantum Process Tomography (QPT) for ultrafast spectroscopy, to reconstruct the evolving quantum state of excitons in double-walled supramolecular light-harvesting nanotubes at room temperature. The protocol calls for eight transient grating experiments with varied pulse spectra. Our analysis reveals unidirectional energy transfer from the outer to the inner wall excitons, absence of nonsecular processes, and an unexpected coherence between those two states lasting about 150 femtoseconds, indicating weak electronic coupling between the walls. Our work constitutes the first experimental QPT in a 'warm' and complex system, and provides an elegant scheme to maximize information from ultrafast spectroscopy experiments.
△ Less
Submitted 19 August, 2013;
originally announced August 2013.
-
Solar constraints on new couplings between electromagnetism and gravity
Authors:
S. K. Solanki,
O. Preuss,
M. P. Haugan,
A. Gandorfer,
H. P. Povel,
P. Steiner,
K. Stucki,
P. N. Bernasconi,
D. Soltau
Abstract:
The unification of quantum field theory and general relativity is a fundamental goal of modern physics. In many cases, theoretical efforts to achieve this goal introduce auxiliary gravitational fields, ones in addition to the familiar symmetric second-rank tensor potential of general relativity, and lead to nonmetric theories because of direct couplings between these auxiliary fields and matter.…
▽ More
The unification of quantum field theory and general relativity is a fundamental goal of modern physics. In many cases, theoretical efforts to achieve this goal introduce auxiliary gravitational fields, ones in addition to the familiar symmetric second-rank tensor potential of general relativity, and lead to nonmetric theories because of direct couplings between these auxiliary fields and matter. Here, we consider an example of a metric-affine gauge theory of gravity in which torsion couples nonminimally to the electromagnetic field. This coupling causes a phase difference to accumulate between different polarization states of light as they propagate through the metric-affine gravitational field. Solar spectropolarimetric observations are reported and used to set strong constraints on the relevant coupling constant k: k^2 < (2.5 km)^2.
△ Less
Submitted 12 February, 2004;
originally announced February 2004.
