-
Counting and Algorithmic Generalization with Transformers
Authors:
Simon Ouellette,
Rolf Pfister,
Hansueli Jud
Abstract:
Algorithmic generalization in machine learning refers to the ability to learn the underlying algorithm that generates data in a way that generalizes out-of-distribution. This is generally considered a difficult task for most machine learning algorithms. Here, we analyze algorithmic generalization when counting is required, either implicitly or explicitly. We show that standard Transformers are bas…
▽ More
Algorithmic generalization in machine learning refers to the ability to learn the underlying algorithm that generates data in a way that generalizes out-of-distribution. This is generally considered a difficult task for most machine learning algorithms. Here, we analyze algorithmic generalization when counting is required, either implicitly or explicitly. We show that standard Transformers are based on architectural decisions that hinder out-of-distribution performance for such tasks. In particular, we discuss the consequences of using layer normalization and of normalizing the attention weights via softmax. With ablation of the problematic operations, we demonstrate that a modified transformer can exhibit a good algorithmic generalization performance on counting while using a very lightweight architecture.
△ Less
Submitted 12 January, 2024; v1 submitted 12 October, 2023;
originally announced October 2023.
-
Hydration Dynamics and IR Spectroscopy of 4-Fluorophenol
Authors:
Seyedeh Maryam Salehi,
Silvan Käser,
Kai Töpfer,
Polydefkis Diamantis,
Rolf Pfister,
Peter Hamm,
Ursula Röthlisberger,
Markus Meuwly
Abstract:
Halogenated groups are relevant in pharmaceutical applications and potentially useful spectroscopic probes for infrared spectroscopy. In this work, the structural dynamics and infrared spectroscopy of $para$-fluorophenol (F-PhOH) and phenol (PhOH) is investigated in the gas phase and in water using a combination of experiment and molecular dynamics (MD) simulations. The gas phase and solvent dynam…
▽ More
Halogenated groups are relevant in pharmaceutical applications and potentially useful spectroscopic probes for infrared spectroscopy. In this work, the structural dynamics and infrared spectroscopy of $para$-fluorophenol (F-PhOH) and phenol (PhOH) is investigated in the gas phase and in water using a combination of experiment and molecular dynamics (MD) simulations. The gas phase and solvent dynamics around F-PhOH and PhOH is characterized from atomistic simulations using empirical energy functions with point charges or multipoles for the electrostatics, Machine-Learning (ML) based parametrization and with full $\textit{ab initio}$ (QM) and mixed Quantum Mechanical/Molecular Mechanics (QM/MM) simulations with a particular focus on the CF- and OH-stretch region. The CF-stretch band is heavily mixed with other modes whereas the OH-stretch in solution displays a characteristic high-frequency peak around 3600 cm$^{-1}$ most likely associated with the -OH group of PhOH and F-PhOH together with a characteristic progression below 3000 cm$^{-1}$ due to coupling with water modes which is also reproduced by several of the simulations. Solvent and radial distribution functions indicate that the CF-site is largely hydrophobic except for simulations using point charges which renders them unsuited for correctly describing hydration and dynamics around fluorinated sites.
△ Less
Submitted 21 June, 2022;
originally announced June 2022.
-
Tin Pest: A Forgotten Issue in the Field of Applied Superconductivity?
Authors:
R. Pfister,
P. Pugnat
Abstract:
Shear ruptures of Cu samples soldered with Sn96Ag4 and Sn60Pb40 alloys have been measured at 300 K and 77 K. An average degradation of about 37 % of the shear rupture strength has been observed at cold for samples soldered with the lead-free alloy. This effect can be attributed to the tin pest.
Shear ruptures of Cu samples soldered with Sn96Ag4 and Sn60Pb40 alloys have been measured at 300 K and 77 K. An average degradation of about 37 % of the shear rupture strength has been observed at cold for samples soldered with the lead-free alloy. This effect can be attributed to the tin pest.
△ Less
Submitted 6 April, 2012;
originally announced April 2012.
