Criticality versus uniformity in deep neural networks
Authors:
Aleksandar Bukva,
Jurriaan de Gier,
Kevin T. Grosvenor,
Ro Jefferson,
Koenraad Schalm,
Eliot Schwander
Abstract:
Deep feedforward networks initialized along the edge of chaos exhibit exponentially superior training ability as quantified by maximum trainable depth. In this work, we explore the effect of saturation of the tanh activation function along the edge of chaos. In particular, we determine the line of uniformity in phase space along which the post-activation distribution has maximum entropy. This line…
▽ More
Deep feedforward networks initialized along the edge of chaos exhibit exponentially superior training ability as quantified by maximum trainable depth. In this work, we explore the effect of saturation of the tanh activation function along the edge of chaos. In particular, we determine the line of uniformity in phase space along which the post-activation distribution has maximum entropy. This line intersects the edge of chaos, and indicates the regime beyond which saturation of the activation function begins to impede training efficiency. Our results suggest that initialization along the edge of chaos is a necessary but not sufficient condition for optimal trainability.
△ Less
Submitted 10 April, 2023;
originally announced April 2023.
IQUAFLOW: A new framework to measure image quality
Authors:
P. Gallés,
K. Takats,
M. Hernández-Cabronero,
D. Berga,
L. Pega,
L. Riordan-Chen,
C. Garcia,
G. Becker,
A. Garriga,
A. Bukva,
J. Serra-Sagristà,
D. Vilaseca,
J. Marín
Abstract:
IQUAFLOW is a new image quality framework that provides a set of tools to assess image quality. The user can add custom metrics that can be easily integrated. Furthermore, iquaflow allows to measure quality by using the performance of AI models trained on the images as a proxy. This also helps to easily make studies of performance degradation of several modifications of the original dataset, for i…
▽ More
IQUAFLOW is a new image quality framework that provides a set of tools to assess image quality. The user can add custom metrics that can be easily integrated. Furthermore, iquaflow allows to measure quality by using the performance of AI models trained on the images as a proxy. This also helps to easily make studies of performance degradation of several modifications of the original dataset, for instance, with images reconstructed after different levels of lossy compression; satellite images would be a use case example, since they are commonly compressed before downloading to the ground. In this situation, the optimization problem consists in finding the smallest images that provide yet sufficient quality to meet the required performance of the deep learning algorithms. Thus, a study with iquaflow is suitable for such case. All this development is wrapped in Mlflow: an interactive tool used to visualize and summarize the results. This document describes different use cases and provides links to their respective repositories. To ease the creation of new studies, we include a cookie-cutter repository. The source code, issue tracker and aforementioned repositories are all hosted on GitHub https://github.com/satellogic/iquaflow.
△ Less
Submitted 24 October, 2022;
originally announced October 2022.
