-
Partial Search in a Frozen Network is Enough to Find a Strong Lottery Ticket
Authors:
Hikari Otsuka,
Daiki Chijiwa,
Ángel López García-Arias,
Yasuyuki Okoshi,
Kazushi Kawamura,
Thiem Van Chu,
Daichi Fujiki,
Susumu Takeuchi,
Masato Motomura
Abstract:
Randomly initialized dense networks contain subnetworks that achieve high accuracy without weight learning -- strong lottery tickets (SLTs). Recently, Gadhikar et al. (2023) demonstrated that SLTs can also be found within a randomly pruned source network, thus reducing the SLT search space. However, this limits the search to SLTs that are even sparser than the source, leading to worse accuracy due…
▽ More
Randomly initialized dense networks contain subnetworks that achieve high accuracy without weight learning -- strong lottery tickets (SLTs). Recently, Gadhikar et al. (2023) demonstrated that SLTs can also be found within a randomly pruned source network, thus reducing the SLT search space. However, this limits the search to SLTs that are even sparser than the source, leading to worse accuracy due to unintentionally high sparsity. This paper proposes a method that reduces the SLT search space by an arbitrary ratio independent of the desired SLT sparsity. A random subset of the initial weights is excluded from the search space by freezing it -- i.e., by either permanently pruning them or locking them as a fixed part of the SLT. In addition to reducing search space, the proposed random freezing can also provide the benefit of reducing the model size for inference. Furthermore, experimental results show that the proposed method finds SLTs with better accuracy-to-model size trade-off than the SLTs obtained from dense or randomly pruned source networks. In particular, the SLTs found in Frozen ResNets on image classification using ImageNet significantly improve the accuracy-to-search space and accuracy-to-model size trade-offs over SLTs within dense (non-freezing) or sparse (non-locking) random networks.
△ Less
Submitted 3 June, 2024; v1 submitted 19 February, 2024;
originally announced February 2024.
-
Multicoated and Folded Graph Neural Networks with Strong Lottery Tickets
Authors:
Jiale Yan,
Hiroaki Ito,
Ángel López García-Arias,
Yasuyuki Okoshi,
Hikari Otsuka,
Kazushi Kawamura,
Thiem Van Chu,
Masato Motomura
Abstract:
The Strong Lottery Ticket Hypothesis (SLTH) demonstrates the existence of high-performing subnetworks within a randomly initialized model, discoverable through pruning a convolutional neural network (CNN) without any weight training. A recent study, called Untrained GNNs Tickets (UGT), expanded SLTH from CNNs to shallow graph neural networks (GNNs). However, discrepancies persist when comparing ba…
▽ More
The Strong Lottery Ticket Hypothesis (SLTH) demonstrates the existence of high-performing subnetworks within a randomly initialized model, discoverable through pruning a convolutional neural network (CNN) without any weight training. A recent study, called Untrained GNNs Tickets (UGT), expanded SLTH from CNNs to shallow graph neural networks (GNNs). However, discrepancies persist when comparing baseline models with learned dense weights. Additionally, there remains an unexplored area in applying SLTH to deeper GNNs, which, despite delivering improved accuracy with additional layers, suffer from excessive memory requirements. To address these challenges, this work utilizes Multicoated Supermasks (M-Sup), a scalar pruning mask method, and implements it in GNNs by proposing a strategy for setting its pruning thresholds adaptively. In the context of deep GNNs, this research uncovers the existence of untrained recurrent networks, which exhibit performance on par with their trained feed-forward counterparts. This paper also introduces the Multi-Stage Folding and Unshared Masks methods to expand the search space in terms of both architecture and parameters. Through the evaluation of various datasets, including the Open Graph Benchmark (OGB), this work establishes a triple-win scenario for SLTH-based GNNs: by achieving high sparsity, competitive performance, and high memory efficiency with up to 98.7\% reduction, it demonstrates suitability for energy-efficient graph processing.
△ Less
Submitted 5 December, 2023;
originally announced December 2023.
-
Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks
Authors:
Ángel López García-Arias,
Masanori Hashimoto,
Masato Motomura,
Jaehoon Yu
Abstract:
Deep neural networks (DNNs) are so over-parametrized that recent research has found them to already contain a subnetwork with high accuracy at their randomly initialized state. Finding these subnetworks is a viable alternative training method to weight learning. In parallel, another line of work has hypothesized that deep residual networks (ResNets) are trying to approximate the behaviour of shall…
▽ More
Deep neural networks (DNNs) are so over-parametrized that recent research has found them to already contain a subnetwork with high accuracy at their randomly initialized state. Finding these subnetworks is a viable alternative training method to weight learning. In parallel, another line of work has hypothesized that deep residual networks (ResNets) are trying to approximate the behaviour of shallow recurrent neural networks (RNNs) and has proposed a way for compressing them into recurrent models. This paper proposes blending these lines of research into a highly compressed yet accurate model: Hidden-Fold Networks (HFNs). By first folding ResNet into a recurrent structure and then searching for an accurate subnetwork hidden within the randomly initialized model, a high-performing yet tiny HFN is obtained without ever updating the weights. As a result, HFN achieves equivalent performance to ResNet50 on CIFAR100 while occupying 38.5x less memory, and similar performance to ResNet34 on ImageNet with a memory size 26.8x smaller. The HFN will become even more attractive by minimizing data transfers while staying accurate when it runs on highly-quantized and randomly-weighted DNN inference accelerators. Code available at https://github.com/Lopez-Angel/hidden-fold-networks
△ Less
Submitted 24 November, 2021;
originally announced November 2021.
-
Real-time Tone Map**: A State of the Art Report
Authors:
Yafei Ou,
Prasoon Ambalathankandy,
Masayuki Ikebe,
Shinya Takamaeda,
Masato Motomura,
Tetsuya Asai
Abstract:
The rising demand for high quality display has ensued active research in high dynamic range (HDR) imaging, which has the potential to replace the standard dynamic range imaging. This is due to HDR's features like accurate reproducibility of a scene with its entire spectrum of visible lighting and color depth. But this capability comes with expensive capture, display, storage and distribution resou…
▽ More
The rising demand for high quality display has ensued active research in high dynamic range (HDR) imaging, which has the potential to replace the standard dynamic range imaging. This is due to HDR's features like accurate reproducibility of a scene with its entire spectrum of visible lighting and color depth. But this capability comes with expensive capture, display, storage and distribution resource requirements. Also, display of HDR images/video content on an ordinary display device with limited dynamic range requires some form of adaptation. Many adaptation algorithms, widely known as tone map** operators, have been studied and proposed in the last few decades. In this state of the art report, we present a comprehensive survey of 50+ tone map** algorithms that have been implemented on hardware for acceleration and real-time performance. These algorithms have been adapted or redesigned to make them hardware-friendly. All real-time application poses strict timing constraints which requires time exact processing of the algorithm. This design challenge require novel solution, and in this report we focus on these issues. In this we survey will discuss those tonemap algorithms which have been implemented on GPU [1-10], FPGA [11-41], and ASIC [42-53] in terms of their hardware specifications and performance. Output image quality is an important metric for tonemap algorithms. From our literature survey we found that, various objective quality metrics have been used to demonstrate the functionality of adapting the algorithm on hardware platform. We have compiled and studied all the metrics used in this survey [54-67]. Finally, in this report we demonstrate the link between hardware cost and image quality thereby illustrating the underlying trade-off which will be useful for the research community.
△ Less
Submitted 6 March, 2020;
originally announced March 2020.
-
Unbound or Distant Planetary Mass Population Detected by Gravitational Microlensing
Authors:
T. Sumi,
K. Kamiya,
A. Udalski,
D. P. Bennett,
I. A. Bond,
F. Abe,
C. S. Botzler,
A. Fukui,
K. Furusawa,
J. B. Hearnshaw,
Y. Itow,
P. M. Kilmartin,
A. Korpela,
W. Lin,
C. H. Ling,
K. Masuda,
Y. Matsubara,
N. Miyake,
M. Motomura,
Y. Muraki,
M. Nagaya,
S. Nakamura,
K. Ohnishi,
T. Okumura,
Y. C. Perrott
, et al. (14 additional authors not shown)
Abstract:
Since 1995, more than 500 exoplanets have been detected using different techniques, of which 11 were detected with gravitational microlensing. Most of these are gravitationally bound to their host stars. There is some evidence of free-floating planetary mass objects in young star-forming regions, but these objects are limited to massive objects of 3 to 15 Jupiter masses with large uncertainties in…
▽ More
Since 1995, more than 500 exoplanets have been detected using different techniques, of which 11 were detected with gravitational microlensing. Most of these are gravitationally bound to their host stars. There is some evidence of free-floating planetary mass objects in young star-forming regions, but these objects are limited to massive objects of 3 to 15 Jupiter masses with large uncertainties in photometric mass estimates and their abundance. Here, we report the discovery of a population of unbound or distant Jupiter-mass objects, which are almost twice (1.8_{-0.8}^{+1.7}) as common as main-sequence stars, based on two years of gravitational microlensing survey observations toward the Galactic Bulge. These planetary-mass objects have no host stars that can be detected within about ten astronomical units by gravitational microlensing. However a comparison with constraints from direct imaging suggests that most of these planetary-mass objects are not bound to any host star. An abrupt change in the mass function at about a Jupiter mass favours the idea that their formation process is different from that of stars and brown dwarfs. They may have formed in proto-planetary disks and subsequently scattered into unbound or very distant orbits.
△ Less
Submitted 18 May, 2011;
originally announced May 2011.
-
OGLE-2005-BLG-153: Microlensing Discovery and Characterization of A Very Low Mass Binary
Authors:
K. -H. Hwang,
A. Udalski,
C. Han,
Y. -H. Ryu,
I. A. Bond,
J. -P. Beaulieu,
M. Dominik,
K. Horne,
A. Gould,
B. S. Gaudi,
M. Kubiak,
M. K. Szymanski,
G. Pietrzynski,
I. Soszynski,
O. Szewczyk,
K. Ulaczyk,
L. Wyrzykowski,
F. Abe,
C. S. Botzler,
J. B. Hearnshaw,
Y. Itow,
K. Kamiya,
P. M. Kilmartin,
K. Masuda,
Y. Matsubara
, et al. (55 additional authors not shown)
Abstract:
The mass function and statistics of binaries provide important diagnostics of the star formation process. Despite this importance, the mass function at low masses remains poorly known due to observational difficulties caused by the faintness of the objects. Here we report the microlensing discovery and characterization of a binary lens composed of very low-mass stars just above the hydrogen-burnin…
▽ More
The mass function and statistics of binaries provide important diagnostics of the star formation process. Despite this importance, the mass function at low masses remains poorly known due to observational difficulties caused by the faintness of the objects. Here we report the microlensing discovery and characterization of a binary lens composed of very low-mass stars just above the hydrogen-burning limit. From the combined measurements of the Einstein radius and microlens parallax, we measure the masses of the binary components of $0.10\pm 0.01\ M_\odot$ and $0.09\pm 0.01\ M_\odot$. This discovery demonstrates that microlensing will provide a method to measure the mass function of all Galactic populations of very low mass binaries that is independent of the biases caused by the luminosity of the population.
△ Less
Submitted 25 April, 2012; v1 submitted 2 September, 2010;
originally announced September 2010.
-
Masses and Orbital Constraints for the OGLE-2006-BLG-109Lb,c Jupiter/Saturn Analog Planetary System
Authors:
D. P. Bennett,
S. H. Rhie,
S. Nikolaev,
B. S. Gaudi,
A. Udalski,
A. Gould,
G. W. Christie,
D. Maoz,
S. Dong,
J. McCormick,
M. K. Szymanski,
P. J. Tristram,
B. Macintosh,
K. H. Cook,
M. Kubiak,
G. Pietrzynski,
I. Soszynski,
O. Szewczyk,
K. Ulaczyk,
L. Wyrzykowski,
D. L. DePoy,
C. Han,
S. Kaspi,
C. -U. Lee,
F. Mallia
, et al. (48 additional authors not shown)
Abstract:
We present a new analysis of the Jupiter+Saturn analog system, OGLE-2006-BLG-109Lb,c, which was the first double planet system discovered with the gravitational microlensing method. This is the only multi-planet system discovered by any method with measured masses for the star and both planets. In addition to the signatures of two planets, this event also exhibits a microlensing parallax signature…
▽ More
We present a new analysis of the Jupiter+Saturn analog system, OGLE-2006-BLG-109Lb,c, which was the first double planet system discovered with the gravitational microlensing method. This is the only multi-planet system discovered by any method with measured masses for the star and both planets. In addition to the signatures of two planets, this event also exhibits a microlensing parallax signature and finite source effects that provide a direct measure of the masses of the star and planets, and the expected brightness of the host star is confirmed by Keck AO imaging, yielding masses of M_* = 0.51(+0.05-0.04) M_sun, M_b = 231+-19 M_earth, M_c = 86+-7 M_earth. The Saturn-analog planet in this system had a planetary light curve deviation that lasted for 11 days, and as a result, the effects of the orbital motion are visible in the microlensing light curve. We find that four of the six orbital parameters are tightly constrained and that a fifth parameter, the orbital acceleration, is weakly constrained. No orbital information is available for the Jupiter-analog planet, but its presence helps to constrain the orbital motion of the Saturn-analog planet. Assuming co-planar orbits, we find an orbital eccentricity of eccentricity = 0.15 (+0.17-0.10) and an orbital inclination of i = 64 (+4-7) deg. The 95% confidence level lower limit on the inclination of i > 49 deg. implies that this planetary system can be detected and studied via radial velocity measurements using a telescope of >30m aperture.
△ Less
Submitted 2 June, 2010; v1 submitted 15 November, 2009;
originally announced November 2009.
-
OGLE-2005-BLG-071Lb, the Most Massive M-Dwarf Planetary Companion?
Authors:
Subo Dong,
Andrew Gould,
Andrzej Udalski,
Jay Anderson,
G. W. Christie,
B. S. Gaudi,
M. Jaroszynski,
M. Kubiak,
M. K. Szymanski,
G. Pietrzynski,
I. Soszynski,
O. Szewczyk,
K. Ulaczyk,
L. Wyrzykowski,
D. L. DePoy,
D. B. Fox,
A. Gal-Yam,
C. Han,
S. Lepine,
J. McCormick,
E. Ofek,
B. -G. Park,
R. W. Pogge,
F. Abe,
D. P. Bennett
, et al. (59 additional authors not shown)
Abstract:
We combine all available information to constrain the nature of OGLE-2005-BLG-071Lb, the second planet discovered by microlensing and the first in a high-magnification event. These include photometric and astrometric measurements from Hubble Space Telescope, as well as constraints from higher order effects extracted from the ground-based light curve, such as microlens parallax, planetary orbital…
▽ More
We combine all available information to constrain the nature of OGLE-2005-BLG-071Lb, the second planet discovered by microlensing and the first in a high-magnification event. These include photometric and astrometric measurements from Hubble Space Telescope, as well as constraints from higher order effects extracted from the ground-based light curve, such as microlens parallax, planetary orbital motion and finite-source effects. Our primary analysis leads to the conclusion that the host of Jovian planet OGLE-2005-BLG-071Lb is an M dwarf in the foreground disk with mass M= 0.46 +/- 0.04 Msun, distance D_l = 3.3 +/- 0.4 kpc, and thick-disk kinematics v_LSR ~ 103 km/s. From the best-fit model, the planet has mass M_p = 3.8 +/- 0.4 M_Jup, lies at a projected separation r_perp = 3.6 +/- 0.2 AU from its host and so has an equilibrium temperature of T ~ 55 K, i.e., similar to Neptune. A degenerate model less favored by Δχ^2 = 2.1 (or 2.2, depending on the sign of the impact parameter) gives similar planetary mass M_p = 3.4 +/- 0.4 M_Jup with a smaller projected separation, r_\perp = 2.1 +/- 0.1 AU, and higher equilibrium temperature T ~ 71 K. These results from the primary analysis suggest that OGLE-2005-BLG-071Lb is likely to be the most massive planet yet discovered that is hosted by an M dwarf. However, the formation of such high-mass planetary companions in the outer regions of M-dwarf planetary systems is predicted to be unlikely within the core-accretion scenario. There are a number of caveats to this primary analysis, which assumes (based on real but limited evidence) that the unlensed light coincident with the source is actually due to the lens, that is, the planetary host. However, these caveats could mostly be resolved by a single astrometric measurement a few years after the event.
△ Less
Submitted 2 June, 2009; v1 submitted 9 April, 2008;
originally announced April 2008.
-
Discovery of a Jupiter/Saturn Analog with Gravitational Microlensing
Authors:
B. S. Gaudi,
D. P. Bennett,
A. Udalski,
A. Gould,
G. W. Christie,
D. Maoz,
S. Dong,
J. McCormick,
M. K. Szymanski,
P. J. Tristram,
S. Nikolaev,
B. Paczynski,
M. Kubiak,
G. Pietrzynski,
I. Soszynski,
O. Szewczyk,
K. Ulaczyk,
L. Wyrzykowski,
D. L. DePoy,
C. Han,
S. Kaspi,
C. -U. Lee,
F. Mallia,
T. Natusch,
R. W. Pogge
, et al. (44 additional authors not shown)
Abstract:
Searches for extrasolar planets have uncovered an astonishing diversity of planetary systems, yet the frequency of solar system analogs remains unknown. The gravitational microlensing planet search method is potentially sensitive to multiple-planet systems containing analogs of all the solar system planets except Mercury. We report the detection of a multiple-planet system with microlensing. We…
▽ More
Searches for extrasolar planets have uncovered an astonishing diversity of planetary systems, yet the frequency of solar system analogs remains unknown. The gravitational microlensing planet search method is potentially sensitive to multiple-planet systems containing analogs of all the solar system planets except Mercury. We report the detection of a multiple-planet system with microlensing. We identify two planets with masses of ~0.71 and ~0.27 times the mass of Jupiter and orbital separations of ~2.3 and ~4.6 astronomical units orbiting a primary star of mass ~0.50 solar masses at a distance of ~1.5 kiloparsecs. This system resembles a scaled version of our solar system in that the mass ratio, separation ratio, and equilibrium temperatures of the planets are similar to those of Jupiter and Saturn. These planets could not have been detected with other techniques; their discovery from only six confirmed microlensing planet detections suggests that solar system analogs may be common.
△ Less
Submitted 19 March, 2008; v1 submitted 14 February, 2008;
originally announced February 2008.
