-
Reweighted Proximal Pruning for Large-Scale Language Representation
Authors:
Fu-Ming Guo,
Sijia Liu,
Finlay S. Mungall,
Xue Lin,
Yanzhi Wang
Abstract:
Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can create state-of-the-art results on a wide range of downstream tasks. Along with continuous significant performance improvement, the size and complexity of these pre-trained neural models continue to increase rapidly. Is…
▽ More
Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can create state-of-the-art results on a wide range of downstream tasks. Along with continuous significant performance improvement, the size and complexity of these pre-trained neural models continue to increase rapidly. Is it possible to compress these large-scale language representation models? How will the pruned language representation affect the downstream multi-task transfer learning objectives? In this paper, we propose Reweighted Proximal Pruning (RPP), a new pruning method specifically designed for a large-scale language representation model. Through experiments on SQuAD and the GLUE benchmark suite, we show that proximal pruned BERT keeps high accuracy for both the pre-training task and the downstream multiple fine-tuning tasks at high prune ratio. RPP provides a new perspective to help us analyze what large-scale language representation might learn. Additionally, RPP makes it possible to deploy a large state-of-the-art language representation model such as BERT on a series of distinct devices (e.g., online servers, mobile phones, and edge devices).
△ Less
Submitted 22 December, 2019; v1 submitted 27 September, 2019;
originally announced September 2019.
-
Meeting the Cool Neighbors X: Ultracool dwarfs from the 2MASS All-Sky Data Release
Authors:
I. Neill Reid,
Kelle L. Cruz,
J. Davy Kirkpatrick,
Peter R. Allen,
F. Mungall,
James Liebert,
Patrick Lowrance,
Anne Sweet
Abstract:
Using data from the 2MASS All-Sky Point Source Catalogue, we have extended our census of nearby ultracool dwarfs to cover the full celestial sphere above Galactic latitute 15 degrees. Starting with an initial catalogue of 2,139,484 sources, we have winnowed the sample to 467 candidate late-type M or L dwarfs within 20 parsecs of the Sun. Fifty-four of those sources already have spectroscopic obs…
▽ More
Using data from the 2MASS All-Sky Point Source Catalogue, we have extended our census of nearby ultracool dwarfs to cover the full celestial sphere above Galactic latitute 15 degrees. Starting with an initial catalogue of 2,139,484 sources, we have winnowed the sample to 467 candidate late-type M or L dwarfs within 20 parsecs of the Sun. Fifty-four of those sources already have spectroscopic observations confirming them as late-type dwarfs. We present optical spectroscopy of 376 of the remaining 413 sources, and identify 44 as ultracool dwarfs with spectroscopic distances less than 20 parsecs. Twenty-five of the 37 sources that lack optical data have near-infrared spectroscopy. Combining the present sample with our previous results and data from the literature, we catalogue 94 L dwarf systems within 20 parsecs. We discuss the distribution of activity, as measured by H-alpha emission, in this volume-limited sample. We have coupled the present ultracool catalogue with data for stars in the northern 8-parsec sample and recent (incomplete) statistics for T dwarfs to provide a snapshot of the current 20-parsec census as a function of spectral type.
△ Less
Submitted 20 June, 2008;
originally announced June 2008.
-
Meeting the Cool Neighbors VII: Spectroscopy of faint, red NLTT dwarfs
Authors:
I. N. Reid,
K. L. Cruz,
P. Allen,
F. Mungall,
D. Kilkenny,
J. Liebert,
S. L. Hawley,
O. J. Fraser,
K. R. Covey,
P. Lowrance
Abstract:
We present low-resolution optical spectroscopy and BVRI photometry of 453 candidate nearby stars drawn from the NLTT proper motion catalogue. The stars were selected based on optical/near-infrared colours, derived by combining the NLTT photographic data with photometry from the 2MASS Second Incremental Data Release. Based on the derived photometric and spectroscopic parallaxes, we identify 111 s…
▽ More
We present low-resolution optical spectroscopy and BVRI photometry of 453 candidate nearby stars drawn from the NLTT proper motion catalogue. The stars were selected based on optical/near-infrared colours, derived by combining the NLTT photographic data with photometry from the 2MASS Second Incremental Data Release. Based on the derived photometric and spectroscopic parallaxes, we identify 111 stars as lying within 20 parsecs of the Sun, including 9 stars with formal distance estimates of less than 10 parsecs. A further 53 stars have distance estimates within 1-sigma of our 20-parsec limit. Almost all of those stars are additions to the nearby star census. In total, our NLTT-based survey has so far identified 496 stars likely to be within 20 parsecs, of which 195 are additions to nearby-star catalogues. Most of the newly-identified nearby stars have spectral types between M4 and M8.
△ Less
Submitted 21 August, 2003;
originally announced August 2003.
