-
Improving Dictionary Learning with Gated Sparse Autoencoders
Authors:
Senthooran Rajamanoharan,
Arthur Conmy,
Lewis Smith,
Tom Lieberum,
Vikrant Varma,
János Kramár,
Rohin Shah,
Neel Nanda
Abstract:
Recent work has found that sparse autoencoders (SAEs) are an effective technique for unsupervised discovery of interpretable features in language models' (LMs) activations, by finding sparse, linear reconstructions of LM activations. We introduce the Gated Sparse Autoencoder (Gated SAE), which achieves a Pareto improvement over training with prevailing methods. In SAEs, the L1 penalty used to enco…
▽ More
Recent work has found that sparse autoencoders (SAEs) are an effective technique for unsupervised discovery of interpretable features in language models' (LMs) activations, by finding sparse, linear reconstructions of LM activations. We introduce the Gated Sparse Autoencoder (Gated SAE), which achieves a Pareto improvement over training with prevailing methods. In SAEs, the L1 penalty used to encourage sparsity introduces many undesirable biases, such as shrinkage -- systematic underestimation of feature activations. The key insight of Gated SAEs is to separate the functionality of (a) determining which directions to use and (b) estimating the magnitudes of those directions: this enables us to apply the L1 penalty only to the former, limiting the scope of undesirable side effects. Through training SAEs on LMs of up to 7B parameters we find that, in typical hyper-parameter ranges, Gated SAEs solve shrinkage, are similarly interpretable, and require half as many firing features to achieve comparable reconstruction fidelity.
△ Less
Submitted 30 April, 2024; v1 submitted 24 April, 2024;
originally announced April 2024.
-
Bogomol'nyi Bounds for Gravitational Cosmic Strings
Authors:
Anne-Christine Davis,
Senthooran Rajamanoharan
Abstract:
We present a new method for finding lower bounds on the energy of topological cosmic string solutions in gravitational field theories. This new method produces bounds that are valid over the entire space of solutions, unlike the traditional approach, where the bounds obtained are only valid for cylindrically symmetric solutions. This method is shown to be a generalisation of the well-known Bogom…
▽ More
We present a new method for finding lower bounds on the energy of topological cosmic string solutions in gravitational field theories. This new method produces bounds that are valid over the entire space of solutions, unlike the traditional approach, where the bounds obtained are only valid for cylindrically symmetric solutions. This method is shown to be a generalisation of the well-known Bogomol'nyi procedure for non-gravitational theories and as such, it can be used to find gravitational Bogomol'nyi bounds for models wherever the traditional Bogomol'nyi procedure can be applied in the non-gravitational limit. Furthermore, this method yields Bogomol'nyi equations that do not rule out the existence of asymmetric bound-saturating solutions.
△ Less
Submitted 8 January, 2009; v1 submitted 25 September, 2008;
originally announced September 2008.
-
Cusps on cosmic superstrings with junctions
Authors:
Anne-Christine Davis,
William Nelson,
Senthooran Rajamanoharan,
Mairi Sakellariadou
Abstract:
The existence of cusps on non-periodic strings ending on D-branes is demonstrated and the conditions, for which such cusps are generic, are derived. The dynamics of F-, D-string and FD-string junctions are investigated. It is shown that pairs of FD-string junctions, such as would form after intercommutations of F- and D-strings, generically contain cusps. This new feature of cosmic superstrings…
▽ More
The existence of cusps on non-periodic strings ending on D-branes is demonstrated and the conditions, for which such cusps are generic, are derived. The dynamics of F-, D-string and FD-string junctions are investigated. It is shown that pairs of FD-string junctions, such as would form after intercommutations of F- and D-strings, generically contain cusps. This new feature of cosmic superstrings opens up the possibility of extra channels of energy loss from a string network. The phenomenology of cusps on such cosmic superstring networks is compared to that of cusps formed on networks of their field theory analogues, the standard cosmic strings.
△ Less
Submitted 12 September, 2008;
originally announced September 2008.
