-
HighNESS Conceptual Design Report: Volume I
Authors:
V. Santoro,
O. Abou El Kheir,
D. Acharya,
M. Akhyani,
K. H. Andersen,
J. Barrow,
P. Bentley,
M. Bernasconi,
M. Bertelsen,
Y. Bessler,
A. Bianchi,
G. Brooijmans,
L. Broussard,
T. Brys,
M. Busi,
D. Campi,
A. Chambon,
J. Chen,
V. Czamler,
P. Deen,
D. D. DiJulio,
E. Dian,
L. Draskovits,
K. Dunne,
M. El Barbari
, et al. (65 additional authors not shown)
Abstract:
The European Spallation Source, currently under construction in Lund, Sweden, is a multidisciplinary international laboratory. Once completed to full specifications, it will operate the world's most powerful pulsed neutron source. Supported by a 3 million Euro Research and Innovation Action within the EU Horizon 2020 program, a design study (HighNESS) has been completed to develop a second neutron…
▽ More
The European Spallation Source, currently under construction in Lund, Sweden, is a multidisciplinary international laboratory. Once completed to full specifications, it will operate the world's most powerful pulsed neutron source. Supported by a 3 million Euro Research and Innovation Action within the EU Horizon 2020 program, a design study (HighNESS) has been completed to develop a second neutron source located below the spallation target. Compared to the first source, designed for high cold and thermal brightness, the new source has been optimized to deliver higher intensity, and a shift to longer wavelengths in the spectral regions of cold (CN, 2--20\,Å), very cold (VCN, 10--120\,Å), and ultracold (UCN, ${>}\,{500}$\,Å) neutrons. The second source comprises a large liquid deuterium moderator designed to produce CN and support secondary VCN and UCN sources. Various options have been explored in the proposed designs, aiming for world-leading performance in neutronics. These designs will enable the development of several new instrument concepts and facilitate the implementation of a high-sensitivity neutron-antineutron oscillation experiment (NNBAR). This document serves as the Conceptual Design Report for the HighNESS project, representing its final deliverable.
△ Less
Submitted 28 May, 2024; v1 submitted 29 September, 2023;
originally announced September 2023.
-
A Conceptual Framework for Externally-influenced Agents: An Assisted Reinforcement Learning Review
Authors:
Adam Bignold,
Francisco Cruz,
Matthew E. Taylor,
Tim Brys,
Richard Dazeley,
Peter Vamplew,
Cameron Foale
Abstract:
A long-term goal of reinforcement learning agents is to be able to perform tasks in complex real-world scenarios. The use of external information is one way of scaling agents to more complex problems. However, there is a general lack of collaboration or interoperability between different approaches using external information. In this work, while reviewing externally-influenced methods, we propose…
▽ More
A long-term goal of reinforcement learning agents is to be able to perform tasks in complex real-world scenarios. The use of external information is one way of scaling agents to more complex problems. However, there is a general lack of collaboration or interoperability between different approaches using external information. In this work, while reviewing externally-influenced methods, we propose a conceptual framework and taxonomy for assisted reinforcement learning, aimed at fostering collaboration by classifying and comparing various methods that use external information in the learning process. The proposed taxonomy details the relationship between the external information source and the learner agent, highlighting the process of information decomposition, structure, retention, and how it can be used to influence agent learning. As well as reviewing state-of-the-art methods, we identify current streams of reinforcement learning that use external information in order to improve the agent's performance and its decision-making process. These include heuristic reinforcement learning, interactive reinforcement learning, learning from demonstration, transfer learning, and learning from multiple sources, among others. These streams of reinforcement learning operate with the shared objective of scaffolding the learner agent. Lastly, we discuss further possibilities for future work in the field of assisted reinforcement learning systems.
△ Less
Submitted 19 September, 2021; v1 submitted 3 July, 2020;
originally announced July 2020.
-
Directed Policy Gradient for Safe Reinforcement Learning with Human Advice
Authors:
Hélène Plisnier,
Denis Steckelmacher,
Tim Brys,
Diederik M. Roijers,
Ann Nowé
Abstract:
Many currently deployed Reinforcement Learning agents work in an environment shared with humans, be them co-workers, users or clients. It is desirable that these agents adjust to people's preferences, learn faster thanks to their help, and act safely around them. We argue that most current approaches that learn from human feedback are unsafe: rewarding or punishing the agent a-posteriori cannot im…
▽ More
Many currently deployed Reinforcement Learning agents work in an environment shared with humans, be them co-workers, users or clients. It is desirable that these agents adjust to people's preferences, learn faster thanks to their help, and act safely around them. We argue that most current approaches that learn from human feedback are unsafe: rewarding or punishing the agent a-posteriori cannot immediately prevent it from wrong-doing. In this paper, we extend Policy Gradient to make it robust to external directives, that would otherwise break the fundamentally on-policy nature of Policy Gradient. Our technique, Directed Policy Gradient (DPG), allows a teacher or backup policy to override the agent before it acts undesirably, while allowing the agent to leverage human advice or directives to learn faster. Our experiments demonstrate that DPG makes the agent learn much faster than reward-based approaches, while requiring an order of magnitude less advice.
△ Less
Submitted 13 August, 2018;
originally announced August 2018.
-
Multi-Grid Detector for Neutron Spectroscopy: Results Obtained on Time-of-Flight Spectrometer CNCS
Authors:
M. Anastasopoulos,
R. Bebb,
K. Berry,
J. Birch,
T. Bryś,
J. -C. Buffet,
J. -F. Clergeau,
P. P. Deen,
G. Ehlers,
P. van Esch,
S. M. Everett,
B. Guerard,
R. Hall-Wilton,
K. Herwig,
L. Hultman,
C. Höglund,
I. Iruretagoiena,
F. Issa,
J. Jensen,
A. Khaplanov,
O. Kirstein,
I. Lopez-Higuera,
F. Piscitelli,
L. Robinson,
S. Schmidt
, et al. (1 additional authors not shown)
Abstract:
The Multi-Grid detector technology has evolved from the proof-of-principle and characterisation stages. Here we report on the performance of the Multi-Grid detector, the MG.CNCS prototype, which has been installed and tested at the Cold Neutron Chopper Spectrometer, CNCS at SNS. This has allowed a side-by-side comparison to the performance of $^3$He detectors on an operational instrument. The demo…
▽ More
The Multi-Grid detector technology has evolved from the proof-of-principle and characterisation stages. Here we report on the performance of the Multi-Grid detector, the MG.CNCS prototype, which has been installed and tested at the Cold Neutron Chopper Spectrometer, CNCS at SNS. This has allowed a side-by-side comparison to the performance of $^3$He detectors on an operational instrument. The demonstrator has an active area of 0.2 m$^2$. It is specifically tailored to the specifications of CNCS. The detector was installed in June 2016 and has operated since then, collecting neutron scattering data in parallel to the He-3 detectors of CNCS. In this paper, we present a comprehensive analysis of this data, in particular on instrument energy resolution, rate capability, background and relative efficiency. Stability, gamma-ray and fast neutron sensitivity have also been investigated. The effect of scattering in the detector components has been measured and provides input to comparison for Monte Carlo simulations. All data is presented in comparison to that measured by the $^3$He detectors simultaneously, showing that all features recorded by one detector are also recorded by the other. The energy resolution matches closely. We find that the Multi-Grid is able to match the data collected by $^3$He, and see an indication of a considerable advantage in the count rate capability. Based on these results, we are confident that the Multi-Grid detector will be capable of producing high quality scientific data on chopper spectrometers utilising the unprecedented neutron flux of the ESS.
△ Less
Submitted 3 April, 2017; v1 submitted 10 March, 2017;
originally announced March 2017.
-
The Multi-Blade Boron-10-based Neutron Detector for high intensity Neutron Reflectometry at ESS
Authors:
Francesco Piscitelli,
Francesco Messi,
Michail Anastasopoulos,
Tomasz Bryś,
Faye Chicken,
Eszter Dian,
Janos Fuzi,
Carina Höglund,
Gabor Kiss,
Janos Orban,
Peter Pazmandi,
Linda Robinson,
Laszlo Rosta,
Susann Schmidt,
Dezso Varga,
Tibor Zsiros,
Richard Hall-Wilton
Abstract:
The Multi-Blade is a Boron-10-based gaseous detector introduced to face the challenge arising in neutron reflectometry at pulsed neutron sources. Neutron reflectometers are the most challenging instruments in terms of instantaneous counting rate and spatial resolution. This detector has been designed to cope with the requirements set for the reflectometers at the upcoming European Spallation Sourc…
▽ More
The Multi-Blade is a Boron-10-based gaseous detector introduced to face the challenge arising in neutron reflectometry at pulsed neutron sources. Neutron reflectometers are the most challenging instruments in terms of instantaneous counting rate and spatial resolution. This detector has been designed to cope with the requirements set for the reflectometers at the upcoming European Spallation Source (ESS) in Sweden. Based on previous results obtained at the Institut Laue-Langevin (ILL) in France, an improved demonstrator has been built at ESS and tested at the Budapest Neutron Centre (BNC) in Hungary and at the Source Testing Facility (STF) at the Lund University in Sweden. A detailed description of the detector and the results of the tests are discussed in this manuscript.
△ Less
Submitted 26 January, 2017;
originally announced January 2017.
-
Using PCA to Efficiently Represent State Spaces
Authors:
William Curran,
Tim Brys,
Matthew Taylor,
William Smart
Abstract:
Reinforcement learning algorithms need to deal with the exponential growth of states and actions when exploring optimal control in high-dimensional spaces. This is known as the curse of dimensionality. By projecting the agent's state onto a low-dimensional manifold, we can represent the state space in a smaller and more efficient representation. By using this representation during learning, the ag…
▽ More
Reinforcement learning algorithms need to deal with the exponential growth of states and actions when exploring optimal control in high-dimensional spaces. This is known as the curse of dimensionality. By projecting the agent's state onto a low-dimensional manifold, we can represent the state space in a smaller and more efficient representation. By using this representation during learning, the agent can converge to a good policy much faster. We test this approach in the Mario Benchmarking Domain. When using dimensionality reduction in Mario, learning converges much faster to a good policy. But, there is a critical convergence-performance trade-off. By projecting onto a low-dimensional manifold, we are ignoring important data. In this paper, we explore this trade-off of convergence and performance. We find that learning in as few as 4 dimensions (instead of 9), we can improve performance past learning in the full dimensional space at a faster convergence rate.
△ Less
Submitted 3 June, 2015; v1 submitted 2 May, 2015;
originally announced May 2015.
-
Off-Policy Reward Sha** with Ensembles
Authors:
Anna Harutyunyan,
Tim Brys,
Peter Vrancx,
Ann Nowe
Abstract:
Potential-based reward sha** (PBRS) is an effective and popular technique to speed up reinforcement learning by leveraging domain knowledge. While PBRS is proven to always preserve optimal policies, its effect on learning speed is determined by the quality of its potential function, which, in turn, depends on both the underlying heuristic and the scale. Knowing which heuristic will prove effecti…
▽ More
Potential-based reward sha** (PBRS) is an effective and popular technique to speed up reinforcement learning by leveraging domain knowledge. While PBRS is proven to always preserve optimal policies, its effect on learning speed is determined by the quality of its potential function, which, in turn, depends on both the underlying heuristic and the scale. Knowing which heuristic will prove effective requires testing the options beforehand, and determining the appropriate scale requires tuning, both of which introduce additional sample complexity. We formulate a PBRS framework that reduces learning speed, but does not incur extra sample complexity. For this, we propose to simultaneously learn an ensemble of policies, shaped w.r.t. many heuristics and on a range of scales. The target policy is then obtained by voting. The ensemble needs to be able to efficiently and reliably learn off-policy: requirements fulfilled by the recent Horde architecture, which we take as our basis. We demonstrate empirically that (1) our ensemble policy outperforms both the base policy, and its single-heuristic components, and (2) an ensemble over a general range of scales performs at least as well as one with optimally tuned components.
△ Less
Submitted 23 March, 2015; v1 submitted 11 February, 2015;
originally announced February 2015.
-
Off-Policy Sha** Ensembles in Reinforcement Learning
Authors:
Anna Harutyunyan,
Tim Brys,
Peter Vrancx,
Ann Nowe
Abstract:
Recent advances of gradient temporal-difference methods allow to learn off-policy multiple value functions in parallel with- out sacrificing convergence guarantees or computational efficiency. This opens up new possibilities for sound ensemble techniques in reinforcement learning. In this work we propose learning an ensemble of policies related through potential-based sha** rewards. The ensemble…
▽ More
Recent advances of gradient temporal-difference methods allow to learn off-policy multiple value functions in parallel with- out sacrificing convergence guarantees or computational efficiency. This opens up new possibilities for sound ensemble techniques in reinforcement learning. In this work we propose learning an ensemble of policies related through potential-based sha** rewards. The ensemble induces a combination policy by using a voting mechanism on its components. Learning happens in real time, and we empirically show the combination policy to outperform the individual policies of the ensemble.
△ Less
Submitted 21 May, 2014;
originally announced May 2014.
