-
Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methods
Authors:
Ken Trotti,
Samuel A. Cruz Alegría,
Alena Kopaničáková,
Rolf Krause
Abstract:
We propose to train neural networks (NNs) using a novel variant of the ``Additively Preconditioned Trust-region Strategy'' (APTS). The proposed method is based on a parallelizable additive domain decomposition approach applied to the neural network's parameters. Built upon the TR framework, the APTS method ensures global convergence towards a minimizer. Moreover, it eliminates the need for computa…
▽ More
We propose to train neural networks (NNs) using a novel variant of the ``Additively Preconditioned Trust-region Strategy'' (APTS). The proposed method is based on a parallelizable additive domain decomposition approach applied to the neural network's parameters. Built upon the TR framework, the APTS method ensures global convergence towards a minimizer. Moreover, it eliminates the need for computationally expensive hyper-parameter tuning, as the TR algorithm automatically determines the step size in each iteration. We demonstrate the capabilities, strengths, and limitations of the proposed APTS training method by performing a series of numerical experiments. The presented numerical study includes a comparison with widely used training methods such as SGD, Adam, LBFGS, and the standard TR method.
△ Less
Submitted 21 December, 2023;
originally announced December 2023.
-
SAT-Based Algorithms for Regular Graph Pattern Matching
Authors:
Miguel Terra-Neves,
José Amaral,
Alexandre Lemos,
Rui Quintino,
Pedro Resende,
Antonio Alegria
Abstract:
Graph matching is a fundamental problem in pattern recognition, with many applications such as software analysis and computational biology. One well-known type of graph matching problem is graph isomorphism, which consists of deciding if two graphs are identical. Despite its usefulness, the properties that one may check using graph isomorphism are rather limited, since it only allows strict equali…
▽ More
Graph matching is a fundamental problem in pattern recognition, with many applications such as software analysis and computational biology. One well-known type of graph matching problem is graph isomorphism, which consists of deciding if two graphs are identical. Despite its usefulness, the properties that one may check using graph isomorphism are rather limited, since it only allows strict equality checks between two graphs. For example, it does not allow one to check complex structural properties such as if the target graph is an arbitrary length sequence followed by an arbitrary size loop.
We propose a generalization of graph isomorphism that allows one to check such properties through a declarative specification. This specification is given in the form of a Regular Graph Pattern (ReGaP), a special type of graph, inspired by regular expressions, that may contain wildcard nodes that represent arbitrary structures such as variable-sized sequences or subgraphs. We propose a SAT-based algorithm for checking if a target graph matches a given ReGaP. We also propose a preprocessing technique for improving the performance of the algorithm and evaluate it through an extensive experimental evaluation on benchmarks from the CodeSearchNet dataset.
△ Less
Submitted 15 December, 2023;
originally announced December 2023.
-
T5QL: Taming language models for SQL generation
Authors:
Samuel Arcadinho,
David Aparício,
Hugo Veiga,
António Alegria
Abstract:
Automatic SQL generation has been an active research area, aiming at streamlining the access to databases by writing natural language with the given intent instead of writing SQL. Current SOTA methods for semantic parsing depend on LLMs to achieve high predictive accuracy on benchmark datasets. This reduces their applicability, since LLMs requires expensive GPUs. Furthermore, SOTA methods are ungr…
▽ More
Automatic SQL generation has been an active research area, aiming at streamlining the access to databases by writing natural language with the given intent instead of writing SQL. Current SOTA methods for semantic parsing depend on LLMs to achieve high predictive accuracy on benchmark datasets. This reduces their applicability, since LLMs requires expensive GPUs. Furthermore, SOTA methods are ungrounded and thus not guaranteed to always generate valid SQL. Here we propose T5QL, a new SQL generation method that improves the performance in benchmark datasets when using smaller LMs, namely T5-Base, by 13pp when compared against SOTA methods. Additionally, T5QL is guaranteed to always output valid SQL using a context-free grammar to constrain SQL generation. Finally, we show that dividing semantic parsing in two tasks, candidate SQLs generation and candidate re-ranking, is a promising research avenue that can reduce the need for large LMs.
△ Less
Submitted 21 September, 2022;
originally announced September 2022.
-
Duplicated Code Pattern Mining in Visual Programming Languages
Authors:
Miguel Terra-Neves,
João Nadkarni,
Miguel Ventura,
Pedro Resende,
Hugo Veiga,
António Alegria
Abstract:
Visual Programming Languages (VPLs), coupled with the high-level abstractions that are commonplace in visual programming environments, enable users with less technical knowledge to become proficient programmers. However, the lower skill floor required by VPLs also entails that programmers are more likely to not adhere to best practices of software development, producing systems with high technical…
▽ More
Visual Programming Languages (VPLs), coupled with the high-level abstractions that are commonplace in visual programming environments, enable users with less technical knowledge to become proficient programmers. However, the lower skill floor required by VPLs also entails that programmers are more likely to not adhere to best practices of software development, producing systems with high technical debt, and thus poor maintainability. Duplicated code is one important example of such technical debt. In fact, we observed that the amount of duplication in the OutSystems VPL code bases can reach as high as $39\%$.
Duplicated code detection in text-based programming languages is still an active area of research with important implications regarding software maintainability and evolution. However, to the best of our knowledge, the literature on duplicated code detection for VPLs is very limited. We propose a novel and scalable duplicated code pattern mining algorithm that leverages the visual structure of VPLs in order to not only detect duplicated code, but also highlight duplicated code patterns that explain the reported duplication. The performance of the proposed approach is evaluated on a wide range of real-world mobile and web applications developed using OutSystems.
△ Less
Submitted 15 July, 2021;
originally announced July 2021.
-
Identifying Weakly Connected Subsystems in Building Energy Model for Effective Load Estimation in Presence of Parametric Uncertainty
Authors:
Arpan Mukherjee,
Anna Kuechle Szweda,
Andrew Alegria,
Rahul Rai,
Tarunraj Singh
Abstract:
It is necessary to estimate the expected energy usage of a building to determine how to reduce energy usage. The expected energy usage of a building can be reliably simulated using a Building Energy Model (BEM). Many of the numerous input parameters in a BEM are uncertain. To ensure that the building simulation is sufficiently accurate, and to better understand the impact of imprecisions in the in…
▽ More
It is necessary to estimate the expected energy usage of a building to determine how to reduce energy usage. The expected energy usage of a building can be reliably simulated using a Building Energy Model (BEM). Many of the numerous input parameters in a BEM are uncertain. To ensure that the building simulation is sufficiently accurate, and to better understand the impact of imprecisions in the input parameters and calculation methods, it is desirable to quantify uncertainty in the BEM throughout the modeling process. Uncertainty quantification (UQ) typically requires a large number of simulations to produce meaningful data, which, due to the vast number of input parameters and the dynamic nature of building simulation, is computationally expensive. Uncertainty Quantification (UQ) in BEM domain is thus intractable due to the size of the problem and parameters involved and hence it needs an advanced methodology for analysis. The current paper outlines a novel Weakly-Connected-Systems (WCSs) identification-based UQ framework developed to propagate the quantifiable uncertainty in the BEM. The overall approach is demonstrated on the physics-based thermal model of an actual building in Central New York.
△ Less
Submitted 17 April, 2020;
originally announced April 2020.
