-
Legal-HNet: Mixing Legal Long-Context Tokens with Hartley Transform
Authors:
Daniele Giofré,
Sneha Ghantasala
Abstract:
Since its introduction, the transformers architecture has seen great adoption in NLP applications, but it also has limitations. Although the self-attention mechanism allows for generating very rich representations of the input text, its effectiveness may be limited in specialized domains such as legal, where, for example, language models often have to process very long texts. In this paper, we exp…
▽ More
Since its introduction, the transformers architecture has seen great adoption in NLP applications, but it also has limitations. Although the self-attention mechanism allows for generating very rich representations of the input text, its effectiveness may be limited in specialized domains such as legal, where, for example, language models often have to process very long texts. In this paper, we explore alternatives to replace the attention-based layers with simpler token-mixing mechanisms: Hartley and Fourier transforms. Using these non-parametric techniques, we train models with long input documents from scratch in the legal domain setting. We also introduce a new hybrid Seq2Seq architecture, a no-attention-based encoder connected with an attention-based decoder, which performs quite well on existing summarization tasks with much less compute and memory requirements. We believe that similar, if not better performance, as in the case of long correlations of abstractive text summarization tasks, can be achieved by adopting these simpler infrastructures. This not only makes training models from scratch accessible to more people, but also contributes to the reduction of the carbon footprint during training.
△ Less
Submitted 8 November, 2023;
originally announced November 2023.
-
Reliability of the g factor over time in Italian INVALSI data (2010-2022): What can achievement-g tell us about the Flynn effect?
Authors:
Jakob Pietschnig,
Sandra Oberleiter,
Enrico Toffalini,
David Giofre
Abstract:
Generational intelligence test score gains over large parts of the 20th century have been observed to be negatively associated with psychometric g. Recent reports about changes in the cross-temporal IQ trajectory suggest that ability differentiation may be responsible for both changes in g as well as increasingly (sub)domain specific and inconsistent trajectories. Schooling is considered to be a m…
▽ More
Generational intelligence test score gains over large parts of the 20th century have been observed to be negatively associated with psychometric g. Recent reports about changes in the cross-temporal IQ trajectory suggest that ability differentiation may be responsible for both changes in g as well as increasingly (sub)domain specific and inconsistent trajectories. Schooling is considered to be a main candidate cause for the Flynn effect, which suggests that school achievement might be expected to show similar cross-temporal developments. In the present study, we investigated evidence for cross-temporal changes in achievement-based g in a formal large-scale student assessment in Italy (i.e., the INVALSI assessment; N = 1,900,000). Based on data of four school grades (i.e., grades 2, 5, 8, and 10) over 13 years (2010-2022), we observed little evidence for changes in achievement g in general. However, cross-temporal trajectories were differentiated according to school grade, indicating cross-temporal g decreases for lower grade students whilst changes for higher grade students were positive. These findings may be interpreted as tentative evidence for age-dependent achievement-g differentiation. The presently observed achievement g trajectory appears to be consistent with recently observed evidence for a potential stagnation or reversal of cognitive test score gains.
△ Less
Submitted 22 July, 2023;
originally announced July 2023.
-
Cognitive characteristics of intellectually gifted children with a diagnosis of ADHD
Authors:
Cesare Cornoldi,
David Giofre,
Enrico Toffalini
Abstract:
Some children may be intellectually gifted, and yet experience behavioral and academic difficulties. We examined 82 twice exceptional children (2eADHD), having an excellent General Ability Index (GAI) derived from the Wechsler Intelligence Scale for Children-IV (GAI >= 125), and a diagnosis of Attention Deficit and Hyperactivity Disorder (ADHD). They accounted for 8.8% of a large sample of childre…
▽ More
Some children may be intellectually gifted, and yet experience behavioral and academic difficulties. We examined 82 twice exceptional children (2eADHD), having an excellent General Ability Index (GAI) derived from the Wechsler Intelligence Scale for Children-IV (GAI >= 125), and a diagnosis of Attention Deficit and Hyperactivity Disorder (ADHD). They accounted for 8.8% of a large sample of children with ADHD, which is twice as high as the proportion of intellectually gifted children in a typical population. This over-representation does not reflect a misdiagnosis of ADHD, as these children showed the typical features predicted on the grounds of data regarding the ADHD sample, including lower scores in working memory and processing speed measures, combined with the inclusion criteria for giftedness. Based on information concerning intellectually gifted children with either a Specific Learning Disorder (SLD) or typical development, we observed that these characteristics of intelligence are similar to those seen in SLD, but not in typical development, irrespective of whether 2e-ADHD children had a comorbid SLD.
△ Less
Submitted 21 February, 2023;
originally announced February 2023.
-
BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?
Authors:
Joel Niklaus,
Daniele Giofré
Abstract:
Pretrained transformer models have achieved state-of-the-art results in many tasks and benchmarks recently. Many state-of-the-art Language Models (LMs), however, do not scale well above the threshold of 512 input tokens. In specialized domains though (such as legal, scientific or biomedical), models often need to process very long text (sometimes well above 10000 tokens). Even though many efficien…
▽ More
Pretrained transformer models have achieved state-of-the-art results in many tasks and benchmarks recently. Many state-of-the-art Language Models (LMs), however, do not scale well above the threshold of 512 input tokens. In specialized domains though (such as legal, scientific or biomedical), models often need to process very long text (sometimes well above 10000 tokens). Even though many efficient transformers have been proposed (such as Longformer, BigBird or FNet), so far, only very few such efficient models are available for specialized domains. Additionally, since the pretraining process is extremely costly in general - but even more so as the sequence length increases - it is often only in reach of large research labs. One way of making pretraining cheaper is the Replaced Token Detection (RTD) task, by providing more signal during training, since the loss can be computed over all tokens. In this work, we train Longformer models with the efficient RTD task on legal data to showcase that pretraining efficient LMs is possible using much less compute. We evaluate the trained models on challenging summarization tasks requiring the model to summarize long texts to show to what extent the models can achieve good performance on downstream tasks. We find that both the small and base models outperform their baselines on the in-domain BillSum and out-of-domain PubMed tasks in their respective parameter range. We publish our code and models for research purposes.
△ Less
Submitted 30 November, 2022;
originally announced November 2022.
-
Structure of Working Memory in Children From 3 to 8 Years Old
Authors:
Barbara Carretti,
David Giofre,
Enrico Toffalini,
Cesare Cornoldi,
Massimiliano Pastore,
Silvia Lanfranchi
Abstract:
Several models of working memory (WM) have been proposed in the literature. Most of the research on the architecture of WM is based on adults or older children, but less is known about younger children. In this study, we tested various models of WM on a sample of 739 Italian children from 3 to 8 years old. Participants were assessed with 12 WM tasks, systematically varying the modality and level o…
▽ More
Several models of working memory (WM) have been proposed in the literature. Most of the research on the architecture of WM is based on adults or older children, but less is known about younger children. In this study, we tested various models of WM on a sample of 739 Italian children from 3 to 8 years old. Participants were assessed with 12 WM tasks, systematically varying the modality and level of executive control required (based on the number of activities to be performed at once: retention alone, ignoring distractors, and dealing with dual tasks). We examined younger children, n = 501, Mage = 56.8 months (SD = 6.4, 48% males) and older children, n = 238, Mage = 80.0 months (SD = 9.0, 58% males) separately using multigroup confirmatory factor analyses. A Bayesian analytical approach was adopted. Our results suggested that a four-factor model distinguishing between verbal, visual, spatial-simultaneous, and spatial-sequential components of WM achieved the best fit. Overall, the WM structure was very similar in the two groups. We further explored this result with an additional model with a central executive factor loaded on high-control tasks only, and found evidence for the presence of an executive control component. The contribution of this factor in terms of explained variance was only modest, however. Our findings demonstrate that it is important to distinguish between WM components in young children.
△ Less
Submitted 12 October, 2022;
originally announced October 2022.
-
The differential role of verbal and visuospatial working memory in mathematics and reading
Authors:
David Giofrè,
Enrica Donolato,
Irene C. Mammarella
Abstract:
Objectives: Several studies have focused on the role of working memory (WM) in predicting mathematical and reading literacy. Alternative models of WM have been proposed and a modality-dependent model of WM, distinguishing between verbal and visuospatial WM modalities, has been advanced. In addition, the relationship between verbal and visuospatial WM and academic achievement has not been extensive…
▽ More
Objectives: Several studies have focused on the role of working memory (WM) in predicting mathematical and reading literacy. Alternative models of WM have been proposed and a modality-dependent model of WM, distinguishing between verbal and visuospatial WM modalities, has been advanced. In addition, the relationship between verbal and visuospatial WM and academic achievement has not been extensively and consistently studied, especially when it comes to distinguishing between mathematical and reading tasks. Method: In the present study, we tested a large group of middle school children in several measures of WM, and in mathematical and reading tasks. Results: Confirmatory factor analyses showed that verbal and visuospatial WM can be differentiated and that these factors have a different predictive power in explaining unique portions of variance in reading and mathematics. Conclusions: Our findings point to the importance of distinguishing between WM modalities in evaluating the relationship between mathematics and reading.
△ Less
Submitted 1 January, 2022;
originally announced January 2022.
-
Identifying the preschool home learning experiences that predict early number skills: Evidence from a longitudinal study
Authors:
Elena Soto-Calvo,
Fiona R. Simmons,
Anne-Marie Adams,
Hannah N. Francis,
Hannah Patel,
David Giofrè
Abstract:
This study examines the longitudinal relationships between home learning experiences and early number skills. The counting, number transcoding and calculation skills of 274 children were assessed in the penultimate term of preschool (Mage=4:0). Prior to these assessments, parents completed questionnaires that surveyed the frequency of the children's home learning experiences. Three types of experi…
▽ More
This study examines the longitudinal relationships between home learning experiences and early number skills. The counting, number transcoding and calculation skills of 274 children were assessed in the penultimate term of preschool (Mage=4:0). Prior to these assessments, parents completed questionnaires that surveyed the frequency of the children's home learning experiences. Three types of experiences were indexed: code-focused home literacy experiences that focus on the phonological and orthographic features of language, meaning-focused home literacy experiences that focus on sharing the meaning of language and text, and home number experiences. The children's language abilities (phonological awareness and vocabulary) and nonverbal abilities (inhibitory control and nonverbal reasoning) were assessed in the final term of preschool (Mage=4:3). Their number skills were reassessed in the final term of the first year of primary school (Mage=5:3). Home letter-sound interaction experiences (interactive code-focused literacy experiences) had significant longitudinal relationships with counting and number transcoding that were independent of language and nonverbal abilities. The relationship between letter-sound interaction experiences and later counting was also independent of the autoregressive influence of baseline counting ability. We extend previous findings by demonstrating that interactive code-focused home literacy experiences in the preschool period predict growth in counting skills even when a broad range of language and cognitive abilities are controlled.
△ Less
Submitted 31 December, 2021;
originally announced January 2022.
-
Decoding gender differences: Intellectual profiles of children with specific learning disabilities
Authors:
David Giofrè,
Katie Allen,
Enrico Toffalini,
Irene C. Mammarella,
Sara Caviola
Abstract:
There has been a significant amount of debate around gender differences in intellectual functioning, however, most of this research concerns typically develo** populations and lacks research into atypically develo** populations and those with specific learning disabilities (SLD). To address this, we examined performance on the WISC-IV in children with SLDs (N=1238, N female= 539, Age range = 7…
▽ More
There has been a significant amount of debate around gender differences in intellectual functioning, however, most of this research concerns typically develo** populations and lacks research into atypically develo** populations and those with specific learning disabilities (SLD). To address this, we examined performance on the WISC-IV in children with SLDs (N=1238, N female= 539, Age range = 7-16 years). We further divided the sample into those with specific deficits in reading, mathematics, and those with mixed disorder. Results indicate that gender predicts significant differences in the working memory index and processing speed index only, indicating a small but significant female superiority. Results also show different profiles for the different disorders investigated, with some gender differences emerging. The most prominent gender difference appears to be in the coding subtest indicating a female advantage, particularly in those with SLDs with mathematical difficulties. We discuss the theoretical and clinical implications of the findings.
△ Less
Submitted 23 December, 2021;
originally announced December 2021.
-
A population level analysis of the gender gap in mathematics: Results on over 13 million children using the INVALSI dataset
Authors:
David Giofrè,
Cesare Cornoldi,
Angela Martini,
Enrico Toffalini
Abstract:
Whether males outperform females in mathematics is still debated. Such a gender gap varies across countries, but the determinants of the differences are unclear and could be produced by heterogeneity in the instructional systems or cultures and may vary across school grades. To clarify this issue, we took advantage of the INVALSI dataset, that offered over 13 million observations covering one sing…
▽ More
Whether males outperform females in mathematics is still debated. Such a gender gap varies across countries, but the determinants of the differences are unclear and could be produced by heterogeneity in the instructional systems or cultures and may vary across school grades. To clarify this issue, we took advantage of the INVALSI dataset, that offered over 13 million observations covering one single instructional system (i.e., the Italian system) in grades 2, 5, and 8, in the period 2010-2018. Results showed that males outperformed females in mathematics (and vice versa in reading), with gaps widening from the 2nd through to the 8th grade. The gender gap in mathematics was larger in the richer northern Italian regions (also characterized by greater gender equality) than in southern regions. This was not explained by average performance or fully accounted for by economic factors. No such north-south difference of the gap emerged in reading. Results are discussed with reference to the literature showing that the gender gap varies across world regions.
△ Less
Submitted 21 December, 2021;
originally announced December 2021.
-
Anxiety profiles and protective factors: A latent profile analysis in children
Authors:
Irene C. Mammarella,
Enrica Donolato,
Sara Caviola,
David Giofrè
Abstract:
The current study investigated the presence of different anxiety profiles in schoolchildren in order to understand whether Mathematics and Test Anxiety are a manifestation of a general form of anxiety, or the expression of specific forms of anxiety. Moreover, we also examined the influence of personal protective factors. The results of a latent profile analysis, conducted on 664 children attending…
▽ More
The current study investigated the presence of different anxiety profiles in schoolchildren in order to understand whether Mathematics and Test Anxiety are a manifestation of a general form of anxiety, or the expression of specific forms of anxiety. Moreover, we also examined the influence of personal protective factors. The results of a latent profile analysis, conducted on 664 children attending grades 3 to 6, clearly identified three different profiles distinguished on the basis of the level of general, test and mathematics anxiety. Protective factors, such as self concept and resilience, were differently related to anxiety: The former was clearly lower when the risk profile was higher, whereas students were able to maintain a certain level of resilience up to an average risk of develo** forms of anxiety. The implications of these findings may lead to the development of specific intervention programs aimed at reducing students' anxiety and fostering self concept and resilience.
△ Less
Submitted 17 December, 2021;
originally announced December 2021.
-
Ab initio Modelling of the Early Stages of Precipitation in Al-6000 Alloys
Authors:
Daniele Giofré,
Till Junge,
W. A. Curtin,
Michele Ceriotti
Abstract:
Age hardening induced by the formation of (semi)-coherent precipitate phases is crucial for the processing and final properties of the widely used Al-6000 alloys. Early stages of precipitation are particularly important from the fundamental and technological side, but are still far from being fully understood. Here, an analysis of the energetics of nanometric precipitates of the meta-stable $β''$…
▽ More
Age hardening induced by the formation of (semi)-coherent precipitate phases is crucial for the processing and final properties of the widely used Al-6000 alloys. Early stages of precipitation are particularly important from the fundamental and technological side, but are still far from being fully understood. Here, an analysis of the energetics of nanometric precipitates of the meta-stable $β''$ phases is performed, identifying the bulk, elastic strain and interface energies that contribute to the stability of a nucleating cluster. Results show that needle-shape precipitates are unstable to growth even at the smallest size $β''$ formula unit, i.e. there is no energy barrier to growth. The small differences between different compositions points toward the need for the study of possible precipitate/matrix interface reconstruction. A classical semi-quantitative nucleation theory approach including elastic strain energy captures the trends in precipitate energy versus size and composition. This validates the use of mesoscale models to assess stability and interactions of precipitates. Studies of smaller 3d clusters also show stability relative to the solid solution state, indicating that the early stages of precipitation may be diffusion-limited. Overall, these results demonstrate the important interplay among composition-dependent bulk, interface, and elastic strain energies in determining nanoscale precipitate stability and growth.
△ Less
Submitted 25 August, 2017;
originally announced August 2017.
-
Electronic transport in BN-substituted bilayer graphene nano-junctions
Authors:
Daniele Giofré,
Davide Ceresoli,
Mario I. Trioni
Abstract:
We investigated a suspended bilayer graphene where the bottom (top) layer is doped by boron (nitrogen) substitutional atoms by using Density Functional Theory (DFT) calculations. We found that at high dopant concentration (one B-N pair every 32 C atoms) the electronic structure of the bilayer does not depend on the B-N distance but on the relative occupation of the bilayer graphene sub-lattices by…
▽ More
We investigated a suspended bilayer graphene where the bottom (top) layer is doped by boron (nitrogen) substitutional atoms by using Density Functional Theory (DFT) calculations. We found that at high dopant concentration (one B-N pair every 32 C atoms) the electronic structure of the bilayer does not depend on the B-N distance but on the relative occupation of the bilayer graphene sub-lattices by B and N. We found that a large built in electric field is established between layers, giving rise to an energy gap. We further investigated the transport properties and found that intra-layer electron current is weakly influenced by the presence of these dopants while the inter-layer current is significantly enhanced for biases allowing the energy alignment of N and B states. This effect leads to current rectification in asymmetric junctions.
△ Less
Submitted 15 January, 2014;
originally announced January 2014.
