-
InSaAF: Incorporating Safety through Accuracy and Fairness | Are LLMs ready for the Indian Legal Domain?
Authors:
Yogesh Tripathi,
Raghav Donakanti,
Sahil Girhepuje,
Ishan Kavathekar,
Bhaskara Hanuma Vedula,
Gokul S Krishnan,
Shreya Goyal,
Anmol Goel,
Balaraman Ravindran,
Ponnurangam Kumaraguru
Abstract:
Recent advancements in language technology and Artificial Intelligence have resulted in numerous Language Models being proposed to perform various tasks in the legal domain ranging from predicting judgments to generating summaries. Despite their immense potential, these models have been proven to learn and exhibit societal biases and make unfair predictions. In this study, we explore the ability o…
▽ More
Recent advancements in language technology and Artificial Intelligence have resulted in numerous Language Models being proposed to perform various tasks in the legal domain ranging from predicting judgments to generating summaries. Despite their immense potential, these models have been proven to learn and exhibit societal biases and make unfair predictions. In this study, we explore the ability of Large Language Models (LLMs) to perform legal tasks in the Indian landscape when social factors are involved. We present a novel metric, $β$-weighted $\textit{Legal Safety Score ($LSS_β$)}$, which encapsulates both the fairness and accuracy aspects of the LLM. We assess LLMs' safety by considering its performance in the $\textit{Binary Statutory Reasoning}$ task and its fairness exhibition with respect to various axes of disparities in the Indian society. Task performance and fairness scores of LLaMA and LLaMA--2 models indicate that the proposed $LSS_β$ metric can effectively determine the readiness of a model for safe usage in the legal sector. We also propose finetuning pipelines, utilising specialised legal datasets, as a potential method to mitigate bias and improve model safety. The finetuning procedures on LLaMA and LLaMA--2 models increase the $LSS_β$, improving their usability in the Indian legal domain. Our code is publicly released.
△ Less
Submitted 17 June, 2024; v1 submitted 16 February, 2024;
originally announced February 2024.
-
Evidence that the AT transition disappears below six dimensions
Authors:
Bharadwaj Vedula,
M. A. Moore,
Auditya Sharma
Abstract:
One of the key predictions of Parisi's broken replica symmetry theory of spin glasses is the existence of a phase transition in an applied field to a state with broken replica symmetry. This transition takes place at the de Almeida-Thouless (AT) line in the $h-T$ plane. We have studied this line in the power-law diluted Heisenberg spin glass in which the probability that two spins separated by a d…
▽ More
One of the key predictions of Parisi's broken replica symmetry theory of spin glasses is the existence of a phase transition in an applied field to a state with broken replica symmetry. This transition takes place at the de Almeida-Thouless (AT) line in the $h-T$ plane. We have studied this line in the power-law diluted Heisenberg spin glass in which the probability that two spins separated by a distance $r$ interact with each other falls as $1/r^{2σ}$. In the presence of a random vector-field of variance $h_r^2$ the phase transition is in the universality class of the Ising spin glass in a field. Tuning $σ$ is equivalent to changing the dimension $d$ of the short-range system, with the relation being $d =2/(2σ-1)$ for $σ< 2/3$. We have found by numerical simulations that $h_{\text{AT}}^2 \sim (2/3 -σ)$ implying that the AT line does not exist below $6$ dimensions and that the Parisi scheme is not appropriate for spin glasses in three dimensions.
△ Less
Submitted 3 July, 2024; v1 submitted 6 February, 2024;
originally announced February 2024.
-
Study of the de Almeida-Thouless (AT) line in the one-dimensional diluted power-law XY spin glass
Authors:
Bharadwaj Vedula,
M. A. Moore,
Auditya Sharma
Abstract:
We study the AT line in the one-dimensional power-law diluted XY spin glass model, in which the probability that two spins separated by a distance $r$ interact with each other, decays as $1/r^{2σ}$. We develop a heat bath algorithm to equilibrate XY spins; using this in conjunction with the standard parallel tempering and overrelaxation sweeps, we carry out large scale Monte Carlo simulations. For…
▽ More
We study the AT line in the one-dimensional power-law diluted XY spin glass model, in which the probability that two spins separated by a distance $r$ interact with each other, decays as $1/r^{2σ}$. We develop a heat bath algorithm to equilibrate XY spins; using this in conjunction with the standard parallel tempering and overrelaxation sweeps, we carry out large scale Monte Carlo simulations. For $σ=0.6$ which is in the mean-field regime, we find clear evidence for an AT line. For $σ= 0.75$, there is evidence from finite size scaling studies for an AT transition but for $σ= 0.85$, the evidence for a transition is non-existent. We have also studied these systems at fixed temperature varying the field and discovered that at both $σ= 0.75$ and at $σ=0.85$ there is evidence of an AT transition! Confusingly, the correlation length and spin glass susceptibility as a function of the field are both entirely consistent with the predictions of the droplet picture and hence the non-existence of an AT line. The evidence from our simulations points to the complete absence of the AT line in dimensions outside the mean-field region and to the correctness of the droplet picture. Previous simulations which suggested there was an AT line can be attributed to the consequences of studying systems which are just too small. The collapse of our data to the droplet scaling form is poor for $σ= 0.75$ and to some extent also for $σ= 0.85$, when the correlation length becomes of the order of the length of the system, due to the existence of excitations which only cost a free energy of $O(1)$, just as envisaged in the TNT picture of the ordered state of spin glasses. However, for the case of $σ= 0.85$ we can provide evidence that for larger system sizes, droplet scaling will prevail even when the correlation length is comparable to the system size.
△ Less
Submitted 24 July, 2023; v1 submitted 9 January, 2023;
originally announced January 2023.
