-
Analyzing the Performance of Large Language Models on Code Summarization
Authors:
Rajarshi Haldar,
Julia Hockenmaier
Abstract:
Large language models (LLMs) such as Llama 2 perform very well on tasks that involve both natural language and source code, particularly code summarization and code generation. We show that for the task of code summarization, the performance of these models on individual examples often depends on the amount of (subword) token overlap between the code and the corresponding reference natural languag…
▽ More
Large language models (LLMs) such as Llama 2 perform very well on tasks that involve both natural language and source code, particularly code summarization and code generation. We show that for the task of code summarization, the performance of these models on individual examples often depends on the amount of (subword) token overlap between the code and the corresponding reference natural language descriptions in the dataset. This token overlap arises because the reference descriptions in standard datasets (corresponding to docstrings in large code bases) are often highly similar to the names of the functions they describe. We also show that this token overlap occurs largely in the function names of the code and compare the relative performance of these models after removing function names versus removing code structure. We also show that using multiple evaluation metrics like BLEU and BERTScore gives us very little additional insight since these metrics are highly correlated with each other.
△ Less
Submitted 10 April, 2024;
originally announced April 2024.
-
Effect of Ambient-Intrinsic Dimension Gap on Adversarial Vulnerability
Authors:
Rajdeep Haldar,
Yue Xing,
Qifan Song
Abstract:
The existence of adversarial attacks on machine learning models imperceptible to a human is still quite a mystery from a theoretical perspective. In this work, we introduce two notions of adversarial attacks: natural or on-manifold attacks, which are perceptible by a human/oracle, and unnatural or off-manifold attacks, which are not. We argue that the existence of the off-manifold attacks is a nat…
▽ More
The existence of adversarial attacks on machine learning models imperceptible to a human is still quite a mystery from a theoretical perspective. In this work, we introduce two notions of adversarial attacks: natural or on-manifold attacks, which are perceptible by a human/oracle, and unnatural or off-manifold attacks, which are not. We argue that the existence of the off-manifold attacks is a natural consequence of the dimension gap between the intrinsic and ambient dimensions of the data. For 2-layer ReLU networks, we prove that even though the dimension gap does not affect generalization performance on samples drawn from the observed data space, it makes the clean-trained model more vulnerable to adversarial perturbations in the off-manifold direction of the data space. Our main results provide an explicit relationship between the $\ell_2,\ell_{\infty}$ attack strength of the on/off-manifold attack and the dimension gap.
△ Less
Submitted 23 March, 2024; v1 submitted 6 March, 2024;
originally announced March 2024.
-
On Neural Network approximation of ideal adversarial attack and convergence of adversarial training
Authors:
Rajdeep Haldar,
Qifan Song
Abstract:
Adversarial attacks are usually expressed in terms of a gradient-based operation on the input data and model, this results in heavy computations every time an attack is generated. In this work, we solidify the idea of representing adversarial attacks as a trainable function, without further gradient computation. We first motivate that the theoretical best attacks, under proper conditions, can be r…
▽ More
Adversarial attacks are usually expressed in terms of a gradient-based operation on the input data and model, this results in heavy computations every time an attack is generated. In this work, we solidify the idea of representing adversarial attacks as a trainable function, without further gradient computation. We first motivate that the theoretical best attacks, under proper conditions, can be represented as smooth piece-wise functions (piece-wise Hölder functions). Then we obtain an approximation result of such functions by a neural network. Subsequently, we emulate the ideal attack process by a neural network and reduce the adversarial training to a mathematical game between an attack network and a training model (a defense network). We also obtain convergence rates of adversarial loss in terms of the sample size $n$ for adversarial training in such a setting.
△ Less
Submitted 29 July, 2023;
originally announced July 2023.
-
Steering of Quantum Walks through Coherent Control of High-dimensional Bi-photon Quantum Frequency Combs with Tunable State Entropies
Authors:
Raktim Haldar,
Robert Johanning,
Philip Rübeling,
Anahita Khodadad Kashi,
Thomas Bækkegaard,
Surajit Bose,
Nikolaj Thomas Zinner,
Michael Kues
Abstract:
Quantum walks are central to a wide range of applications such as quantum search, quantum information processing, and entanglement transport. Gaining control over the duration and the direction of quantum walks (QWs) is crucial to implementing dedicated processing. However, in current systems, it is cumbersome to achieve in a scalable format. High-dimensional quantum states, encoded in the photons…
▽ More
Quantum walks are central to a wide range of applications such as quantum search, quantum information processing, and entanglement transport. Gaining control over the duration and the direction of quantum walks (QWs) is crucial to implementing dedicated processing. However, in current systems, it is cumbersome to achieve in a scalable format. High-dimensional quantum states, encoded in the photons' frequency degree of freedom in on-chip devices are great assets for the scalable generation and reliable manipulation of large-scale complex quantum systems. These states, viz. quantum frequency combs (QFCs) accommodating huge information in a single spatial mode, are intrinsically noise tolerant, and suitable for transmission through optical fibers, thereby promising to revolutionize quantum technologies. Existing literature aimed to generate maximally entangled QFCs excited from continuous-wave lasers either from nonlinear microcavities or from waveguides with the help of filter arrays. QWs with flexible depth/duration have been lately demonstrated from such QFCs. Here, instead of maximally-entangled QFCs, we generate high-dimensional quantum photonic states with tunable entropies from periodically poled lithium niobate waveguides by exploiting a novel pulsed excitation and filtering scheme. We confirm the generation of QFCs with normalized entropies from $\sim 0.35$ to $1$ by performing quantum tomography with high fidelities. These states can be an excellent testbed for several quantum computation and communication protocols in nonideal scenarios and enable artificial neural networks to classify unknown quantum states. Further, we experimentally demonstrate the steering and coherent control of the directionality of QWs initiated from such QFCs with tunable entropies. Our findings offer a new control mechanism for QWs as well as novel modification means for joint probability distributions.
△ Less
Submitted 12 October, 2022;
originally announced October 2022.
-
Fully on-chip photonic turnkey quantum source for entangled qubit/qudit state generation
Authors:
Hatam Mahmudlu,
Robert Johanning,
Anahita Khodadad Kashi,
Albert van Rees,
Jörn P. Ep**,
Raktim Haldar,
Klaus-J. Boller,
Michael Kues
Abstract:
Integrated photonics has recently become a leading platform for the realization and processing of optical entangled quantum states in compact, robust and scalable chip formats with applications in long-distance quantum-secured communication, quantum-accelerated information processing and non-classical metrology. However, the quantum light sources developed so far have relied on external bulky exci…
▽ More
Integrated photonics has recently become a leading platform for the realization and processing of optical entangled quantum states in compact, robust and scalable chip formats with applications in long-distance quantum-secured communication, quantum-accelerated information processing and non-classical metrology. However, the quantum light sources developed so far have relied on external bulky excitation lasers making them impractical, not reproducible prototype devices, hindering scalability and the transfer out of the lab into real-world applications. Here we demonstrate a fully integrated quantum light source, which overcomes these challenges through the combined integration of a laser cavity, a highly efficient tunable noise suppression filter ($> 55$ dB) exploiting the Vernier effect and a nonlinear microring for entangled photon pair generation through spontaneous four-wave mixing. The hybrid quantum source employs an electrically-pumped InP gain section and a Si$_3$N$_4$ low-loss microring filter system, and demonstrates high performance parameters, i.e., a pair emission over four resonant modes in the telecom band (bandwidth $\sim 1$ THz), and a remarkable pair detection rate of $\sim 620$ Hz at a high coincidence-to-accidental ratio of $\sim 80$. The source directly creates high-dimensional frequency-bin entangled quantum states (qubits/qudits), verified by quantum interference measurements with visibilities up to $96\%$ (violating Bell-inequality) and by density matrix reconstruction through state tomography showing fidelities of up to $99\%$. Our approach, leveraging a hybrid photonic platform, enables commercial-viable, low-cost, compact, light-weight, and field-deployable entangled quantum sources, quintessential for practical, out-of-lab applications, e.g., in quantum processors and quantum satellite communications systems.
△ Less
Submitted 17 June, 2022;
originally announced June 2022.
-
A Multi-Perspective Architecture for Semantic Code Search
Authors:
Rajarshi Haldar,
Lingfei Wu,
**jun Xiong,
Julia Hockenmaier
Abstract:
The ability to match pieces of code to their corresponding natural language descriptions and vice versa is fundamental for natural language search interfaces to software repositories. In this paper, we propose a novel multi-perspective cross-lingual neural framework for code--text matching, inspired in part by a previous model for monolingual text-to-text matching, to capture both global and local…
▽ More
The ability to match pieces of code to their corresponding natural language descriptions and vice versa is fundamental for natural language search interfaces to software repositories. In this paper, we propose a novel multi-perspective cross-lingual neural framework for code--text matching, inspired in part by a previous model for monolingual text-to-text matching, to capture both global and local similarities. Our experiments on the CoNaLa dataset show that our proposed model yields better performance on this cross-lingual text-to-code matching task than previous approaches that map code and text to a single joint embedding space.
△ Less
Submitted 6 May, 2020;
originally announced May 2020.
-
Cluster Aware Mobility Encounter Dataset Enlargement
Authors:
Rajarshi Haldar,
Salih Safa Bacanli,
Moayad Aloqaily,
Adel Ben Mnaouer,
Damla Turgut
Abstract:
The recent emerging fields in data processing and manipulation has facilitated the need for synthetic data generation. This is also valid for mobility encounter dataset generation. Synthetic data generation might be useful to run research-based simulations and also create mobility encounter models. Our approach in this paper is to generate a larger dataset by using a given dataset which includes t…
▽ More
The recent emerging fields in data processing and manipulation has facilitated the need for synthetic data generation. This is also valid for mobility encounter dataset generation. Synthetic data generation might be useful to run research-based simulations and also create mobility encounter models. Our approach in this paper is to generate a larger dataset by using a given dataset which includes the clusters of people. Based on the cluster information, we created a framework. Using this framework, we can generate a similar dataset that is statistically similar to the input dataset. We have compared the statistical results of our approach with the real dataset and an encounter mobility model generation technique in the literature. The results showed that the created datasets have similar statistical structure with the given dataset.
△ Less
Submitted 20 February, 2020;
originally announced February 2020.
-
CL Scholar: The ACL Anthology Knowledge Graph Miner
Authors:
Mayank Singh,
Pradeep Dogga,
Sohan Patro,
Dhiraj Barnwal,
Ritam Dutt,
Rajarshi Haldar,
Pawan Goyal,
Animesh Mukherjee
Abstract:
We present CL Scholar, the ACL Anthology knowledge graph miner to facilitate high-quality search and exploration of current research progress in the computational linguistics community. In contrast to previous works, periodically crawling, indexing and processing of new incoming articles is completely automated in the current system. CL Scholar utilizes both textual and network information for kno…
▽ More
We present CL Scholar, the ACL Anthology knowledge graph miner to facilitate high-quality search and exploration of current research progress in the computational linguistics community. In contrast to previous works, periodically crawling, indexing and processing of new incoming articles is completely automated in the current system. CL Scholar utilizes both textual and network information for knowledge graph construction. As an additional novel initiative, CL Scholar supports more than 1200 scholarly natural language queries along with standard keyword-based search on constructed knowledge graph. It answers binary, statistical and list based natural language queries. The current system is deployed at http://cnerg.iitkgp.ac.in/aclakg. We also provide REST API support along with bulk download facility. Our code and data are available at https://github.com/CLScholar.
△ Less
Submitted 16 April, 2018;
originally announced April 2018.
-
Generalized Mathematical Formalism Governing Free-carrier Driven Kerr Frequency Comb in Optical Micro-cavities
Authors:
Raktim Haldar,
Arkadev Roy,
Partha Mondal,
Vishwatosh Mishra,
Shailendra K. Varshney
Abstract:
Continuous-wave pumped optical microresonators have been vastly exploited to generate frequency comb (FC) utilizing the Kerr nonlinearity. Most of the nonlinear materials used to build photonic platforms exhibit nonlinear losses such as multi-photon absorption, free-carrier absorption (FCA), and free-carrier dispersion (FCD) which can strongly affect the nonlinear characteristics of the devices ma…
▽ More
Continuous-wave pumped optical microresonators have been vastly exploited to generate frequency comb (FC) utilizing the Kerr nonlinearity. Most of the nonlinear materials used to build photonic platforms exhibit nonlinear losses such as multi-photon absorption, free-carrier absorption (FCA), and free-carrier dispersion (FCD) which can strongly affect the nonlinear characteristics of the devices made out of these materials. In this work, we model the Kerr FC based on modified Lugiato-Lefever Equation (LLE) along with the rate equation and develop analytical formulations to make quick estimations of the steady-state, modulation instability (MI) gain, bandwidth and the dynamics of Kerr Frequency-Comb (FC) in presence of nonlinear losses. Our analytical model is valid over a broad wavelength range of interest as it includes the effects of all nonlinear losses. Higher order (>3) characteristic polynomial of intra-cavity power describing the steady-state homogeneous solution of the modified LLE are discussed in detail. We derive the generalized analytical expressions for the threshold of normalized pump detuning to initiate the optical bistability which is a necessary condition for the FC generation. Free-carrier dispersion-led nonlinear cavity detuning is observed through the reverse Kerr-tilt of the resonant-peaks. We further deduce the expressions for the threshold pump intensity and the range of possible cavity detuning for the initiation of the MI when all the nonlinear losses are present. To corroborate our analytical findings, LLE along with the rate equations are solved numerically through split-step Fourier method. Our theoretical study can explain several experimental results which are previously reported and thereby is able to provide a better understanding of the comb dynamics.
△ Less
Submitted 1 February, 2018;
originally announced February 2018.
-
Applying Chatbots to the Internet of Things: Opportunities and Architectural Elements
Authors:
Rohan Kar,
Rishin Haldar
Abstract:
Internet of Things (IoT) is emerging as a significant technology in sha** the future by connecting physical devices or things with internet. It also presents various opportunities for intersection of other technological trends which can allow it to become even more intelligent and efficient. In this paper we focus our attention on the integration of Intelligent Conversational Software Agents or…
▽ More
Internet of Things (IoT) is emerging as a significant technology in sha** the future by connecting physical devices or things with internet. It also presents various opportunities for intersection of other technological trends which can allow it to become even more intelligent and efficient. In this paper we focus our attention on the integration of Intelligent Conversational Software Agents or Chatbots with IoT. Literature surveys have looked into various applications, features, underlying technologies and known challenges of IoT. On the other hand, Chatbots are being adopted in greater numbers due to major strides in development of platforms and frameworks. The novelty of this paper lies in the specific integration of Chatbots in the IoT scenario. We analyzed the shortcomings of existing IoT systems and put forward ways to tackle them by incorporating chatbots. A general architecture is proposed for implementing such a system, as well as platforms and frameworks, both commercial and open source, which allow for implementation of such systems. Identification of the newer challenges and possible future directions with this new integration, have also been addressed.
△ Less
Submitted 11 November, 2016;
originally announced November 2016.
-
Levenshtein Distance Technique in Dictionary Lookup Methods: An Improved Approach
Authors:
Rishin Haldar,
Debajyoti Mukhopadhyay
Abstract:
Dictionary lookup methods are popular in dealing with ambiguous letters which were not recognized by Optical Character Readers. However, a robust dictionary lookup method can be complex as apriori probability calculation or a large dictionary size increases the overhead and the cost of searching. In this context, Levenshtein distance is a simple metric which can be an effective string approximatio…
▽ More
Dictionary lookup methods are popular in dealing with ambiguous letters which were not recognized by Optical Character Readers. However, a robust dictionary lookup method can be complex as apriori probability calculation or a large dictionary size increases the overhead and the cost of searching. In this context, Levenshtein distance is a simple metric which can be an effective string approximation tool. After observing the effectiveness of this method, an improvement has been made to this method by grou** some similar looking alphabets and reducing the weighted difference among members of the same group. The results showed marked improvement over the traditional Levenshtein distance technique.
△ Less
Submitted 6 January, 2011;
originally announced January 2011.
-
Combined effects of thermal radiation and Hall current on MHD free-convective flow and mass transfer over a stretching sheet with variable viscosity
Authors:
G. C. Shit,
R. Haldar
Abstract:
An analysis has been investigated for the effects of thermal radiation and Hall current on magnetohydrodynamic free-convective flow and mass transfer over a stretching sheet with variable viscosity in the presence of heat generation/absorption. The fluid viscosity is assumed to vary as an inverse linear function of temperature. The boundary-layer equations governing the flow problem under consider…
▽ More
An analysis has been investigated for the effects of thermal radiation and Hall current on magnetohydrodynamic free-convective flow and mass transfer over a stretching sheet with variable viscosity in the presence of heat generation/absorption. The fluid viscosity is assumed to vary as an inverse linear function of temperature. The boundary-layer equations governing the flow problem under consideration have been reduced to a system of non-linear ordinary differential equations by employing a similarity transformation. Using the finite difference scheme, numerical solutions to the transform ordinary differential equations have been solved and the results that obtained are presented graphically. With an aim to test the accuracy, the numerical results have been compared with the existing scientific literature and found excellent agreement.
△ Less
Submitted 1 August, 2010;
originally announced August 2010.
