-
INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Models
Authors:
Hanseok Oh,
Hyunji Lee,
Seonghyeon Ye,
Haebin Shin,
Hansol Jang,
Changwook Jun,
Minjoon Seo
Abstract:
Despite the critical need to align search targets with users' intention, retrievers often only prioritize query information without delving into the users' intended search context. Enhancing the capability of retrievers to understand intentions and preferences of users, akin to language model instructions, has the potential to yield more aligned search targets. Prior studies restrict the applicati…
▽ More
Despite the critical need to align search targets with users' intention, retrievers often only prioritize query information without delving into the users' intended search context. Enhancing the capability of retrievers to understand intentions and preferences of users, akin to language model instructions, has the potential to yield more aligned search targets. Prior studies restrict the application of instructions in information retrieval to a task description format, neglecting the broader context of diverse and evolving search scenarios. Furthermore, the prevailing benchmarks utilized for evaluation lack explicit tailoring to assess instruction-following ability, thereby hindering progress in this field. In response to these limitations, we propose a novel benchmark,INSTRUCTIR, specifically designed to evaluate instruction-following ability in information retrieval tasks. Our approach focuses on user-aligned instructions tailored to each query instance, reflecting the diverse characteristics inherent in real-world search scenarios. Through experimental analysis, we observe that retrievers fine-tuned to follow task-style instructions, such as INSTRUCTOR, can underperform compared to their non-instruction-tuned counterparts. This underscores potential overfitting issues inherent in constructing retrievers trained on existing instruction-aware retrieval datasets.
△ Less
Submitted 22 February, 2024;
originally announced February 2024.
-
Block Selection Method for Using Feature Norm in Out-of-distribution Detection
Authors:
Yeonguk Yu,
Sungho Shin,
Seongju Lee,
Changhyun Jun,
Kyoobin Lee
Abstract:
Detecting out-of-distribution (OOD) inputs during the inference stage is crucial for deploying neural networks in the real world. Previous methods commonly relied on the output of a network derived from the highly activated feature map. In this study, we first revealed that a norm of the feature map obtained from the other block than the last block can be a better indicator of OOD detection. Motiv…
▽ More
Detecting out-of-distribution (OOD) inputs during the inference stage is crucial for deploying neural networks in the real world. Previous methods commonly relied on the output of a network derived from the highly activated feature map. In this study, we first revealed that a norm of the feature map obtained from the other block than the last block can be a better indicator of OOD detection. Motivated by this, we propose a simple framework consisting of FeatureNorm: a norm of the feature map and NormRatio: a ratio of FeatureNorm for ID and OOD to measure the OOD detection performance of each block. In particular, to select the block that provides the largest difference between FeatureNorm of ID and FeatureNorm of OOD, we create Jigsaw puzzle images as pseudo OOD from ID training samples and calculate NormRatio, and the block with the largest value is selected. After the suitable block is selected, OOD detection with the FeatureNorm outperforms other OOD detection methods by reducing FPR95 by up to 52.77% on CIFAR10 benchmark and by up to 48.53% on ImageNet benchmark. We demonstrate that our framework can generalize to various architectures and the importance of block selection, which can improve previous OOD detection methods as well.
△ Less
Submitted 2 March, 2023; v1 submitted 5 December, 2022;
originally announced December 2022.
-
ANNA: Enhanced Language Representation for Question Answering
Authors:
Changwook Jun,
Hansol Jang,
Myoseop Sim,
Hyun Kim,
Jooyoung Choi,
Kyungkoo Min,
Kyunghoon Bae
Abstract:
Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance indiv…
▽ More
Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance individually, and that the language model performs the best results on a specific question answering task when those approaches are jointly considered in pre-training models. In particular, we propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7\% F1 and 90.6\% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet on the SQuAD 2.0 benchmark.
△ Less
Submitted 3 April, 2022; v1 submitted 28 March, 2022;
originally announced March 2022.
-
Korean-Specific Dataset for Table Question Answering
Authors:
Changwook Jun,
Jooyoung Choi,
Myoseop Sim,
Hyun Kim,
Hansol Jang,
Kyungkoo Min
Abstract:
Existing question answering systems mainly focus on dealing with text data. However, much of the data produced daily is stored in the form of tables that can be found in documents and relational databases, or on the web. To solve the task of question answering over tables, there exist many datasets for table question answering written in English, but few Korean datasets. In this paper, we demonstr…
▽ More
Existing question answering systems mainly focus on dealing with text data. However, much of the data produced daily is stored in the form of tables that can be found in documents and relational databases, or on the web. To solve the task of question answering over tables, there exist many datasets for table question answering written in English, but few Korean datasets. In this paper, we demonstrate how we construct Korean-specific datasets for table question answering: Korean tabular dataset is a collection of 1.4M tables with corresponding descriptions for unsupervised pre-training language models. Korean table question answering corpus consists of 70k pairs of questions and answers created by crowd-sourced workers. Subsequently, we then build a pre-trained language model based on Transformer and fine-tune the model for table question answering with these datasets. We then report the evaluation results of our model. We make our datasets publicly available via our GitHub repository and hope that those datasets will help further studies for question answering over tables, and for the transformation of table formats.
△ Less
Submitted 1 May, 2022; v1 submitted 17 January, 2022;
originally announced January 2022.
-
Importance of Secure Software Development Processes and Tools for Developers
Authors:
Muhammad Danish Roshaidie,
William Pang Han Liang,
Calvin Goh Kai Jun,
Kok Hong Yew,
Fatima-tuz-Zahra
Abstract:
In this research paper of secure software systems, authors have discussed what the proper development process is when it comes to creating a secure software, which will be suited for developers and relevent stakeholders alike. Secure Software Development Process for Developers is of crucial importance for software engineers as more and more software-based devices are becoming commonly available, a…
▽ More
In this research paper of secure software systems, authors have discussed what the proper development process is when it comes to creating a secure software, which will be suited for developers and relevent stakeholders alike. Secure Software Development Process for Developers is of crucial importance for software engineers as more and more software-based devices are becoming commonly available, and cloud services are evolving which require for the software to be constantly connected to the internet. With this in mind, Secure Software Development needs to be transformed to something most developers can rely upon to make applied software safe and have the capability to mitigate against potential attacks by hackers. Furthermore, in this paper, existing Secure Software Development Process ideas and implementations are reviewed and investigated using the research paper pool available online. Thereafter, an approach is proposed to enhance the security aspect in software development process to resolve security issues. Lastly, the paper concludes with final remarks on practical implementation of security features in software development phases for production of secure and reliable software programs and systems.
△ Less
Submitted 30 December, 2020;
originally announced December 2020.
-
Geometric projection-based switching policy for multiple energy harvesting transmitters
Authors:
Chen Hongbin,
Zhou Fangfang,
Cai Jun,
Zhao Feng,
He Qian
Abstract:
Transmitter switching can provide resiliency and robustness to a communication system with multiple energy harvesting transmitters. However, excessive transmitter switching will bring heavy control overhead. In this paper, a geometric projection-based transmitter switching policy is proposed for a communication system with multiple energy harvesting transmitters and one receiver, which can reduce…
▽ More
Transmitter switching can provide resiliency and robustness to a communication system with multiple energy harvesting transmitters. However, excessive transmitter switching will bring heavy control overhead. In this paper, a geometric projection-based transmitter switching policy is proposed for a communication system with multiple energy harvesting transmitters and one receiver, which can reduce the number of switches. The results show that the proposed transmitter switching policy outperforms several heuristic ones.
△ Less
Submitted 19 May, 2014;
originally announced May 2014.
