Showing 1–2 of 2 results for author: Duvenhage, B

Short Text Language Identification for Under Resourced Languages

Authors: Bernardt Duvenhage

Abstract: The paper presents a hierarchical naive Bayesian and lexicon based classifier for short text language identification (LID) useful for under resourced languages. The algorithm is evaluated on short pieces of text for the 11 official South African languages some of which are similar languages. The algorithm is compared to recent approaches using test sets from previous works on South African languag… ▽ More The paper presents a hierarchical naive Bayesian and lexicon based classifier for short text language identification (LID) useful for under resourced languages. The algorithm is evaluated on short pieces of text for the 11 official South African languages some of which are similar languages. The algorithm is compared to recent approaches using test sets from previous works on South African languages as well as the Discriminating between Similar Languages (DSL) shared tasks' datasets. Remaining research opportunities and pressing concerns in evaluating and comparing LID approaches are also discussed. △ Less

Submitted 21 November, 2019; v1 submitted 18 November, 2019; originally announced November 2019.

Comments: Presented at NeurIPS 2019 Workshop on Machine Learning for the Develo** World

MSC Class: 68T50

Improved Text Language Identification for the South African Languages

Authors: Bernardt Duvenhage, Mfundo Ntini, Phala Ramonyai

Abstract: Virtual assistants and text chatbots have recently been gaining popularity. Given the short message nature of text-based chat interactions, the language identification systems of these bots might only have 15 or 20 characters to make a prediction. However, accurate text language identification is important, especially in the early stages of many multilingual natural language processing pipelines.… ▽ More Virtual assistants and text chatbots have recently been gaining popularity. Given the short message nature of text-based chat interactions, the language identification systems of these bots might only have 15 or 20 characters to make a prediction. However, accurate text language identification is important, especially in the early stages of many multilingual natural language processing pipelines. This paper investigates the use of a naive Bayes classifier, to accurately predict the language family that a piece of text belongs to, combined with a lexicon based classifier to distinguish the specific South African language that the text is written in. This approach leads to a 31% reduction in the language detection error. In the spirit of reproducible research the training and testing datasets as well as the code are published on github. Hopefully it will be useful to create a text language identification shared task for South African languages. △ Less

Submitted 1 November, 2017; originally announced November 2017.

Comments: Accepted to appear in the proceedings of The 28th Annual Symposium of the Pattern Recognition Association of South Africa, 2017