Using Covariance as a Similarity Measure for Document Language Identification in Hard Contexts

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Existing Language Identification (LID) approaches achieve 100% precision in most common situations, dealing with sufficiently large documents, written in just one language. However, there are many situations where text language is hard to identify and where current LID approaches do not provide a reliable solution. One such situation occurs when it is necessary to discriminate the correct variant of the language used in a text. In this paper, we present a fully statistics-based LID approach which is shown to be correct for common texts and maintains its robustness when classifying hard LID documents. For that, character sequences were used as base features. The Discriminant Ability of each sequence, in each training situation, is measured and used to filter out less important character sequences. Document similarity measure, based on the covariance concept, was defined. In the training phase, document clusters are built in a reduced $k$ uncorrelated dimensions space. In the classification phase the Quadratic Discriminant Score decides which cluster (language) must be assigned to the documents one needs to classify.
Original languageEnglish
Title of host publicationProceedings of the XII International Summer Conference on Probability and Statistics and Seminar on Statistical Data Analysis, SDA, 2006
EditorsN. Yanev
Place of PublicationSofia
PublisherInstitute of Mathematics and Informatics, Bulgarian Academy of Sciences
Pages341-360
Number of pages20
Publication statusPublished - 19 Mar 2007
EventXII International Summer Conference on Probability and Statistics and Seminar on Statistical Data Analysis, SDA - Sofia, Bulgaria
Duration: 1 Jan 2016 → …

Publication series

NamePliska Studia Mathematica Bulgarica, Bulgaria
Volume18
ISSN (Print)0204-9805

Conference

ConferenceXII International Summer Conference on Probability and Statistics and Seminar on Statistical Data Analysis, SDA
Country/TerritoryBulgaria
CitySofia
Period1/01/16 → …

Keywords

  • Statistical applications

Cite this