A Parallel Algorithm for Statistical Multiword Term Extraction from Very Large Corpora

Carlos Gonçalves; Joaquim Francisco Ferreira Silva; José Alberto Cardoso e Cunha

doi:10.1109/HPCC-CSS-ICESS.2015.72

A Parallel Algorithm for Statistical Multiword Term Extraction from Very Large Corpora

Carlos Gonçalves, Joaquim Francisco Ferreira Silva, José Alberto Cardoso e Cunha

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Citations (Scopus)

Abstract

Multi-word Relevant Expressions (REs) can be defined as sequences of words (n grams) with strong semantic meaning, such as “ice melting” and “Ministère des Affaires Étrangères”, useful in Information Retrieval, Document Clustering or Classification and Indexing of Documents. The need of extracting REs in several languages led research on statistical approaches rather than symbolic methods, since the former allow language-independence. Based on the assumption that REs have strong cohesion between their consecutive n-grams, the LocalMaxs algorithm is a language independent approach that extracts REs. Apart from its good precision, this extractor is time-consuming, being inoperable for Big Data if implemented in a sequential manner. This paper presents the first parallel and distributed version of this algorithm, achieving almost linear speedup and sizeup when processing corpora up to 1 billion words, using up to 54 virtual machines in a public cloud.

Original language	English
Title of host publication	Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems (HPCC-CSS-ICESS 2015)
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	219-224
Number of pages	6
ISBN (Electronic)	978-1-4799-8937-9
ISBN (Print)	978-1-4799-8938-6
DOIs	https://doi.org/10.1109/HPCC-CSS-ICESS.2015.72
Publication status	Published - 2015
Event	17th IEEE International Conference on High Performance Computing and Communications, IEEE 7th International Symposium on Cyberspace Safety and Security and IEEE 12th International Conference on Embedded Software and Systems (HPCC-ICESS-CSS 2015) - New York, United States Duration: 24 Aug 2015 → 26 Aug 2015

Conference

Conference	17th IEEE International Conference on High Performance Computing and Communications, IEEE 7th International Symposium on Cyberspace Safety and Security and IEEE 12th International Conference on Embedded Software and Systems (HPCC-ICESS-CSS 2015)
Abbreviated title	HPCC-ICESS-CSS 2015
Country/Territory	United States
City	New York
Period	24/08/15 → 26/08/15

Keywords

Cloud
Large Corpora
Multiword Terms
Parallel Processing
Statistical Extraction
Text Mining

Access to Document

10.1109/HPCC-CSS-ICESS.2015.72Licence: Unspecified

Cite this

Gonçalves, C., Silva, J. F. F., & Cunha, J. A. C. E. (2015). A Parallel Algorithm for Statistical Multiword Term Extraction from Very Large Corpora. In Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems (HPCC-CSS-ICESS 2015) (pp. 219-224). [7336167] Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/HPCC-CSS-ICESS.2015.72

Gonçalves, Carlos ; Silva, Joaquim Francisco Ferreira ; Cunha, José Alberto Cardoso e. / A Parallel Algorithm for Statistical Multiword Term Extraction from Very Large Corpora. Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems (HPCC-CSS-ICESS 2015) . Institute of Electrical and Electronics Engineers (IEEE), 2015. pp. 219-224

@inproceedings{90c52f7f1ecd42858698031b57cfb9b2,

title = "A Parallel Algorithm for Statistical Multiword Term Extraction from Very Large Corpora",

abstract = "Multi-word Relevant Expressions (REs) can be defined as sequences of words (n grams) with strong semantic meaning, such as “ice melting” and “Minist{\`e}re des Affaires {\'E}trang{\`e}res”, useful in Information Retrieval, Document Clustering or Classification and Indexing of Documents. The need of extracting REs in several languages led research on statistical approaches rather than symbolic methods, since the former allow language-independence. Based on the assumption that REs have strong cohesion between their consecutive n-grams, the LocalMaxs algorithm is a language independent approach that extracts REs. Apart from its good precision, this extractor is time-consuming, being inoperable for Big Data if implemented in a sequential manner. This paper presents the first parallel and distributed version of this algorithm, achieving almost linear speedup and sizeup when processing corpora up to 1 billion words, using up to 54 virtual machines in a public cloud.",

keywords = "Cloud , Large Corpora , Multiword Terms , Parallel Processing , Statistical Extraction , Text Mining",

author = "Carlos Gon{\c c}alves and Silva, {Joaquim Francisco Ferreira} and Cunha, {Jos{\'e} Alberto Cardoso e}",

note = "Sem PDF conforme Despacho.; 17th IEEE International Conference on High Performance Computing and Communications, IEEE 7th International Symposium on Cyberspace Safety and Security and IEEE 12th International Conference on Embedded Software and Systems (HPCC-ICESS-CSS 2015), HPCC-ICESS-CSS 2015 ; Conference date: 24-08-2015 Through 26-08-2015",

year = "2015",

doi = "10.1109/HPCC-CSS-ICESS.2015.72",

language = "English",

isbn = "978-1-4799-8938-6",

pages = "219--224",

booktitle = "Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems (HPCC-CSS-ICESS 2015)",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

}

Gonçalves, C, Silva, JFF & Cunha, JACE 2015, A Parallel Algorithm for Statistical Multiword Term Extraction from Very Large Corpora. in Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems (HPCC-CSS-ICESS 2015) ., 7336167, Institute of Electrical and Electronics Engineers (IEEE), pp. 219-224, 17th IEEE International Conference on High Performance Computing and Communications, IEEE 7th International Symposium on Cyberspace Safety and Security and IEEE 12th International Conference on Embedded Software and Systems (HPCC-ICESS-CSS 2015), New York, United States, 24/08/15. https://doi.org/10.1109/HPCC-CSS-ICESS.2015.72

A Parallel Algorithm for Statistical Multiword Term Extraction from Very Large Corpora. / Gonçalves, Carlos; Silva, Joaquim Francisco Ferreira ; Cunha, José Alberto Cardoso e.
Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems (HPCC-CSS-ICESS 2015) . Institute of Electrical and Electronics Engineers (IEEE), 2015. p. 219-224 7336167.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Parallel Algorithm for Statistical Multiword Term Extraction from Very Large Corpora

AU - Gonçalves, Carlos

AU - Silva, Joaquim Francisco Ferreira

AU - Cunha, José Alberto Cardoso e

N1 - Sem PDF conforme Despacho.

PY - 2015

Y1 - 2015

N2 - Multi-word Relevant Expressions (REs) can be defined as sequences of words (n grams) with strong semantic meaning, such as “ice melting” and “Ministère des Affaires Étrangères”, useful in Information Retrieval, Document Clustering or Classification and Indexing of Documents. The need of extracting REs in several languages led research on statistical approaches rather than symbolic methods, since the former allow language-independence. Based on the assumption that REs have strong cohesion between their consecutive n-grams, the LocalMaxs algorithm is a language independent approach that extracts REs. Apart from its good precision, this extractor is time-consuming, being inoperable for Big Data if implemented in a sequential manner. This paper presents the first parallel and distributed version of this algorithm, achieving almost linear speedup and sizeup when processing corpora up to 1 billion words, using up to 54 virtual machines in a public cloud.

AB - Multi-word Relevant Expressions (REs) can be defined as sequences of words (n grams) with strong semantic meaning, such as “ice melting” and “Ministère des Affaires Étrangères”, useful in Information Retrieval, Document Clustering or Classification and Indexing of Documents. The need of extracting REs in several languages led research on statistical approaches rather than symbolic methods, since the former allow language-independence. Based on the assumption that REs have strong cohesion between their consecutive n-grams, the LocalMaxs algorithm is a language independent approach that extracts REs. Apart from its good precision, this extractor is time-consuming, being inoperable for Big Data if implemented in a sequential manner. This paper presents the first parallel and distributed version of this algorithm, achieving almost linear speedup and sizeup when processing corpora up to 1 billion words, using up to 54 virtual machines in a public cloud.

KW - Cloud

KW - Large Corpora

KW - Multiword Terms

KW - Parallel Processing

KW - Statistical Extraction

KW - Text Mining

U2 - 10.1109/HPCC-CSS-ICESS.2015.72

DO - 10.1109/HPCC-CSS-ICESS.2015.72

M3 - Conference contribution

SN - 978-1-4799-8938-6

SP - 219

EP - 224

BT - Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems (HPCC-CSS-ICESS 2015)

PB - Institute of Electrical and Electronics Engineers (IEEE)

T2 - 17th IEEE International Conference on High Performance Computing and Communications, IEEE 7th International Symposium on Cyberspace Safety and Security and IEEE 12th International Conference on Embedded Software and Systems (HPCC-ICESS-CSS 2015)

Y2 - 24 August 2015 through 26 August 2015

ER -

Gonçalves C, Silva JFF , Cunha JACE. A Parallel Algorithm for Statistical Multiword Term Extraction from Very Large Corpora. In Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems (HPCC-CSS-ICESS 2015) . Institute of Electrical and Electronics Engineers (IEEE). 2015. p. 219-224. 7336167 doi: 10.1109/HPCC-CSS-ICESS.2015.72

A Parallel Algorithm for Statistical Multiword Term Extraction from Very Large Corpora

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this