A Review of Big Data and Machine Learning Operations in Official Statistics: MLOps and Feature Store Adoption

Carlos Nunes; Afshin Ashofteh

doi:10.1109/COMPSAC61105.2024.00101

A Review of Big Data and Machine Learning Operations in Official Statistics: MLOps and Feature Store Adoption

Carlos Nunes, Afshin Ashofteh

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

https://www.youtube.com/watch?v=3brq4oDerMY

Abstract

Integrating machine learning (ML) into the official statisticians' toolset is gaining popularity as National Statistical Offices (NSOs) strive to improve their methodologies. This trend poses new challenges and implications for incorporating innovative techniques that ensure the reliability of the official statistical production process. A comprehensive literature review was conducted using Scopus and Web of Science databases to explore the contemporary applications of data science in official statistics. A total of 178 research articles were identified, focusing on areas such as big data, machine learning, and data quality. While the literature review revealed extensive proposals on utilizing alternative data and applying machine learning techniques to support official statistics production, it also identified research gaps in the post-training steps of the machine learning process. Areas requiring further investigation include machine learning operations in a production environment, data quality assurance, and governance.

Original language	English
Title of host publication	2024 IEEE 48th Annual Computers, Software, and Applications Conference
Subtitle of host publication	COMPSAC 2024
Editors	Hossain Shahriar, Hiroyuki Ohsaki, Moushumi Sharmin, Dave Towey, AKM Jahangir Alam Majumder, Yoshiaki Hori, Ji-Jiang Yang, Michiharu Takemoto, Nazmus Sakib, Ryohei Banno, Sheikh Iqbal Ahamed
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	711-718
Number of pages	8
ISBN (Print)	979-8-3503-7696-8
DOIs	https://doi.org/10.1109/COMPSAC61105.2024.00101
Publication status	Published - Jul 2024
Event	48th IEEE Annual Computers, Software, and Applications Conference - Nakanoshima Love Central, Osaka, Japan Duration: 2 Jul 2024 → 4 Jul 2024 Conference number: 48 https://ieeecompsac.computer.org/2024/

Publication series

Name	Proceedings of the IEEE Annual Computer Software and Applications Conference
Publisher	IEEE
ISSN (Print)	2836-3795

Conference

Conference	48th IEEE Annual Computers, Software, and Applications Conference
Abbreviated title	COMPSAC 2024
Country/Territory	Japan
City	Osaka
Period	2/07/24 → 4/07/24
Internet address	https://ieeecompsac.computer.org/2024/

Keywords

Feature store
Official statistics
Machine learning operations
Data science
Big data
Data quality

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1109/COMPSAC61105.2024.00101Licence: Unspecified

Cite this

Nunes, C., & Ashofteh, A. (2024). A Review of Big Data and Machine Learning Operations in Official Statistics: MLOps and Feature Store Adoption. In H. Shahriar, H. Ohsaki, M. Sharmin, D. Towey, AKM. J. A. Majumder, Y. Hori, J-J. Yang, M. Takemoto, N. Sakib, R. Banno, & S. I. Ahamed (Eds.), 2024 IEEE 48th Annual Computers, Software, and Applications Conference: COMPSAC 2024 (pp. 711-718). (Proceedings of the IEEE Annual Computer Software and Applications Conference). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/COMPSAC61105.2024.00101

Nunes, Carlos ; Ashofteh, Afshin. / A Review of Big Data and Machine Learning Operations in Official Statistics : MLOps and Feature Store Adoption. 2024 IEEE 48th Annual Computers, Software, and Applications Conference: COMPSAC 2024. editor / Hossain Shahriar ; Hiroyuki Ohsaki ; Moushumi Sharmin ; Dave Towey ; AKM Jahangir Alam Majumder ; Yoshiaki Hori ; Ji-Jiang Yang ; Michiharu Takemoto ; Nazmus Sakib ; Ryohei Banno ; Sheikh Iqbal Ahamed. Institute of Electrical and Electronics Engineers (IEEE), 2024. pp. 711-718 (Proceedings of the IEEE Annual Computer Software and Applications Conference).

@inproceedings{9f90e1ce86dd48a884ed7950f918c7a5,

title = "A Review of Big Data and Machine Learning Operations in Official Statistics: MLOps and Feature Store Adoption",

abstract = "Integrating machine learning (ML) into the official statisticians' toolset is gaining popularity as National Statistical Offices (NSOs) strive to improve their methodologies. This trend poses new challenges and implications for incorporating innovative techniques that ensure the reliability of the official statistical production process. A comprehensive literature review was conducted using Scopus and Web of Science databases to explore the contemporary applications of data science in official statistics. A total of 178 research articles were identified, focusing on areas such as big data, machine learning, and data quality. While the literature review revealed extensive proposals on utilizing alternative data and applying machine learning techniques to support official statistics production, it also identified research gaps in the post-training steps of the machine learning process. Areas requiring further investigation include machine learning operations in a production environment, data quality assurance, and governance. ",

keywords = "Feature store, Official statistics, Machine learning operations, Data science, Big data, Data quality",

author = "Carlos Nunes and Afshin Ashofteh",

note = "Nunes, C. E. R., & Ashofteh, A. (2024). A Review of Big Data and Machine Learning Operations in Official Statistics: MLOps and Feature Store Adoption. In H. Shahriar, H. Ohsaki, M. Sharmin, D. Towey, AKM. J. A. Majumder, Y. Hori, J-J. Yang, M. Takemoto, N. Sakib, R. Banno, & S. I. Ahamed (Eds.), 2024 IEEE 48th Annual Computers, Software, and Applications Conference: COMPSAC 2024 (pp. 711-718). (Proceedings of the IEEE Annual Computer Software and Applications Conference). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/COMPSAC61105.2024.00101; 48th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2024 ; Conference date: 02-07-2024 Through 04-07-2024",

year = "2024",

month = jul,

doi = "10.1109/COMPSAC61105.2024.00101",

language = "English",

isbn = "979-8-3503-7696-8",

series = "Proceedings of the IEEE Annual Computer Software and Applications Conference",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

pages = "711--718",

editor = "Hossain Shahriar and Hiroyuki Ohsaki and Moushumi Sharmin and Dave Towey and Majumder, {AKM Jahangir Alam} and Yoshiaki Hori and Ji-Jiang Yang and Michiharu Takemoto and Nazmus Sakib and Ryohei Banno and Ahamed, {Sheikh Iqbal}",

booktitle = "2024 IEEE 48th Annual Computers, Software, and Applications Conference",

address = "United States",

url = "https://ieeecompsac.computer.org/2024/",

}

Nunes, C & Ashofteh, A 2024, A Review of Big Data and Machine Learning Operations in Official Statistics: MLOps and Feature Store Adoption. in H Shahriar, H Ohsaki, M Sharmin, D Towey, AKMJA Majumder, Y Hori, J-J Yang, M Takemoto, N Sakib, R Banno & SI Ahamed (eds), 2024 IEEE 48th Annual Computers, Software, and Applications Conference: COMPSAC 2024. Proceedings of the IEEE Annual Computer Software and Applications Conference, Institute of Electrical and Electronics Engineers (IEEE), pp. 711-718, 48th IEEE Annual Computers, Software, and Applications Conference, Osaka, Japan, 2/07/24. https://doi.org/10.1109/COMPSAC61105.2024.00101

A Review of Big Data and Machine Learning Operations in Official Statistics: MLOps and Feature Store Adoption. / Nunes, Carlos ; Ashofteh, Afshin.
2024 IEEE 48th Annual Computers, Software, and Applications Conference: COMPSAC 2024. ed. / Hossain Shahriar; Hiroyuki Ohsaki; Moushumi Sharmin; Dave Towey; AKM Jahangir Alam Majumder; Yoshiaki Hori; Ji-Jiang Yang; Michiharu Takemoto; Nazmus Sakib; Ryohei Banno; Sheikh Iqbal Ahamed. Institute of Electrical and Electronics Engineers (IEEE), 2024. p. 711-718 (Proceedings of the IEEE Annual Computer Software and Applications Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Review of Big Data and Machine Learning Operations in Official Statistics

T2 - 48th IEEE Annual Computers, Software, and Applications Conference

AU - Nunes, Carlos

AU - Ashofteh, Afshin

N1 - Conference code: 48

PY - 2024/7

Y1 - 2024/7

N2 - Integrating machine learning (ML) into the official statisticians' toolset is gaining popularity as National Statistical Offices (NSOs) strive to improve their methodologies. This trend poses new challenges and implications for incorporating innovative techniques that ensure the reliability of the official statistical production process. A comprehensive literature review was conducted using Scopus and Web of Science databases to explore the contemporary applications of data science in official statistics. A total of 178 research articles were identified, focusing on areas such as big data, machine learning, and data quality. While the literature review revealed extensive proposals on utilizing alternative data and applying machine learning techniques to support official statistics production, it also identified research gaps in the post-training steps of the machine learning process. Areas requiring further investigation include machine learning operations in a production environment, data quality assurance, and governance.

AB - Integrating machine learning (ML) into the official statisticians' toolset is gaining popularity as National Statistical Offices (NSOs) strive to improve their methodologies. This trend poses new challenges and implications for incorporating innovative techniques that ensure the reliability of the official statistical production process. A comprehensive literature review was conducted using Scopus and Web of Science databases to explore the contemporary applications of data science in official statistics. A total of 178 research articles were identified, focusing on areas such as big data, machine learning, and data quality. While the literature review revealed extensive proposals on utilizing alternative data and applying machine learning techniques to support official statistics production, it also identified research gaps in the post-training steps of the machine learning process. Areas requiring further investigation include machine learning operations in a production environment, data quality assurance, and governance.

KW - Feature store

KW - Official statistics

KW - Machine learning operations

KW - Data science

KW - Big data

KW - Data quality

UR - http://www.scopus.com/inward/record.url?scp=85204036933&partnerID=8YFLogxK

UR - https://www.webofscience.com/wos/woscc/full-record/WOS:001308581200092

UR - https://ieeecompsac.computer.org/2024/

UR - https://conferences.computer.org/compsacpub24/#!/home

UR - https://data.mendeley.com/datasets/96mxz7jvkr/1

UR - https://youtu.be/3brq4oDerMY

U2 - 10.1109/COMPSAC61105.2024.00101

DO - 10.1109/COMPSAC61105.2024.00101

M3 - Conference contribution

SN - 979-8-3503-7696-8

T3 - Proceedings of the IEEE Annual Computer Software and Applications Conference

SP - 711

EP - 718

BT - 2024 IEEE 48th Annual Computers, Software, and Applications Conference

A2 - Shahriar, Hossain

A2 - Ohsaki, Hiroyuki

A2 - Sharmin, Moushumi

A2 - Towey, Dave

A2 - Majumder, AKM Jahangir Alam

A2 - Hori, Yoshiaki

A2 - Yang, Ji-Jiang

A2 - Takemoto, Michiharu

A2 - Sakib, Nazmus

A2 - Banno, Ryohei

A2 - Ahamed, Sheikh Iqbal

PB - Institute of Electrical and Electronics Engineers (IEEE)

Y2 - 2 July 2024 through 4 July 2024

ER -

Nunes C , Ashofteh A. A Review of Big Data and Machine Learning Operations in Official Statistics: MLOps and Feature Store Adoption. In Shahriar H, Ohsaki H, Sharmin M, Towey D, Majumder AKMJA, Hori Y, Yang J-J, Takemoto M, Sakib N, Banno R, Ahamed SI, editors, 2024 IEEE 48th Annual Computers, Software, and Applications Conference: COMPSAC 2024. Institute of Electrical and Electronics Engineers (IEEE). 2024. p. 711-718. (Proceedings of the IEEE Annual Computer Software and Applications Conference). doi: 10.1109/COMPSAC61105.2024.00101

A Review of Big Data and Machine Learning Operations in Official Statistics: MLOps and Feature Store Adoption

Abstract

Publication series

Conference

Keywords

UN SDGs

Access to Document

Other files and links

Embargoed Document

Fingerprint

Cite this