Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark

Afshin Ashofteh

doi:10.1007/978-3-031-40055-1_14

Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Recently, Big Data has become an increasingly important source to support traditional credit scoring. Personal credit evaluation based on machine learning approaches focuses on the application data of clients in open banking and new banking platforms with challenges about Big Data quality and model risk. This paper represents a PySpark code for computationally efficient use of statistical learning and machine learning algorithms for the application scenario of personal credit evaluation with a performance comparison of models including logistic regression, decision tree, random forest, neural network, and support vector machine. The findings of this study reveal that the logistic regression methodology represents a more reasonable coefficient of determination and a lower false negative rate than other models. Additionally, it is computationally less expensive and more comprehensible. Finally, the paper highlights the steps, perils, and benefits of using Big Data and machine learning algorithms in credit scoring.

Original language	English
Title of host publication	Statistical Modeling and Simulation for Experimental Design and Machine Learning Applications
Subtitle of host publication	Selected Contributions from SimStat 2019 and Invited Papers
Editors	Jürgen Pilz, Viatcheslav B. Melas, Arne Bathke
Place of Publication	Gewerbestrasse, Switzerland
Publisher	Springer, Cham
Chapter	14
Pages	245-265
Number of pages	21
ISBN (Electronic)	978-3-031-40055-1
ISBN (Print)	978-3-031-40054-4, 978-3-031-40057-5
DOIs	https://doi.org/10.1007/978-3-031-40055-1_14
Publication status	Published - 19 Oct 2023
Event	10th International Workshop on Simulation and Statistics - Faculty of Natural Sciences Hellbrunner Strasse 34 5020, Salzburg, Austria Duration: 2 Sept 2023 → 6 Sept 2023 https://datascience.plus.ac.at/SimStatSalzburg2019/

Publication series

Name	Contributions to Statistics
Publisher	Springer Cham
ISSN (Print)	1431-1968

Conference

Conference	10th International Workshop on Simulation and Statistics
Abbreviated title	SimStat 2019
Country/Territory	Austria
City	Salzburg
Period	2/09/23 → 6/09/23
Internet address	https://datascience.plus.ac.at/SimStatSalzburg2019/

Keywords

Credit score
Big Data
Machine learning
Risk Management
Finance

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-031-40055-1_14Licence: Unspecified

Cite this

Ashofteh, A. (2023). Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark. In J. Pilz, V. B. Melas, & A. Bathke (Eds.), Statistical Modeling and Simulation for Experimental Design and Machine Learning Applications: Selected Contributions from SimStat 2019 and Invited Papers (pp. 245-265). (Contributions to Statistics). Springer, Cham. https://doi.org/10.1007/978-3-031-40055-1_14

Ashofteh, Afshin. / Big Data for Credit Risk Analysis : Efficient Machine Learning Models Using PySpark. Statistical Modeling and Simulation for Experimental Design and Machine Learning Applications: Selected Contributions from SimStat 2019 and Invited Papers. editor / Jürgen Pilz ; Viatcheslav B. Melas ; Arne Bathke. Gewerbestrasse, Switzerland : Springer, Cham, 2023. pp. 245-265 (Contributions to Statistics).

@inproceedings{fe9c5b273a0a43c7b5c17c4821f96293,

title = "Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark",

abstract = "Recently, Big Data has become an increasingly important source to support traditional credit scoring. Personal credit evaluation based on machine learning approaches focuses on the application data of clients in open banking and new banking platforms with challenges about Big Data quality and model risk. This paper represents a PySpark code for computationally efficient use of statistical learning and machine learning algorithms for the application scenario of personal credit evaluation with a performance comparison of models including logistic regression, decision tree, random forest, neural network, and support vector machine. The findings of this study reveal that the logistic regression methodology represents a more reasonable coefficient of determination and a lower false negative rate than other models. Additionally, it is computationally less expensive and more comprehensible. Finally, the paper highlights the steps, perils, and benefits of using Big Data and machine learning algorithms in credit scoring.",

keywords = "Credit score, Big Data, Machine learning, Risk Management, Finance",

author = "Afshin Ashofteh",

note = "Ashofteh, A. (2023). Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark. In J. Pilz, V. B. Melas, & A. Bathke (Eds.), Statistical Modeling and Simulation for Experimental Design and Machine Learning Applications: Selected Contributions from SimStat 2019 and Invited Papers (pp. 245-265). (Contributions to Statistics). Springer, Cham. https://doi.org/10.1007/978-3-031-40055-1_14; 10th International Workshop on Simulation and Statistics, SimStat 2019 ; Conference date: 02-09-2023 Through 06-09-2023",

year = "2023",

month = oct,

day = "19",

doi = "10.1007/978-3-031-40055-1_14",

language = "English",

isbn = "978-3-031-40054-4",

series = "Contributions to Statistics",

publisher = "Springer, Cham",

pages = "245--265",

editor = "J{\"u}rgen Pilz and Melas, {Viatcheslav B.} and Arne Bathke",

booktitle = "Statistical Modeling and Simulation for Experimental Design and Machine Learning Applications",

url = "https://datascience.plus.ac.at/SimStatSalzburg2019/",

}

Ashofteh, A 2023, Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark. in J Pilz, VB Melas & A Bathke (eds), Statistical Modeling and Simulation for Experimental Design and Machine Learning Applications: Selected Contributions from SimStat 2019 and Invited Papers. Contributions to Statistics, Springer, Cham, Gewerbestrasse, Switzerland, pp. 245-265, 10th International Workshop on Simulation and Statistics, Salzburg, Austria, 2/09/23. https://doi.org/10.1007/978-3-031-40055-1_14

Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark. / Ashofteh, Afshin.
Statistical Modeling and Simulation for Experimental Design and Machine Learning Applications: Selected Contributions from SimStat 2019 and Invited Papers. ed. / Jürgen Pilz; Viatcheslav B. Melas; Arne Bathke. Gewerbestrasse, Switzerland: Springer, Cham, 2023. p. 245-265 (Contributions to Statistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Big Data for Credit Risk Analysis

T2 - 10th International Workshop on Simulation and Statistics

AU - Ashofteh, Afshin

N1 - Ashofteh, A. (2023). Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark. In J. Pilz, V. B. Melas, & A. Bathke (Eds.), Statistical Modeling and Simulation for Experimental Design and Machine Learning Applications: Selected Contributions from SimStat 2019 and Invited Papers (pp. 245-265). (Contributions to Statistics). Springer, Cham. https://doi.org/10.1007/978-3-031-40055-1_14

PY - 2023/10/19

Y1 - 2023/10/19

N2 - Recently, Big Data has become an increasingly important source to support traditional credit scoring. Personal credit evaluation based on machine learning approaches focuses on the application data of clients in open banking and new banking platforms with challenges about Big Data quality and model risk. This paper represents a PySpark code for computationally efficient use of statistical learning and machine learning algorithms for the application scenario of personal credit evaluation with a performance comparison of models including logistic regression, decision tree, random forest, neural network, and support vector machine. The findings of this study reveal that the logistic regression methodology represents a more reasonable coefficient of determination and a lower false negative rate than other models. Additionally, it is computationally less expensive and more comprehensible. Finally, the paper highlights the steps, perils, and benefits of using Big Data and machine learning algorithms in credit scoring.

AB - Recently, Big Data has become an increasingly important source to support traditional credit scoring. Personal credit evaluation based on machine learning approaches focuses on the application data of clients in open banking and new banking platforms with challenges about Big Data quality and model risk. This paper represents a PySpark code for computationally efficient use of statistical learning and machine learning algorithms for the application scenario of personal credit evaluation with a performance comparison of models including logistic regression, decision tree, random forest, neural network, and support vector machine. The findings of this study reveal that the logistic regression methodology represents a more reasonable coefficient of determination and a lower false negative rate than other models. Additionally, it is computationally less expensive and more comprehensible. Finally, the paper highlights the steps, perils, and benefits of using Big Data and machine learning algorithms in credit scoring.

KW - Credit score

KW - Big Data

KW - Machine learning

KW - Risk Management

KW - Finance

UR - https://github.com/AfshinAshofteh/creditscore_pyspark

U2 - 10.1007/978-3-031-40055-1_14

DO - 10.1007/978-3-031-40055-1_14

M3 - Conference contribution

SN - 978-3-031-40054-4

SN - 978-3-031-40057-5

T3 - Contributions to Statistics

SP - 245

EP - 265

BT - Statistical Modeling and Simulation for Experimental Design and Machine Learning Applications

A2 - Pilz, Jürgen

A2 - Melas, Viatcheslav B.

A2 - Bathke, Arne

PB - Springer, Cham

CY - Gewerbestrasse, Switzerland

Y2 - 2 September 2023 through 6 September 2023

ER -

Ashofteh A. Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark. In Pilz J, Melas VB, Bathke A, editors, Statistical Modeling and Simulation for Experimental Design and Machine Learning Applications: Selected Contributions from SimStat 2019 and Invited Papers. Gewerbestrasse, Switzerland: Springer, Cham. 2023. p. 245-265. (Contributions to Statistics). doi: 10.1007/978-3-031-40055-1_14

Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark

Abstract

Publication series

Conference

Keywords

UN SDGs

Access to Document

Other files and links

Embargoed Document

Fingerprint

First Prize for the presentation of "A non-parametric-based computationally efficient approach for credit scoring using non-traditional data", at the 8th International Conference on Risk Analysis and Design of Experiments

Cite this

Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark

Abstract

Publication series

Conference

Keywords

UN SDGs

Access to Document

Other files and links

Embargoed Document

Fingerprint

Prizes

First Prize for the presentation of "A non-parametric-based computationally efficient approach for credit scoring using non-traditional data", at the 8th International Conference on Risk Analysis and Design of Experiments

Cite this