Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Recently, Big Data has become an increasingly important source to support traditional credit scoring. Personal credit evaluation based on machine learning approaches focuses on the application data of clients in open banking and new banking platforms with challenges about Big Data quality and model risk. This paper represents a PySpark code for computationally efficient use of statistical learning and machine learning algorithms for the application scenario of personal credit evaluation with a performance comparison of models including logistic regression, decision tree, random forest, neural network, and support vector machine. The findings of this study reveal that the logistic regression methodology represents a more reasonable coefficient of determination and a lower false negative rate than other models. Additionally, it is computationally less expensive and more comprehensible. Finally, the paper highlights the steps, perils, and benefits of using Big Data and machine learning algorithms in credit scoring.
Original languageEnglish
Title of host publicationStatistical Modeling and Simulation for Experimental Design and Machine Learning Applications
Subtitle of host publicationSelected Contributions from SimStat 2019 and Invited Papers
EditorsJürgen Pilz, Viatcheslav B. Melas, Arne Bathke
Place of PublicationGewerbestrasse, Switzerland
PublisherSpringer, Cham
Number of pages21
ISBN (Electronic)978-3-031-40055-1
ISBN (Print)978-3-031-40054-4, 978-3-031-40057-5
Publication statusPublished - 19 Oct 2023
Event10th International Workshop on Simulation and Statistics - Faculty of Natural Sciences Hellbrunner Strasse 34 5020, Salzburg, Austria
Duration: 2 Sept 20236 Sept 2023

Publication series

NameContributions to Statistics
PublisherSpringer Cham
ISSN (Print)1431-1968


Conference10th International Workshop on Simulation and Statistics
Abbreviated titleSimStat 2019
Internet address


  • Credit score
  • Big Data
  • Machine learning
  • Risk Management
  • Finance


Dive into the research topics of 'Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark'. Together they form a unique fingerprint.

Cite this