Training Multilayer Perceptrons with Threshold Activation Functions with Mixed Integer Linear Programming

- In the last four years we developed techniques to solve simple nonlinear problems with linear optimisation models like Mixed Integer Linear Programming (MILP) [1] and then we reach the conclusion that we can use those techniques to solve a much more complex nonlinear problem: the calculation of the weights of a hardlimit multilayer perceptron (HMLP), an open problem since the work Perceptrons of Minsky and Papert [2]. We describe the MIP model that solves this problem implemented with the GAMS software and then we apply it to the three variable XOR and we obtained a solution with only three hardlimit neurons. We also made an experiment with the function sin(x)/x, x varying between 0 and 120p, x initialised in 120p, and in few seconds our adapted MIP model converged to the global maximum at 0.