ANAL YSIS OF TRAINED NEURALNETWORKS

Neural Networks are typically thought of as black boxes trained to a specific task on a large number of data samples. In many applications it becomes necessary to “look inside” of these black boxes before they can be used in practice. This is done in case of high risk applications or applications with a limited number of training samples. This paper describes several techniques of analysis of trained networks, which can be used to verify that the networks meet requirements of the application. The two main approaches advocated are sensitivity analysis and analysis by means of graphs and polytopes. The algorithms developed for the neural network analysis have been coded in form of a Matlab toolbox.