Artificial intelligence

Abstract: Modern approaches in deep neural networks have a number of issues related to the learning process and computational costs. This article considers the architecture grounded on an alternative approach to the basic unit of the neural network. This approach achieves optimization in the calculations and gives rise to an alternative way to solve the problems of the vanishing and exploding gradient. The main issue of the article is the usage of the deep stacked neo-fuzzy system, which uses a generalized neo-fuzzy neuron to optimize the learning process. This approach is non-standard from a theoretical point of view, so the paper presents the necessary mathematical calculations and describes all the intricacies of using this architecture from a practical point of view. From a theoretical point, the network learning process is fully disclosed. Derived all necessary calculations for the use of the backpropagation algorithm for network training. A feature of the network is the rapid calculation of the derivative for the activation functions of neurons. This is achieved through the use of fuzzy membership functions. The paper shows that the derivative of such function is a constant, and this is a reason for the statement of increasing in the optimization rate in comparison with neural networks which use neurons with more common activation functions (ReLU, sigmoid). The paper highlights the main points that can be improved in further theoretical developments on this topic. In general, these issues are related to the calculation of the activation function. The proposed methods cope with these points and allow approximation using the network, but the authors already have theoretical justifications for improving the speed and approximation properties of the network. The results of the comparison of the proposed network with standard neural network architectures are shown.

Keywords: deep stacking neural network, neo-fuzzy neuron, multilayer neural network, F-transform.

References:

1. Bengio Y, LeCun Y, Hinton G. Deep Learning – Nature – 2015-521 – p.436-444.
2. Schmidhuber J Deep learning in neural networks: An overview – Neural Networks – 2015–01 – p.85-117.
3. Googfellow I, Bengio Y, Courville A. Deep Learning – MIT Press, 2016-787p.
4. Graupe D. Deep Learning Neural Networks: Design and Case Studies- New York: World Scientific, 2016 – 260p.
5. Caterini A.L., Chang D.E. Deep Neural Networks in a Mathematical Framework – Springer, 2018 –79p.
6. Cichocki A, Unbehauen R. Neural Networks for Optimization and Signal Processing – Stuttgart: Teubner, 1993-526p.
7. Cybenko G. Approximation by superpositions of a sigmoidal function – Math. Control Signals Systems. – 1985 – 2 – p.303-314.
8. Hornik K. Approximation capabilities of multilayer feedforward networks – Neural Networks, - 1994 – 4 – p.251-257.
9. Aggarwal Ch.C. Neural Networks and Deep Learning –Springer, 2018-512p.
10. Yamakawa T, Uchino E, Miki T., Kusanagi H. A neo fuzzy neuron and its applications to system identification and predictions to system behavior. – Proc. 2nd Int. Conf. on Fuzzy Logic and Neural Networks, pp. 477-483, 1992.
11. Uchino E, Yamakava T. Neo-fuzzy neuron based new approach to system modeling with application to actual system - Proceedings Sixth International Conference on Tools with Artificial Intelligence – New Orlean, LA, USA, 1994 – p.564-570.
12. Miki T, Yamakawa T, “Analog implementation of neo-fuzzy neuron and its on-board learning,” In Computational Intelligence and Applications, Piraeus: WSES Press, 1999, pp. 144-149.
13. Kolodyazhniy V, Bodyanskiy Ye. Fuzzy Kolmogorov's network – Lecture Notes in Computer Science. – 3214 – Heidelberg: Springer Verlag, 2004. – p.764-771.
14. Bodyanskiy Ye, Kolodyazhniy V, Otto P. Neuro-fuzzy Kolmogorov's network for time series prediction and pattern classification – Lecture Notes in Artificial Intelligence – 3698 – Heidelberg: Springer Verlog, 2005. – p.191-202.
15. Bodyanskiy Ye,Popov S, Rybalchenko T. Multilayer neuro-fuzzy network for short term electric load forecasting – Lecture Notes in Computer Science. – 5010 – Berlin-Heidelberg: Springer Verlag, 2008. – p.339-348.
16. Bodyanskiy Ye,Vynokurova O, Setlak G, Peleshko D, Mulesa P. Adaptive multivariate hybrid neuro-fuzzy system and its on-board fast learning – Neurocomputing – 2017 – 230-p.409-416.
17. Perfilieva I. Fuzzy transforms: Theory and applications – Fuzzy Sets and Systems – 2006 – 157 – p.993-1023.
18. Bodyanskiy Ye, Kolodyazhniy V, Stephan A. An adaptive learning algorithm for a neuro-fuzzy network - Ed. by B.Reush “Computitional Intelligence. Theory and Application” – Berlin-Heidelberg: Ney York: Springer, 2001. – p.68-75.
19. Otto P, Bodyanskiy Ye, Kolodyazhniy V. A new learning algorithm for a forecasting neuro-fuzzy network - Integrated Computer Aided Engineering – 2003 – 10(4) – p.399-409.