Artificial intelligence

Abstract: The technology, which is implemented with cross platform tools, is proposed for modeling of gesture units of sign language, animation between states of gesture units with a combination of gestures (words). Implemented technology simulates sequence of gestures using virtual spatial hand model and performs recognition of dactyl items from camera input using trained on collected training dataset set convolutional neural network, based on the MobileNetv3 architecture, and with the optimal configuration of layers and network parameters. On the collected test dataset accuracy of over 98% is achieved.

Keywords: cross platform, sing language, dactyl modeling, dactyl recognition, convolutional neural net-works, mobilenet

References:

1. Mell, P., Grance, T. (2011). The NIST Definition of Cloud Computing (Technical report). National Institute of Standards and Technology: U.S. Department of Commerce. doi: 10.6028/NIST.SP.800-145
2. The Linux Information Project, Cross-platform Definition.
3. Smith, J., Nair, R. (2005). The Architecture of Virtual Machines. Computer. IEEE Computer Society. 38 (5): 32–38.
4. Krak, I., Kondratiuk, S. (2017). Cross-platform software for the development of sign communication system: Dactyl language modelling, Proceedings of the 12th International Scientific and Technical Conference on Computer Sciences and Information Technologies, CSIT, 1, 167-170. DOI: 10.1109/STC-CSIT.2017.8098760
5. Krak, Y.G., Krak, Y.V., Barchukova, B.A. (2011). Human hand motion parametrization for dactylemes modeling, Journal of Automation and Information Sciences, 43 (12), 1-11.
6. Krizhevsky, A., Sutskever, I., Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 1097-1105.
7. Karpathy, A., Toderici, G., Shetty, S., Leung, T.,Sukthankar, R., Fei-Fei, L. (2014). Large-scale video classification with convolutional neuralnetworks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 1725-1732.
8. Zisserman, A., Carriera, J. (2017). Action recognition a new model and the kinetics dataset. In Computer Vision and Pattern Recognition (CVPR). IEEE Conference on pages 4724-4733, IEEE.
9. Materzynska, J., Berger, G., Bax, I., Memisevic, R. (2019). The Jester Dataset: A Large-Scale Video Dataset of Human Gestures.
10. Haha, K., Kataoka, H., Satoh, Y. (2018). Canspatiotemporal 3D CNNs retrace the history of 2DCNNs and ImageNet. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognitions, Salt Lake City, UT, USA, 18-22.
11. Freeman, W., Roth, M. (1995). Orientation histograms for hand gesture recognition. In International workshop on automatic face and gesture recognition, vol. 12, 296-301.
12. Prasuhn, L., Oyamada, Y., Mochizuki, Y., Ishikawa, H. (2014). A HOG-based had gesture recognition system on a mobile device. In 2014 IEEE International Conference on Image Processing (ICIP), 3973-3977, IEEE.
13. Dardas, N., Georganas, D. (2011). Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Transactions on instrumentation and measurement, 60: 3592-3607.
14. Kopuklu, O., Kose, N., Rigoll, G. (2018). Motion fused frames: Data level fusion strategy for hand gesture recognition. arXiv preprint arXiv:1804.07187
15. Molchanov, P., Gupta, S., Kim, K., Pulli. K. (2015). Multi-sensor system for driver’s hand-gesture recognition. In Automatic Face and Gesture Recognition (FG), 11th IEEE International Conference and Workshops on, vol. 1, 1–8. IEEE
16. Molchanov, P., Gupta, S., Kim, K., Kautz, J. (2015). Hand gesture recognition with 3d convolutional neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops
17. Hu, J., Shen, L., Sun, G. (2018) Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 7132–7141.
18. He, K., Zhang, X., Ren, S., Sun., J. (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
19. Iandola, F., Han, S., Moskewicz, M., Ashraf, K., Dally, W., Keutzer, K. (2016). Squeezenet: AlexNet-level accuracy with 50x fewer parameters and 0.5 mb model size. arXiv preprint arXiv:1602.07360.
20. Howard, A., Zhu, M., Chen, B., Kalenichenko, D.,Wang, W., Weyand, T., Andreetto, M., Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861
21. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4510–4520. IEEE.
22. Zhang, X., Zhou, X., Lin, M., Sun, J. (2018). Shufflenet: An extremely efficient convolutional neural network for mobile devices. In 2018IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6848–6856. IEEE.
23. Ma, N., Zhang, X., Zheng, H., Sun, J., (2018). Shufflenet v2: Practical guidelines for efficient CNN architecture design. arXiv preprint arXiv:1807.11164, 5.
24. Howard, A., Sandler, M., Chu, G., Chen, L.,Chen, B., Tan, M., Wang, W. (2019). Searching for MobileNetV3. axXiv:1905.02244, 5
25. ASL Sing language dictionary. URL: http://www.signasl.org/sign/model
26. Unity3D framework. URL: https://unity3d.com/
27. Tensorflow framework documentation. URL: https://www.tensorflow.org/api/
28. Ong, E. (2012). Sign language recognition using sequential pattern trees. In: Computer Vision andPattern Recognition (CVPR), IEEE Conference on IEEE pp. 2200-2207.
29. American Sign language: Real-time American Sign Language Recognition with Convolutional Neural Networks (2015). Brandon Garcia Stanford University Stanford, CA.
30. Bobic, V. (2016). Hand gesture recognition using neural network based techniques, School of Electrical Engineering, University of Belgrade
31. PostgreSQL official web site. URL: https://www.postgresql.org/