Training of MLP Neural Network for Data Classification by GSA Method
Subject Areas : electrical and computer engineering
M.
Dehbashian
1
Seyed-Hamid
Zahiri
2
(University of Birjand)
Keywords: Data classification gravitational search algorithm heuristic algorithms MLP neural network,
Abstract :
Nowadays, several techniques have presented for data classification. One of these techniques is neural network that has attracted many interests. In this classifier, selection a suitable learning method is very important for training of the network. Error back propagation is the most usual training method of neural networks that late convergence and stopping in local optimum points are its weakness. New approach in neural networks training is the usage of heuristic algorithms. This paper suggests a new learning method namely gravitational search algorithm (GSA) in training of neural network for data classification. GSA method is the latest and the most novel version of swarm intelligence optimization methods. This algorithm is inspired fby the law of Newtonian gravity and mass concept in nature. In this paper, a MLP neural network is trained for classification of five benchmark data set by GSA method. Also, the proposed method efficiency in training and testing of neural network compared with those of two training methods error back propagation and particle swarm optimization. Final results showed the GSA method extraordinary performance for data correct classification in most of cases. Also, in these experiments the GSA method produced stable results in all of cases. In addition, the run time of GSA method is shorter than that of the PSO.
[1] R. S. Kulkarni and M. Vidyasagar, "Learning decision rules for pattern classification under a family of probability measures," IEEE Trans. on Information Theory, vol. 43, no. 1, pp. 154-166, Jan. 1997.
[2] J. R. Qinlan, "Introduction of decision trees," Machine Learning, vol. 1, no. 1, pp. 86-101, Mar. 1986.
[3] Y. Freund, "Boosting a weak learning algorithm by majority," Information Computation, vol. 121, no. 2, pp. 256-285, Sep. 1995.
[4] C. Hongjun, R. Setiono, and L. Huan, "Effect data mining using neural networks," IEEE Trans. on Knowledge and Data Engineering, vol. 8, no. 6, pp. 957-961, Dec. 1996.
[5] B. E. Boser, I. M. Guyon, and V. N. Vapnik, "A training algorithm for optimal margin classifiers," in Proc. 5th Annual ACM Workshop on Computational Learning Theory, New York, NY, US, pp. 144-152, 13-15 Jun. 1992.
[6] V. N. Vapink, The Nature of Statistical Learning Theory, Springer Verlag, 1995.
[7] م. ب. منهاج، هوش محاسباتی، جلد اول: مبانی شبکه‌های عصبی، انتشارات دانشگاه صنعتی اميرکبير، تهران، ويرايش اول، 1379.
[8] ع. کاشفی کاویانی، س .ع. پورموسوی کانی و ع. جهانبانی اردکانی، "آموزش شبکه‌های عصبی چندلایه با به‌کارگیری الگوریتم PSO،" هشتمين كنفرانس سیستم‌های هوشمند، دانشگاه فردوسی، مشهد، شهریور 1386.
[9] A. A. Miniani and R. D. Williams, "Acceleration of back- propagation through learning rate and momentum adaptation," in Proc. Inter. Joint Conf. on Neural Networks, San Diego, CA, US, vol. 1, pp. 676-679, 1990.
[10] R. A. Jacobs, "Increased rate of convergence through learning rate adaptation," Neural Networks, vol. 1, no. 4, pp. 295-308, 1988.
[11] K. Balakrishnan and V. Honavar, "Improving convergence of back propagation by handling flat - spots in the output layer," in Proc. Second Int. Conf. on Artificial Neural Networks, Brighton, UK, vol. 1, pp. 139-144, 1992.
[12] M. T. Hagan and M. Menhaj, "Training feed forward networks with the Marquardt algorithm," IEEE Trans. on Neural Networks, vol. 5, no. 6, pp. 989-993, Nov. 1994.
[13] W. Yan and S. Hongbao, "Improvement of neural network learning algorithm and its application in control," in Proc. 3rd World Congress on Neural Networks, vol. 2, pp. 971-975, Hefei, Anhui, Jul. 2000.
[14] B. Bazartseren, G. Hildebrandt, and K. P. Holz, "Short - term water level prediction using neural networks and neuro - fuzzy approach," Neuro Computing, vol. 55, no. 3-4, pp. 439-450, Oct. 2003.
[15] M. Engin, "ECG beat classification using neuro - fuzzy network," Pattern Recognition Letters, vol. 25, no. 15, pp. 1715-1722, Nov. 2004.
[16] P. Kumar, S. N. Merchant, and U. B. Desai, "Improving performance in pulse radar detection using Bayesian regularization for neural network training," Digital Signal Processing, vol. 14, no. 5, pp. 438-448, Jul. 2004.
[17] K. S. Tang, K. F. Man, and S. Kwong, Q. He, "Genetic algorithms and their applications," IEEE Signal Processing Magazine, vol. 13, no. 6, pp. 22-37, Nov. 1996.
[18] L. L. Rogers, F. U. Dowla, and V. M. Johnson, "Optimal field - scale groundwater remediation using neural networks and the genetic algorithm," Environmental Science and Technology, vol. 29, no. 5, pp. 1145-1155, 1995.
[19] D. J. Montana and L. Davis, "Training feed forward neural networks using genetic algorithms," Machine Learning, vol. 1, pp. 762-767, Aug. 1989.
[20] J. Branke, "Genetic algorithms for neural network design and training," in Proc. of the First Nordic Workshop on Genetic Algorithms, vol. 1, pp. 145-163, University of Vaasa, Finland, Jan. 2003.
[21] A. Ullah Khan, T. K. Bandopadhyaya, and S. Sharma, "Genetic algorithm based back propagation neural network performs better than back propagation neural network in stock rates prediction," Inter. J. of Computer Science and Network Security, vol. 8, no. 7, pp. 162-166, Jul. 2008.
[22] L. A. Tarca, B. P. A. Grandjean, and F. Larachi, "Embedding monotonicity and concavity in the training of neural networks by means of genetic algorithms application to multiphase flow," Computers and Chemical Engineering, vol. 28, no. 9, pp. 1701-1713, 2004.
[23] R. S. Sexton and R. E. Dorsey, "Reliable classification using neural network: a genetic algorithm and back propagation computation," Decision Support Systems, vol. 30, no. 1, pp. 11-22, Dec. 2000.
[24] S. Kirkpatrick, C. D. Gelatto, and M. P. Vecchi, "Optimization by simulated annealing," Science, New Series, vol. 220, no. 4598, pp. 671-680, May. 1983.
[25] A. L. Arnaud, P. J. L. Adeodato, G. C. Vasconcelos, and R. F. O. Neto, "MLP neural networks optimization through simulated annealing in a hybrid approach for time series prediction," Congresso de Soceidade Brasileira de Computacäo, pp. 1110-1113, Jul. 2005.
[26] R. S. Sexton, R. E. Dorsey, and J. D. Johnson, "Optimization of neural networks: a comparative analysis of the genetic algorithm and simulated annealing," European J. of Operational Research, vol. 114, no. 3, pp. 589-601, May 1999.
[27] S. W. Lin, T. Y. Tseng, S. Y. Chou, and S. C. Chen, "A simulated-annealing - based approach for simultaneous parameter optimization and feature selection of back - propagation networks," Expert Systems with Applications, vol. 34, no. 2, pp. 1491-1499, Feb. 2008.
[28] J. D. Farmer, N. Packard, and A. Perelson, "The immune system, adaptation and machine learning," Physica D, vol. 2, no. 1-3, pp. 187-204, 1986
[29] R. Pasti and L. N. De Castro, "The influence of diversity in an immune-based algorithm to train MLP networks," in Proc. of the 6th Inter. Conf. on Artificial Immune Systems, ICARIS'07, vol. 4628, pp. 71-82, 2007.
[30] J. Kennedy and R. C. Eberhart, "Particle swarm optimization," in Proc. of IEEE Int. Conf. on Neural Networks, vol. 4, pp. 1942-1948, 1995.
[31] C. Marcio and B. L. Teresa, "An analysis of PSO hybrid algorithms for feed-forward neural networks training," in Proc. of the Ninth Brazilian Symp. on Neural Networks, SBRN'06, pp. 2-7, 2006.
[32] F. A. Guerra and L. D. S. Coelho, "Multi - step ahead nonlinear identification of Lorenz’s chaotic system using radial basis neural network with learning by clustering and particle swarm optimization," Chaos, Solitons and Fractals, vol. 35, no. 5, pp. 967-979, Mar. 2008.
[33] J. R. Zhang, J. Zhang, T. M. Lok, and M. R. Lyu, "A hybrid particle swarm optimization - back - propagation algorithm for feedforward neural network training," Applied Mathematics and Computation, vol. 185, no. 2, pp. 1026-1037, Feb. 2007.
[34] Y. P. Chang and C. N. Ko, "A PSO method with nonlinear time - varying evolution based on neural network for design of optimal harmonic filters," Expert Systems with Applications, vol. 36, no. 3, pp. 6809-6816, Apr. 2009.
[35] R. Pasti and L. N. De Castro, "Bio - inspired and gradient - based algorithms to train MLPs: the influence of diversity," Information Sciences, vol. 179, no. 10, pp. 1441-1453, Apr. 2009.
[36] A. E. Zade Sherme and R. Ghaderi, "An intelligent system for classification of the communication formats using PSO," Informatica, vol. 32, no. 2, pp. 213-218, Jul. 2008.
[37] Y. Marinakis, M. Marinaki, M. Doumpos, and C. Zopounidis, "Ant colony and particle swarm optimization for financial classification problems," Expert Systems with Applications, vol. 36, no. 7, pp. 10604-10611, Sep. 2009.
[38] E. Rashedi, H. Nezamabadi - pour, and S. Saryazdi, "GSA: a gravitational search algorithm," Information Sciences, vol. 179, no. 13, pp. 2232-2248, Mar. 2009.
[39] ع. راشدي و ح. نظام‌آبادي‌پور، "طراحی فیلترهای IIR به وسیله الگوریتم جستجوی گرانشی،" شانزدهمين كنفرانس مهندسي برق ايران، دانشگاه تربیت مدرس، جلد اول، صص. 474-469، تهران، اردیبهشت 1387.
[40] ع. راشدي و ح. نظام‌آبادي‌پور، "انتخاب ویژگی با استفاده از الگوریتم جستجوی گرانشی،" سومین كنفرانس فناوری اطلاعات و دانش، دانشگاه فردوسی، مشهد، آذر 1386.
[41] E. Rashedi, H. Nezamabadi - pour, and S. Saryazdi, "Allocation of static var compensator using gravitational search algorithm," First Joint Congress on Fuzzy and Intelligent Systems Ferdowsi University of Mashhad, Iran, Aug. 2007.
[42] م. ده‌باشیان، سیدحمید ظهیری و ن. مهرشاد، "آموزش شبکه عصبی MLP در فشرده‌سازی تصاویر با استفاده از روش GSA،" ششمین کنفرانس ماشین بینایی و پردازش تصویر ایران، دانشگاه اصفهان، جلد اول، صص. 476-470، آبان 1389.
[43] ع. راشدی، ح. نظام‌آبادی‌پور و سعید سریزدی، "الگوریتم جستجوی گرانشی باینری،" هشتمین کنفرانس سیستم‌های هوشمند، دانشگاه فردوسی مشهد، شهریور 1386.
[44] م. ده‌باشیان و سیدحمید ظهیری، "الگوریتم جستجوی گرانشی نخبه‌گرای پیشرفته،" اولین کنفرانس انرژی‌های تجدیدپذیر و تولید پراکنده ایران، دانشگاه بیرجند، اسفند 1388.
[45] م. ده‌باشیان و س. ح. ظهیری، "MOGSA: روشی جدید در بهینه‌سازی چندهدفه مبتنی بر الگوریتم جستجوی گرانشی،" شانزدهمین کنفرانس انجمن کامپیوتر ایران، دانشگاه صنعتی شریف، جلد اول، صص 507-502، اسفند 1389.
[46] D. E. Rumelhart, E. Hinton, and J. Williams, "Learning internal representation by error propagation," Parallel Distributed Processing, vol. 1, 318-362, 1986.
[47] UCI Repository of Machine Learning Data Bases, Department of Information and Computer Science, University of California, Irvine, CA, http://www.ics.uci.edu/~mlearn/MLRepository.html.