Effect of Hidden Neuron Size on Different Training Algorithm in Neural Network
DOI:
https://doi.org/10.26713/cma.v13i1.1680Keywords:
Neural Network, Training algorithm, Pattern network, Levenberg_Marquardt (LM), Bayesian regularization backpropagation (BR)Abstract
We use different types of training algorithms in the neural network. But, we cannot say which kind of training algorithm is fast for a given problem. So, in this survey paper, we are trying to find which types of training algorithm are better for categorization problems. For this purpose, we used ten types of training algorithms in the pattern network in MATLAB. We used Levenberg-Marquardt (LM), Bayesian regularization backpropagation (BR), BFGS Quasi-Newton (BFG), Resilient Backpropagation (RP), Scaled Conjugate gradient backpropagation (SCG), Conjugate Gradient with Powell/Beale Restarts (CGB), Fletcher-Powell Conjugate Gradient (CGF), Polak-Ribiere Conjugate Gradient (GDM), One Step Secant (OSS), and Variable Learning Rate Backpropagation (GD) algorithm. In this survey paper, we also check, affects of these algorithms on neural network when we applied different types of hidden neuron size. During this survey we found some new facts. We found that RP, SCG, CGB, CGF and OSS are fastest algorithms. BFG takes more time with respect to hidden neuron size. GDM and GD take more epochs. BR algorithm is not acceptable for image categorization.
Downloads
References
A.R. Abd-Ellah, M.H. Essai and A. Yahya, Comparison of different backpropagation training algo-rithms using robust M-estimators performance functions, IEEE 10th International Conference on Computer Engineering & Systems (ICCES), (2015), 384 – 388, DOI: 10.1109/ICCES.2015.7393080.
C.K. Arthur, V.A. Temeng and Y.Y. Ziggah, Performance evaluation of training algorithms in backpropagation neural network approach to blast-induced ground vibration prediction, Ghana Mining Journal 20(1) (2020), 20 – 33, DOI: 10.4314/gm.v20i1.3.
K. Atanassov, M. Krawczak and S. Sotirov, Generalized net model for parallel optimization of feed-forward neural network with variable learning rate backpropagation algorithm, IEEE 4th International IEEE Conference Intelligent Systems, (2008), 2, 16, DOI: 10.1109/IS.2008.4670540.
C. Baigen, Y. Jiaming, S. Weif and C. Bin, Research on fault recognition method of on-board equipment based on BP neural network optimized by Bayesian regularized, IEEE Chinese Automation Congress (CAC), (2017), 6835 – 6840, DOI: 10.1109/CAC.2017.8244008.
R. Bhati, S. Jain, N. Maltare and D.K. Mishra, A comparative analysis of different neural networks for face recognition using principal component analysis, wavelets and efficient variable learning rate, IEEE International Conference on Computer and Communication Technology (ICCCT), (2010), 526 – 531, DOI: 10.1109/ICCCT.2010.5640486.
D. Chang, S. Sun and C. Zhang, An accelerated linearly convergent stochastic L-BFGS algorithm, IEEE Transactions on Neural Networks and Learning Systems 30(11) (2019), 3338 – 3346, DOI: 10.1109/TNNLS.2019.2891088.
Y. Cui, Z. Yi, J. Duan, D. Shi and Z. Wang, A Rprop-Neural-Network-Based PV maximum power point tracking algorithm with short-circuit current limitation, IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT), (2019), 1 – 5, DOI: 10.1109/ISGT.2019.8791596.
F. Dalipi and S.Y. Yayilgan, The impact of environmental factors to skiing injuries: Bayesian regularization neural network model for predicting skiing injuries, IEEE 6th International Conference on Computing, Communication and Networking Technologies (ICCCNT), (2015), 1 – 6, DOI: 10.1109/ICCCNT.2015.7395218.
J.E. Dennis and J.R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, SIAM Publications, Philadelphia, (1996), URL: https://books.google.co.in/books?hl=en&lr=&id=ksvJTtJCx9cC&oi=fnd&pg=PR1&ots=BKmNCQ5Hxq&sig=g7DGZc4S60CQTlCaXQcOfEcXEYQ#v=onepage&q&f=false.
D.K. Destiani and D.Q. Utama, Study of wavelet and line search techniques on modified backpropagation Polak-Ribiere algorithm for heart failure detection, IEEE 6th International Conference on Information and Communication Technology (ICoICT), (2018), 319 – 323, DOI: 10.1109/ICoICT.2018.8528790.
R. Fletcher and C.M. Reeves, Function minimization by conjugate gradients, The Computer Journal 7(2) (1964), 149 – 154, DOI: 10.1093/comjnl/7.2.149.
M. Forouzanfar, H.R. Dajani, V.Z. Groza, M. Bolic and S. Rajan, Comparison of feed-forward neural network training algorithms for oscillometric blood pressure estimation, IEEE 4th International Workshop on Soft Computing Applications, (2010), 119 – 123, DOI: 10.1109/SOFA.2010.5565614.
T. Gao, J. Wang, B. Zhang, H. Zhang, P. Ren and N.R. Pal, A Polak-Ribière-Polyak conjugate gradient-based neuro-fuzzy network and its convergence, IEEE Access 6 (2018), 41551 – 41565, DOI: 10.1109/ACCESS.2018.2848117.
I. Goodfellow, Y. Bengio and A. Courvlle, Deep Learning, MIT Press (2016), URL: https://mitpress.mit.edu/books/deep-learning.
M.T. Hagan, Neural Network Design, 2nd edition, Kindle e-Book, (2014), URL: https://hagan.okstate.edu/NNDesign.pdf.
A.N. Handayani, N. Lathifah, H.W. Herwanto, R.A. Asmara and K. Arai, Neural network bayesian regularization backpropagation to solve inverse kinematics on planar manipulator, IEEE 7th International Conference on Informatics, Electronics & Vision (ICIEV) and 2nd International Conference on Imaging, Vision & Pattern Recognition (icIVPR), (2018), 99 – 104, DOI: 10.1109/ICIEV.2018.8640958.
A.K.I. Hassan and A. Abraham, Modeling consumer loan default prediction using ensemble neural networks, IEEE International Conference on Computing, Electrical and Electronic Engineering (ICCEEE), (2013), 719 – 724, DOI: 10.1109/ICCEEE.2013.6634029.
S. Haykin, Neural Networks and Learning Machines, 3rd edition, Pearson Prentice Hall (2009), URL: https://www.pearson.com/us/higher-education/program/Haykin-Neural-Networks-and-Learning-Machines-3rd-Edition/PGM320370.html.
Y. Ju, J. Wang, Z. Zhang, Y. Huang and Y. Lin, A calculation method for three-phase power flow in micro-grid based on smooth function, IEEE Transactions on Power Systems 35(6) (2020), 4896 – 4903, DOI: 10.1109/TPWRS.2020.2995521.
C.B. Khadse, M.A. Chaudhari and V.B. Borghate, Comparison of seven backpropagation algorithms for three phase power quality assessment, IEEE TENCON IEEE Region 10 Conference, (2017), 2548 – 2553, DOI: 10.1109/TENCON.2017.8228291.
T. Kliment, P. Lipovsky, K. Draganova and V. Moucha, Comparison of Speed and Accuracy of Chosen Magnetometer Calibration Algorithms in the Presence of External Interference, IEEE New Trends in Signal Processing (NTSP), (2018), 1 – 5, DOI: 10.23919/NTSP.2018.8524105.
S. Leholo, P. Owolawi and K. Akindeji, Solar Energy Potential Forecasting and Optimization Using Artificial Neural Network: South Africa Case Study, IEEE Amity International Conference on Artificial Intelligence (AICAI), (2019), 533-536, DOI: 10.1109/AICAI.2019.8701372.
F. Li, L. Ge, W. Dan, Y. Gu, Q. He and K. Sun, Application of improved variable learning rate back propagation neural network in energy dispersion x-ray fluorescence quantitative analysis, IEEE 4th International Conference on Cloud Computing and Big Data Analysis (ICCCBDA), (2019), 428 – 432, DOI: 10.1109/ICCCBDA.2019.8725682.
S.Y. Liang and Y.T. Ning, Parameter optimization of load frequency control system composed of hydroelectric and thermal power units based on Levenberg-Marquardt algorithm, IEEE 5th Asia Conference on Power and Electrical Engineering (ACPEE), (2020), 75 – 80, DOI: 10.1109/ACPEE48638.2020.9136561.
J. Lv, S. Deng and Z. Wan, An efficient single-parameter scaling memoryless Broyden-Fletcher-Goldfarb-Shanno algorithm for solving large scale unconstrained optimization problems, IEEE Access 8 (2020), 85664 – 85674, DOI: 10.1109/ACCESS.2020.2992340.
D.W. Marquardt, An algorithm for least-squares estimation of nonlinear parameters, Journal of the Society for Industrial and Applied Mathematics 11(2) (1963), 431 – 441, DOI: 10.1137/0111030.
M.F. Møller, A scaled conjugate gradient algorithm for fast supervised learning, Neural Networks 6(1993), 525 – 533, DOI: 10.1016/S0893-6080(05)80056-5.
A. Payal, C.S. Rai and B.V.R. Reddy, Comparative analysis of Bayesian regularization and Levenberg-Marquardt training algorithm for localization in wireless sensor network, IEEE 15th International Conference on Advanced Communications Technology (ICACT), (2013), 191 – 194, URL: https://ieeexplore.ieee.org/abstract/document/6488169.
Perona Lab, http://www.vision.caltech.edu (Photos/Pictures), Caltech Vision Lab, (2022).
M.J.D. Powell, Restart procedures for the conjugate gradient method, Mathematical Programming 12(1) (1977), 241 – 254, DOI: 10.1007/BF01593790.
M.S. Pratiwi and A. Aditsania, Cancer detection based on microarray data classification using Genetic Bee Colony (GBC) and conjugate Gradient Backpropagation with modified Polak Ribiere (MBP-CGP), IEEE International Conference on Computer, Control, Informatics and its Applications (IC3INA), (2018), 163 – 168, DOI: 10.1109/IC3INA.2018.8629538.
M. Riedmiller and H. Braun, A direct adaptive method for faster backpropagation learning: the RPROP algorithm, IEEE International Conference on Neural Networks, San Francisco, CA, USA, (1993), (1), 586 – 591, DOI: 10.1109/ICNN.1993.298623.
B. Roberto, First- and second-order methods for learning: between steepest descent and Newton’s method, Neural Computation 4(2) (1992), 141 – 166, DOI: 10.1162/neco.1992.4.2.141.
S. Särkkä and L. Svensson, Levenberg-Marquardt and line-search extended Kalman smoothers, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (2020), 5875 – 5879, DOI: 10.1109/ICASSP40776.2020.9054686.
L.E. Scales, Introduction to Non-Linear Optimization, Macmillan International Higher Education (1985), DOI: 10.1007/978-1-349-17741-7.
The MathWorks, Inc., Deep Learning Toolbox https://www.mathworks.com/help/deeplearning, (2022).
P.K. Upadhyay and C. Nagpal, SCG backpropagation based prediction of stressed EEG spectrum, IEEE Advances in Science and Engineering Technology International Conferences (ASET), (2020), 1 – 5, DOI: 10.1109/ASET48392.2020.9118324.
P.K. Upadhyay, A. Pandita and N. Joshi, Scaled conjugate gradient backpropagation based SLA violation prediction in cloud computing, IEEE International Conference on Computational Intelligence and Knowledge Economy (ICCIKE), (2019), 203 – 208, DOI: 10.1109/ICCIKE47802.2019.9004240.
J. Wang, B. Liu, C. Zhang, F. Yang, T. Zhang and X. Miao, Partial discharge type identification in GIS based on SCG algorithm, IEEE 3rd International Electrical and Energy Conference (CIEEC), (2019), 511 – 515, DOI: 10.1109/CIEEC47146.2019.CIEEC-2019213.
U.N. Wisesty, Comparative study of conjugate gradient to optimize learning process of neural network for Intrusion Detection System (IDS), IEEE 3rd International Conference on Science in Information Technology (ICSITech), (2017), 459 – 464, DOI: 10.1109/ICSITech.2017.8257156.
F. Yang, Z. Li, H. Li, H. Huang and Z. Pan, Method of neural network modulation recognition based on clustering and Polak-Ribiere algorithm, Journal of Systems Engineering and Electronics 25(5) (2014), 742 – 747, DOI: 10.1109/JSEE.2014.00085.
Z. Yumei, B. Shulin, C. Ping and Q. Shiru, A novel second-order DFP-based volterra filter and its applications to chaotic time series prediction, in: Proceedings of the IEEE 31st Chinese Control Conference, (2012), 1036 – 1039, URL: https://ieeexplore.ieee.org/abstract/document/6390076.
Y. Zhang and S Bai, A novel adaptive filter algorithm based on DFP technique, in: Proceedings of the IEEE 30th Chinese Control Conference, (2011), 1688 – 1691, URL: https://ieeexplore.ieee.org/abstract/document/6001590.
Y. Zhang, S. Bai, G. Lu and X. Wu, Kernel estimation of truncated Volterra filter model based on DFP technique and its application to chaotic time series prediction, Chinese Journal of Electronics 28(1) (2019), 127 – 135, DOI: 10.1049/cje.2018.04.014.
R. Zhao, W.B. Haskell and V.Y. Tan, Stochastic L-BFGS: Improved convergence rates and practical acceleration strategies, IEEE Transactions on Signal Processing 66(5) (2018), 1155 – 1169, URL: http://www.auai.org/uai2017/proceedings/papers/86.pdf.
N.S.A. Zubir, M.A. Abas, N. Ismail, N.A.M. Ali, M.H.F. Rahiman, N.K. Mun and N.T. Saiful, Pattern classifier of chemical compounds in different qualities of agarwood oil parameter using scale conjugate gradient algorithm in MLP, IEEE 13th International Colloquium on Signal Processing & its Applications (CSPA), (2017), 18 – 22, DOI: 10.1109/CSPA.2017.8064917.
Downloads
Published
How to Cite
Issue
Section
License
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a CCAL that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
