Concluding remarks
Contents
Concluding remarks¶
In a programmer’s life, building a well-functioning ANN, even for simple problems as used for illustrations in these lectures, can be a truly frustrating experience! There are many subtleties involved on the way. To list just a few that we have encountered in this course, one faces a choice of the network architecture and freedom in the initialization of weights (hyperparameters). Then, one has to select an initial learning speed, a neighborhood distance, in general, some parameters controlling the performance/convergence, as well as their update strategy as the algorithm progresses. Further, one frequently tackles with an emergence of a massive number of local minima in the space of hyperparameters, and many optimization methods may be applied here, way more sophisticated than our simplest steepest-descent method. Moreover, a proper choice of the neuron activation functions is crucial for success, which relates to avoiding the problem of “dead neurons”. And so on and on, many choices to be made before we start gazing in the screen in hopes that the results of our code converge …
Important
Taking the right decisions for the above issues is an art more than science, based on long experience of multitudes of code developers and piles of empty pizza boxes!
Now, having completed this course and understood the basic principles behind the simplest ANNs inside out, the reader may safely jump to using professional tools of modern machine learning, with the conviction that inside the black boxes there sit essentially the same little codes he met here, but with all the knowledge, experience, tricks, provisions, and options built in. Achieving this conviction, through appreciation of simplicity, has been one of the guiding goals of this course.
Acknowledgments¶
The author thanks Jan Broniowski for priceless technical help and for remarks to the text.
- Bar16
P. Barry. Head First Python: A Brain-Friendly Guide. O'Reilly Media, 2016. ISBN 9781491919491. URL: https://books.google.pl/books?id=NIqNDQAAQBAJ.
- BH69
A. E. Bryson and Y.-C. Ho. Applied optimal control: Optimization, estimation, and control. Waltham, Mass: Blaisdell Pub. Co., 1969.
- FR13
J. Feldman and R. Rojas. Neural Networks: A Systematic Introduction. Springer Berlin Heidelberg, 2013. ISBN 9783642610684.
- Fre93
James A. Freeman. Simulating Neural Networks with Mathematica. Addison-Wesley Professional, 1993. ISBN 9780201566291.
- FS91
James A. Freeman and David M. Skapura. Neural Networks: Algorithms, Applications, and Programming Techniques (Computation and Neural Systems Series). Addison-Wesley, 1991. ISBN 9780201513769.
- Gut16
John Guttag. Introduction to computation and programming using Python: With application to understanding data. MIT Press, 2016.
- HR72
K. K. Hartline and F. Ratcliff. Inhibitory interactions in the retina of limulus. Handbook of sensory physiology, VII(2):381–447, 1972.
- KSJ+12
E.R. Kandel, J.H. Schwartz, T.M. Jessell, S.A. Siegelbaum, and A.J. Hudspeth. Principles of Neural Science, Fifth Edition. McGraw-Hill Education, 2012. ISBN 9780071810012. URL: https://books.google.pl/books?id=Z2yVUTnlIQsC.
- Mat19
E. Matthes. Python Crash Course, 2nd Edition: A Hands-On, Project-Based Introduction to Programming. No Starch Press, 2019. ISBN 9781593279295. URL: https://books.google.pl/books?id=boBxDwAAQBAJ.
- MP43
Warren S. McCulloch and Walter Pitts. The logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4):115–133, 1943. URL: https://doi.org/10.1007/BF02478259.
- MullerRS12
B. Müller, J. Reinhardt, and M.T. Strickland. Neural Networks: An Introduction. Physics of Neural Networks. Springer Berlin Heidelberg, 2012. ISBN 9783642577604. URL: https://books.google.pl/books?id=on0QBwAAQBAJ.
- RIV91
A. K. Rigler, J. M. Irvine, and Thomas P. Vogl. Rescaling of variables in back propagation learning. Neural Networks, 4(2):225–229, 1991. URL: https://doi.org/10.1016/0893-6080(91)90006-Q, doi:10.1016/0893-6080(91)90006-Q.