{"id":1084,"date":"2021-02-18T08:36:45","date_gmt":"2021-02-18T08:36:45","guid":{"rendered":"https:\/\/www.gravityjack.com\/?p=1084"},"modified":"2023-11-28T13:33:25","modified_gmt":"2023-11-28T13:33:25","slug":"understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning","status":"publish","type":"post","link":"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/","title":{"rendered":"UNDERSTANDING YOUR NEURAL NETWORK \u2013 PART 4: CLOSING THOUGHTS ON MACHINE LEARNING"},"content":{"rendered":"\n

You made it to the Neural Network series finale! It\u2019s been a fascinating journey, but there\u2019s still work to be done and questions to answer. So saddle up, grab a drink, and let\u2019s jump right in. The Jupyter notebooks mentioned throughout can be found here<\/a>.<\/p>\n\n\n\n

Experiment 1-1: Less Filters<\/strong><\/h5>\n\n\n\n

It\u2019s often the case that clients want their networks deployed on mobile devices and ask to keep the networks as small as possible. So, potentially cutting down on the number of filters could be a huge win. Inspecting experiments\/experiment1\/ex1-1<\/strong> will reveal several prepared notebooks. In ex1-1-train.ipynb<\/strong>, the number of filters per convolution layer was reduced by half. This yields a network with half the amount of parameters than the original model. <\/p>\n\n\n\n

The results on the testing set derived from CMU faces look great, 100%. Looking at ex1-1-test.ipynb<\/strong>, it looks like the tests on Gilbert, Ryan, Sarah, (the three amigos from Part 2<\/a>) are also about the same. So, on one hand there\u2019s a network that performs as well on the CMU faces data as our original network from experiment0<\/strong>. On the other hand, it performs as poorly as the experment0 network on the three amigos. Now, let us look at the visualization results. The entire set of visualizations can be seen in view_layers.ipynb<\/strong>. <\/p>\n\n\n

\n
\"\"
Image 1a: These HappyMaps for the experiment 1-1 decision layer look very familiar.<\/figcaption><\/figure><\/div>\n\n\n

The first thing to notice is that the HappyMaps for dense_3<\/strong>, the decision layer, look about the same as they do for experiment0<\/strong>\u2019s <\/strong>HappyMaps. This is a confirmation that the decisions made by the two networks are about the same. Before feature visualization, we would have to check the results on individual examples to see how they corresponded. With feature visualization, it\u2019s clear from the get-go that yes, the decisions are about the same. So now there\u2019s a network that is about half as big as the original AND has learned roughly the same thing. What\u2019s more, there\u2019s plenty of evidence to attest to that fact. <\/p>\n\n\n

\n
\"\"
Image 1b: Convolutional layer maps, also looking familiar.<\/figcaption><\/figure><\/div>\n\n\n

Looking at the conv layer happy maps reveals a couple of things. The first is that the set of HappyMaps produced for both networks are also about the same. The biggest difference in the new set is that there are far fewer of the static or \u201call on\u201d maps\u2013 confirming the suspicion that these types of neurons were not useful. <\/p>\n\n\n\n

Experiment 1-2: Deeper Dense Layers<\/strong><\/h5>\n\n\n\n

Our team has often noticed that Google gets incredibly clear pictures of their features, and it makes us wonder how much of an effect our architecture had on it. Of course, there are many experiments that can be run in this regard, but oftentimes it\u2019s best to start with the easiest solutions. In this experiment, we added several more dense layers to deep parts of the network. <\/p>\n\n\n\n

In e<\/strong>xperiments\/experiment1\/ex1-2<\/strong> you will find the notebooks for this experiment. Looking at the training and testing notebooks reveals a similar pattern to experiment0 and ex1-1.<\/p>\n\n\n\n

The training accuracy was very high, the testing on the CMU faces testing set was also very high, but the performance on the three amigos was still suboptimal. In fact, the performance on the Ryan dataset was exceptionally bad. Leading us to ask, what does the visualization say?<\/p>\n\n\n

\n
\"\"
Image 1c: Decision layer maps for experiment 1-2. They look very familiar.<\/figcaption><\/figure><\/div>\n\n\n

Right off the bat, looking at the decision layer, dense_6<\/strong> shows that HappyMaps are very similar to what we have already seen. So, this network is not likely to have learned any new tricks \u2014 making it almost the same as the others. The convolution and other dense layers are also about the same as previously seen, even though there are quite a bit more of them. This leads us to conclude that these extra dense layers are not doing us any good. <\/p>\n\n\n\n

Experiment 1-3: Wider Dense Layers<\/strong><\/h5>\n\n\n\n

Still concerned with how architecture affects learnability, the size of each layer is increased. The notebooks can be found in experiments\/experiments1\/ex-1-3<\/strong>. The training performance was a little worse, the results on the three amigos was about the same, and more importantly the visualizations are the same. <\/p>\n\n\n

\n
\"\"
Image 2: Decision layer maps for experiment 1-3. Again, notice the similarities.<\/figcaption><\/figure><\/div>\n\n\n

This network has learned roughly the same thing as all of the other experiments. The visualizations allow us to clearly state that all of our experiments so far have not significantly altered how the networks perform and how they are going about it. <\/p>\n\n\n\n

Experiment 1-4: The Four Class Problem<\/strong><\/h5>\n\n\n\n

This experiment looked to change the problem from just detecting faces left or not, and try to detect if the face is looking left, right, up, or straight. The relevant notebooks are located in experiments\/experiments1\/ex1-4<\/strong>. The training and the performance on the CMU Faces testing set went well. However, the testing on the three amigos was very poor. The performance on Gilbert and Sarah are roughly the same. The performance on Ryan, however, was a disaster. <\/p>\n\n\n

\n
\"\"
Image 3: Decision layer from 1-4. Filter 0 codes for straight ahead, filter 1 is facing camera left, filter 2 is facing right, and facing 3 is facing up.<\/figcaption><\/figure><\/div>\n\n\n

Looking at the visualizations, the decision layer is a mess. Frankly, it\u2019s really hard to see what\u2019s been learned. <\/p>\n\n\n

\n
\"\"
Image 4a: Convolutional layers look similar to others from the past.<\/figcaption><\/figure><\/div>\n\n\n

The interesting thing to note is that the maps from the convolutional layers are roughly the same as all of the other experiments. This certainly lends credibility to the ideas behind transfer learning. Transfer learning involves taking a network that is already trained and using most of that network in a new task. Since the lower level HappyMaps appear to be the same, perhaps retraining these lower level neurons isn\u2019t needed and focus can shift to the deeper levels. <\/p>\n\n\n\n

Experiment 1-5: Remove the Dense Layers<\/strong><\/h5>\n\n\n\n

The purpose of this experience is to see how useful the dense layers are to classification. <\/p>\n\n\n\n

The notebooks for this experiment can be found in experiments\/experiments1\/ex1-5<\/strong>. Technically, there is one dense layer, the final layer, still present in the model. <\/p>\n\n\n\n

Again, the typical pattern is present: good training results, good testing results, poor results on the three amigos. Even our amiga, Sarah, who is usually respectably classified had poor results.<\/p>\n\n\n

\n
\"\"
Image 4b: With just a two element dense layer, the familiar maps appear.<\/figcaption><\/figure><\/div>\n\n\n

Examining the visualizations we see the same pattern. Our decision layer appears to be the same as our other experiments. The convolutional layers are also not terribly different. <\/p>\n\n\n\n

So it would seem that our experiment has confirmed that we just need the one dense layer.  <\/p>\n\n\n\n

Experiment 1-6: Remove Softmax<\/strong><\/h5>\n\n\n\n

In object classification problems, selecting categorical_crossentropy<\/strong> as the loss function and softmax as the activation function in the decision layer is pretty standard. One of the first observations was that the softmax activation function yields HappyMaps that are complements of each other. Would removing that function produce clearer maps? To test this, the decision layer\u2019s activation function was switched to linear<\/strong>. The loss function was also changed because categorical_crossentropy<\/strong> only works with a softmax<\/strong> activation function. This experiment used the old standby of mean_squared_error<\/strong>. The notebooks can be found in experiments\/experiments1\/ex1-6<\/strong>. As one might have expected, the training and testing phases were about the same, even with the changes. What\u2019s more exciting are the visualizations. <\/p>\n\n\n

\n
\"\"
Image 5: Decision layer maps, is this clearer?<\/figcaption><\/figure><\/div>\n\n\n

First of all, when comparing the decision neurons, they are less complementary. No longer will adding the two maps together yield a perfectly activated map. Looking at the neuron that detects camera left faces, it looks like the activation patterns clearly show a face looking at camera left. Examining the \u201cnot left\u201d neuron reveals that it\u2019s more than just \u201cnot left\u201d. It\u2019s possible that the activations represent chin positions for looking straight, right, or up. <\/p>\n\n\n

\n
\"\"
Image 6: Convolutional layer maps. Again, pretty familiar.<\/figcaption><\/figure><\/div>\n\n\n

Looking at the convolutional layers, they appear to be similar to the other experiments. At this point, conventional wisdom about categorial_crossentropy and softmax seems to hold true. Our previous experiences indicate that these two functions tend to train classification networks faster and more accurately. However, this experiment shows it does produce better visualizations. <\/p>\n\n\n\n

Conclusion<\/strong><\/h5>\n\n\n\n

While feature clarity could be better, there\u2019s definitely still value to feature visualization. The first point of appreciation is for the HappyMaps of the output layer. Although the output maps were hard to interpret for the most part, they did help with network optimization efforts. Instead of performing extensive testing to see if network A performed as well as network B, we could simply rely on visualization to see that they had the same HappyMaps. Additionally, the visualizations of the interior layers showed that the layers were larger than necessary. <\/p>\n\n\n\n

Unfortunately, there\u2019s still no insight into how to deal with the biggest problem: the three amigos. Performance on Gilbert, Ryan, and Sarah is still very poor. The visualizations seem to indicate that the facing left neurons are detecting for something that looks like a face looking left. Maybe the network is only looking for that face in a particular location of the image and not generalizing the concept of a left-looking face. Perhaps there\u2019s a tool that can show how the network performs on a given image. Or maybe the key lies in the ability to see what parts of the image the network is attending to, and perhaps this can explain how to fix the network. <\/p>\n\n\n\n

Thanks for keeping up with the Neural Understanding series! We answered a lot of questions along the way, raising a fair number of questions at the same time. In any case, we hope that the insights revealed along the way have provided some food for thought. Who knows, maybe we\u2019ll return in the future to decode more secrets behind neural networks.<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

You made it to the Neural Network series finale! It\u2019s been a fascinating journey, but there\u2019s still work to be done and questions to answer. So saddle up, grab a drink, and let\u2019s jump right in. The Jupyter notebooks mentioned throughout can be found here. Experiment 1-1: Less Filters It\u2019s often the case that clients want … UNDERSTANDING YOUR NEURAL NETWORK \u2013 PART 4: CLOSING THOUGHTS ON MACHINE LEARNING<\/span><\/a><\/p>\n","protected":false},"author":9,"featured_media":1085,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[9,3],"tags":[],"class_list":["post-1084","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ar-news","category-news"],"acf":[],"yoast_head":"\nUnderstanding Your Neural Network - Part 4: Closing Thoughts on ML<\/title>\n<meta name=\"description\" content=\"We built a neural network to classify images using artificial intelligence. The series finale closes out with some tests on filters and dense layers.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Understanding Your Neural Network - Part 4: Closing Thoughts on ML\" \/>\n<meta property=\"og:description\" content=\"We built a neural network to classify images using artificial intelligence. The series finale closes out with some tests on filters and dense layers.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/\" \/>\n<meta property=\"og:site_name\" content=\"Gravity Jack\" \/>\n<meta property=\"article:published_time\" content=\"2021-02-18T08:36:45+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-11-28T13:33:25+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.gravityjack.com\/wp-content\/uploads\/2023\/11\/Neural-networking-pt4-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"344\" \/>\n\t<meta property=\"og:image:height\" content=\"194\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Gravity Admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Gravity Admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/\",\"url\":\"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/\",\"name\":\"Understanding Your Neural Network - Part 4: Closing Thoughts on ML\",\"isPartOf\":{\"@id\":\"https:\/\/www.gravityjack.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.gravityjack.com\/wp-content\/uploads\/2023\/11\/Neural-networking-pt4-1.jpg\",\"datePublished\":\"2021-02-18T08:36:45+00:00\",\"dateModified\":\"2023-11-28T13:33:25+00:00\",\"author\":{\"@id\":\"https:\/\/www.gravityjack.com\/#\/schema\/person\/63c79a7e30c630a69e6e003ef7476fcc\"},\"description\":\"We built a neural network to classify images using artificial intelligence. The series finale closes out with some tests on filters and dense layers.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/#primaryimage\",\"url\":\"https:\/\/www.gravityjack.com\/wp-content\/uploads\/2023\/11\/Neural-networking-pt4-1.jpg\",\"contentUrl\":\"https:\/\/www.gravityjack.com\/wp-content\/uploads\/2023\/11\/Neural-networking-pt4-1.jpg\",\"width\":344,\"height\":194},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.gravityjack.com\/#website\",\"url\":\"https:\/\/www.gravityjack.com\/\",\"name\":\"Gravity Jack\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.gravityjack.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.gravityjack.com\/#\/schema\/person\/63c79a7e30c630a69e6e003ef7476fcc\",\"name\":\"Gravity Admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.gravityjack.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/aa3d42e00068a5cdefa39bab3155196c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/aa3d42e00068a5cdefa39bab3155196c?s=96&d=mm&r=g\",\"caption\":\"Gravity Admin\"},\"url\":\"https:\/\/www.gravityjack.com\/author\/gjmagravityjack-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Understanding Your Neural Network - Part 4: Closing Thoughts on ML","description":"We built a neural network to classify images using artificial intelligence. The series finale closes out with some tests on filters and dense layers.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/","og_locale":"en_US","og_type":"article","og_title":"Understanding Your Neural Network - Part 4: Closing Thoughts on ML","og_description":"We built a neural network to classify images using artificial intelligence. The series finale closes out with some tests on filters and dense layers.","og_url":"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/","og_site_name":"Gravity Jack","article_published_time":"2021-02-18T08:36:45+00:00","article_modified_time":"2023-11-28T13:33:25+00:00","og_image":[{"width":344,"height":194,"url":"https:\/\/www.gravityjack.com\/wp-content\/uploads\/2023\/11\/Neural-networking-pt4-1.jpg","type":"image\/jpeg"}],"author":"Gravity Admin","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Gravity Admin","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/","url":"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/","name":"Understanding Your Neural Network - Part 4: Closing Thoughts on ML","isPartOf":{"@id":"https:\/\/www.gravityjack.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/#primaryimage"},"image":{"@id":"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/#primaryimage"},"thumbnailUrl":"https:\/\/www.gravityjack.com\/wp-content\/uploads\/2023\/11\/Neural-networking-pt4-1.jpg","datePublished":"2021-02-18T08:36:45+00:00","dateModified":"2023-11-28T13:33:25+00:00","author":{"@id":"https:\/\/www.gravityjack.com\/#\/schema\/person\/63c79a7e30c630a69e6e003ef7476fcc"},"description":"We built a neural network to classify images using artificial intelligence. The series finale closes out with some tests on filters and dense layers.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.gravityjack.com\/news\/understanding-your-neural-network-part-4-closing-thoughts-on-machine-learning\/#primaryimage","url":"https:\/\/www.gravityjack.com\/wp-content\/uploads\/2023\/11\/Neural-networking-pt4-1.jpg","contentUrl":"https:\/\/www.gravityjack.com\/wp-content\/uploads\/2023\/11\/Neural-networking-pt4-1.jpg","width":344,"height":194},{"@type":"WebSite","@id":"https:\/\/www.gravityjack.com\/#website","url":"https:\/\/www.gravityjack.com\/","name":"Gravity Jack","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.gravityjack.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.gravityjack.com\/#\/schema\/person\/63c79a7e30c630a69e6e003ef7476fcc","name":"Gravity Admin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.gravityjack.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/aa3d42e00068a5cdefa39bab3155196c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/aa3d42e00068a5cdefa39bab3155196c?s=96&d=mm&r=g","caption":"Gravity Admin"},"url":"https:\/\/www.gravityjack.com\/author\/gjmagravityjack-com\/"}]}},"_links":{"self":[{"href":"https:\/\/www.gravityjack.com\/wp-json\/wp\/v2\/posts\/1084","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.gravityjack.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gravityjack.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gravityjack.com\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gravityjack.com\/wp-json\/wp\/v2\/comments?post=1084"}],"version-history":[{"count":0,"href":"https:\/\/www.gravityjack.com\/wp-json\/wp\/v2\/posts\/1084\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.gravityjack.com\/wp-json\/wp\/v2\/media\/1085"}],"wp:attachment":[{"href":"https:\/\/www.gravityjack.com\/wp-json\/wp\/v2\/media?parent=1084"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gravityjack.com\/wp-json\/wp\/v2\/categories?post=1084"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gravityjack.com\/wp-json\/wp\/v2\/tags?post=1084"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}