<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dipanjan (DJ) Sarkar on Medium]]></title>
        <description><![CDATA[Stories by Dipanjan (DJ) Sarkar on Medium]]></description>
        <link>https://medium.com/@djsarkar?source=rss-6278d12b0682------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Wy0cxXZpX-FrMWeXsqOOpg.png</url>
            <title>Stories by Dipanjan (DJ) Sarkar on Medium</title>
            <link>https://medium.com/@djsarkar?source=rss-6278d12b0682------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:44:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@djsarkar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[How Good is Google Gemini 1.5 With a Massive 1 Million Context Window?]]></title>
            <link>https://pub.towardsai.net/how-good-is-google-gemini-1-5-with-a-massive-1-million-context-window-b386d285845d?source=rss-6278d12b0682------2</link>
            <guid isPermaLink="false">https://medium.com/p/b386d285845d</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Dipanjan (DJ) Sarkar]]></dc:creator>
            <pubDate>Tue, 20 Feb 2024 14:01:54 GMT</pubDate>
            <atom:updated>2024-02-20T14:01:54.985Z</atom:updated>
            <content:encoded><![CDATA[<h4>Taking Google’s new Large Language Model on a test drive!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9HO5vb5hiZCWc5jtmQ0-Lg.jpeg" /><figcaption>Created by Author with DALL-E</figcaption></figure><h3>Introduction</h3><p>Artificial Intelligence, and particularly Generative AI, is evolving at a rapid pace; every other week, we hear a new Large Language Model (LLM) being released! Google has once again pushed the boundaries with the introduction of Gemini 1.5, its next-generation large language model. Google claims this new model enhances efficiency and quality through a Mixture-of-Experts approach, where <a href="https://developers.googleblog.com/2024/02/gemini-15-available-for-private-preview-in-google-ai-studio.html">it routes your request to a group of smaller “expert” neural networks so responses are faster and higher quality.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FktkKwkdOz4rxVij87jhsQ.png" /><figcaption>Gemini 1.5 has a MASSIVE token context window; Source: <a href="https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024">Google Tech Blog</a></figcaption></figure><p>This model promises to revolutionize how developers interact with AI, given its multimodal capabilities to work with text, images, and even video. The eye-catching part, Gemini 1.5, promises a MASSIVE 1 Million token context window!<strong> </strong>Yes, you heard it right, 1 Million tokens. Currently, Open AI’s most powerful GPT-4 model has a context window of 128K tokens.</p><p>But just how transformative is Gemini 1.5? Luckily as a<a href="https://cloud.google.com/innovators/innovator?profileId=101168741815690081396"> Google Cloud Champion Innovator</a>, I got early access to the new Gemini 1.5 model. Let’s dive into its key capabilities before taking it for a test drive in Google AI Studio!</p><h3>Key Capabilities</h3><p>Gemini 1.5 stands out with its groundbreaking 1M (million) token context window, a significant leap from the Pro version of Gemini 1.0, which gives a 32K context window. Do note the default version of Gemini 1.5 Pro will come with a 128K token context window but the 1M token context window is right now in private preview, accessible to a select userbase and beta testers. But expect it to be in general availability very soon.</p><p>This is a multimodal LLM, and this new super lengthy 1M context window opens up new possibilities for developers, enabling the processing of extensive documents, code repositories, or even lengthy videos in a single go. Gemini 1.5 Pro reasons across these multiple modalities of data and output text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*litPZ9n1PvTyeNEITNUXlA.gif" /><figcaption>Gemini 1.5 can understand and extract useful information from large documents; Source: <a href="https://developers.googleblog.com/2024/02/gemini-15-available-for-private-preview-in-google-ai-studio.html">Google Dev Blog</a></figcaption></figure><ul><li><strong>Unprecedented Context Window:</strong> With a default 128K token window expandable to 1M tokens, Gemini 1.5 can handle over 700K words of text, allowing for the analysis of entire codebases and large PDFs directly.</li><li><strong>Multimodal Understanding:</strong> Whether it’s text, code, image or video, Gemini 1.5 can reason across different formats, providing insights and solutions by integrating various types of data. Unfortunately, audio support is not there yet for videos.</li><li><strong>Enhanced Developer Tools:</strong> Google AI Studio simplifies integrating the Gemini API into applications, supported by features like easy tuning for specific needs and lower pricing for the Gemini 1.0 Pro model, ensuring a good balance of cost and performance.</li></ul><h3>Test Driving Google Gemini 1.5 in Google AI Studio</h3><p>Here I will showcase two real-world examples where I tried out Google Gemini 1.5 with its 1M context window. The first example will be based on analyzing a large PDF document and the second example will involve analyzing multiple large videos. Before that let’s look at the platform where I will access these LLMs.</p><h4>Getting Started with Google AI Studio</h4><p>You can access the current Google Gemini 1.0 and very soon the new Gemini 1.5 LLMs in <a href="https://aistudio.google.com/">Google AI Studio</a>. Simply go to the link here and sign in with your Google email account, and you should be good to go. You can then click on Create New on the left side and select Chat Prompt, as depicted in the following screenshot. Take a closer look at the right side; that is where I can select the Gemini 1.5 Pro model, and as soon as I do that, you can see the Preview box at the bottom, which shows how much of the 1M context window you are using at the moment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YGevWm6aU-vZ2MWCYT7xyA.jpeg" /><figcaption>Google AI Studio prompt creation page</figcaption></figure><h4>Analyzing a Large Investment Scheme Document</h4><p>You can upload really large documents and make Google Gemini Pro 1.5 understand, reason, and give you useful insights pertaining to these documents. Personally, I invest quite a bit internationally as well as in my home country’s investment schemes. I have been recently looking at a new ETF offering that is launching, which is backed by Gold. You can take a look at <a href="https://assets.zerodhafundhouse.com/offer-documents/sid/Zerodha%20Gold%20ETF%20-%20SID.pdf">this document here</a> if you are interested.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*5DJN_XhxSB_znfefpMV3GQ.png" /><figcaption>Our input will be a huge 96 page investment document</figcaption></figure><p>This is an investment scheme document for an ETF which will closely track the price of gold. Now, if you check the PDF, it is a massive 96-page document! Now, who has the time to read it, considering everyone’s busy schedules? So I uploaded this document into Google AI Studio and asked Gemini 1.5 to summarize and show me the key pros and cons. You can see the output from Gemini 1.5 below, which is pretty comprehensive and showcases the key highlights, being it&#39;s an ETF, easy to transact, cost-effective, and also some risks, including tracking errors, market, and liquidity risks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MEiTdT2jZKFJSYk_BDxqLw.png" /><figcaption>Summarizing the large document with Gemini 1.5</figcaption></figure><p>On asking about how the NAV is computed, it gives a pretty good summary and you can see it has generated this from parts of the document after page 85.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2cbTMdtVbRuH-j7pVaEOWA.png" /><figcaption>Querying specific aspects of the large document using Gemini 1.5</figcaption></figure><p>I also tried to make it act as an advisor for various age groups and to give us a recommendation of if someone who is in their 30s should invest in this scheme vs someone who is in their 60s. It does a pretty good job where you can see it focus on long-term growth for people who are in their 30s because they do have time on their side and it focuses on aspects like capital preservation and volatility as a risk because people above 60 would not want to invest in schemes which are subject to extreme fluctuation in prices in the short-term. Invest allocation remains the same for both age groups, though I would say this is something that the model probably said, considering how much you should put in gold as an asset vs other schemes involving equities, bonds, etc. It is also not surprising that the LLM plays it safe with recommendations, given probably the heavy guardrails and moderation put on it internally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rsM55FoZboO4Fvz19A_1cg.png" /><figcaption>Comparison of investment recommendations from Gemini 1.5 based on different age groups</figcaption></figure><p>Overall, Gemini 1.5 does pretty well on large documents but it is still not perfect but given the rapid advancements taking place, it is quite an achievement in a short time since Gemini 1.0 launched.</p><h4>Extracting Video Game Strategies from YouTube</h4><p>Now Gemini 1.5 is a multi-modal model that can extract insights from videos, too. However, it breaks it down into thousands of frames (without audio) and then analyzes these frames and can perform complex reasoning, summarization, and question-answering. I believe that once the audio dimension is added to this model it will probably get even better as I do notice some shortcomings, which will be shown in my analysis below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h38te9gb9omS19Yv" /><figcaption>Honkai Star Rail; Source: <a href="https://www.eurogamer.net/honkai-star-rail-is-a-goofy-sci-fi-genshin-impact-that-actually-respects-your-time">EuroGamer</a></figcaption></figure><p>The scenario here is that we have a very popular online RPG called Honkai Star Rail which is something I play in my spare time. You usually play with a team of four characters and a large part of this game involves turn-based combat and optimizing character builds with equipment, skills and more! Some folks create really detailed guides for each game character on YouTube and I wanted to see if there was a way to get key insights from these videos about specific character builds using Google Gemini 1.5</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zA__Jt5aqEahRQJw" /><figcaption>Turn-based 4 character team gameplay; Source: <a href="https://www.eurogamer.net/honkai-star-rail-review-more-than-genshin-impact-in-space">EuroGamer</a></figcaption></figure><p>The input videos for this use-case comes from a popular YouTuber <a href="https://www.youtube.com/@Braxophone">Braxophone</a> who makes interesting guides on how to build each game character as they are released. I took three of his videos for building three popular characters from Honkai Star Rail and started prompting Gemini 1.5 as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*21RRl8EO4rZkgbWruqhefg.png" /><figcaption>Each video is processed frame by frame in Gemini 1.5</figcaption></figure><p>It is clear that each video has multiple frames which are then processed by the model and we can see the token count quickly rise up to almost 800K tokens! I started off with a simple question as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qCLMhWjpAY54u0MJ4Gu-QQ.png" /><figcaption>Querying videos for key entities using Gemini 1.5</figcaption></figure><p>Now so far, it does quite well and is able to recognize the characters from each of the three videos. I make the next question a bit more complex asking it to give me a full character build for Blade.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QwwuY1alGB2kf5XMxbQ3mw.png" /><figcaption>Getting a specific character build from a video using Gemini 1.5</figcaption></figure><p>Looks pretty comprehensive and covers all the key components needed for a build actually which include equipment (relics and light cones), stats (partly correct) and possible teams. However it does a fundamental mistake here by suggesting the 4-piece Thief set for relics when that is actually the best set for another character in the 2nd video I put (Ruan Mei). So I tried another prompt now being a bit more specific.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PUXWn9LlD_tOD9nCxiFTow.png" /><figcaption>Specific prompting to make Gemini give more accurate results</figcaption></figure><p>It does get it correct this time and probably gets the information from the <a href="https://youtu.be/-g0c_ZcgjgU?si=SDQFnJ-EqqFZHWia&amp;t=325">relevant video at the 5:25 mark</a>! Of course it’s still not a 100% correct because a relic set comes with 4 pieces like Longevous Disciple and you can stack an additional 2 piece set like Rutilant Arena which is called as planar ornaments. So overall each character can wear a total of 6 pieces to keep it simple. However the LLM probably may not have this context from a short video so that is understandable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fMcqhUDOoDW36IOO7wsGEg.png" /><figcaption>Correct recommendations for Blade’s equipment from the source video; Source: <a href="https://youtu.be/-g0c_ZcgjgU?si=SDQFnJ-EqqFZHWia&amp;t=325">Braxophone</a></figcaption></figure><p>The interesting part however is the above video on Blade was released almost 6 months back if you check it on YouTube. But if you look at the team compositions in the LLM response above, you will see one of the team compositions for Blade being — Blade along with Bronya and Ruan Mei. However, the character Ruan Mei was not even released then!</p><p>The LLM is smart enough to understand Blade being mentioned in the character video of Ruan Mei which was created just a month back on its release <a href="https://www.youtube.com/watch?v=gWUe1mYrZUw&amp;ab_channel=Braxophone">on YouTube here</a> (that was one of the other two videos I had given to the Gemini 1.5 model). This is actually mentioned by Braxophone and you can see it in the following screenshot taken from the <a href="https://youtu.be/gWUe1mYrZUw?si=SDUlcv5enCiQL8Z4&amp;t=833">video at the 13:53 mark</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rG55Cwqm0vUbpZp28fLbsg.png" /><figcaption>Blade’s team composition suggested in a different video but captured correctly by Gemini 1.5; Source: <a href="https://youtu.be/gWUe1mYrZUw?si=SDUlcv5enCiQL8Z4&amp;t=833">Braxophone</a></figcaption></figure><p>This does reinforce the aspect that the LLM is quickly able to understand and relate to entities occuring across different videos even if it was never trained on this data before! It’s not perfect and does mix up aspects amongst different characters but overall performs quite well given its one of Google’s initial multimodal models.</p><h3>Conclusion</h3><p>With its massive 1M token context window and multimodal capabilities, Gemini 1.5 has demonstrated its prowess in handling complex, diverse data types ranging from large documents to video analytics. Through practical use-cases, such as analyzing an elaborate investment scheme document and extracting nuanced strategies from video game guides, Gemini 1.5 definitely shows promise in being able to understand large and diverse modalities of data.</p><p>However, it does come with its own set of challenges. Larger contexts lead to the model sometimes mixing things up. Also, there are limitations in the LLM not being able to leverage the audio from video data which hopefully gets fixed in a future release. Another serious limitation is the compute time. Each of these queries took me between 30–60 seconds which proves that the larger the model and context window is, the longer it takes to process prompts. Hopefully with various optimizations, we see an improvement to this aspect in the future.</p><p>In conclusion, Google Gemini 1.5, despite its nascent state in the realm of multimodal large language models, marks a significant stride towards the progress of Generative AI and I hope you got something useful out of this read!</p><p><em>Reach out to me at </em><a href="https://www.linkedin.com/in/dipanjans/"><em>my LinkedIn</em></a><em> or </em><a href="https://djsarkar.com/"><em>my website</em></a><em> if you want to connect. I do quite a bit of AI consulting, trainings, and projects.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b386d285845d" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/how-good-is-google-gemini-1-5-with-a-massive-1-million-context-window-b386d285845d">How Good is Google Gemini 1.5 With a Massive 1 Million Context Window?</a> was originally published in <a href="https://pub.towardsai.net">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get Started With Google Gemini Pro Using Python in 5 Minutes]]></title>
            <link>https://pub.towardsai.net/get-started-with-google-gemini-pro-using-python-in-5-minutes-00700244f58a?source=rss-6278d12b0682------2</link>
            <guid isPermaLink="false">https://medium.com/p/00700244f58a</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Dipanjan (DJ) Sarkar]]></dc:creator>
            <pubDate>Thu, 15 Feb 2024 16:02:40 GMT</pubDate>
            <atom:updated>2024-02-15T16:02:40.318Z</atom:updated>
            <content:encoded><![CDATA[<h4>Access powerful Large Language Model Capabilities for Free!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QMr0ird6s2QwtwxN_YAARQ.png" /><figcaption>Google Gemini — Source: <a href="https://blog.google/products/gemini/bard-gemini-advanced-app/">Bard becomes Gemini</a></figcaption></figure><h3>Introduction</h3><p>Google Gemini Pro is part of Google’s latest AI model, Gemini, which was announced as their most capable and general AI model to date. This represents a significant step forward in Google’s AI development, designed to handle a wide range of tasks with state-of-the-art performance across many leading benchmarks. Gemini Pro, along with Gemini Ultra and Gemini Nano, was introduced to mark the beginning of what Google DeepMind calls the Gemini era, aiming to unlock new opportunities for people everywhere by leveraging AI’s capabilities​.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*sk8aIMgAbr4_sUWRf7SjCg.png" /><figcaption>Source: <a href="https://blog.google/products/gemini/bard-gemini-advanced-app/">Google Bard is now Gemini</a></figcaption></figure><p>Gemini Pro was globally launched in January 2024, following a collaboration with Samsung to integrate Gemini Nano and Gemini Pro into the Galaxy S24 smartphone lineup. In fact even their ChatGPT competitor assistant app B<a href="https://blog.google/products/gemini/bard-gemini-advanced-app/">ard has been now renamed to Gemini</a> just last week as of writing this article (Feb 8, 2024). We also saw the introduction of “Gemini Advanced with Ultra 1.0” through the AI Premium tier of the Google One subscription service​​.</p><p>One of the key features of Gemini Pro is its API, which is designed to allow developers to develop and integrate AI-powered functionalities into their applications quickly. The API supports a variety of programming languages, including Python, which is what we will use here to show you how to get started with using the Gemini Pro Large Language Model for free (as of Feb 2024)!</p><h3>Gemini Essentials</h3><p>Google’s Gemini is a suite of AI models designed to handle a wide array of tasks, including content generation and problem-solving with both text and image inputs. Here’s a brief overview of the different Gemini models you can access easily via APIs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*EftLH5u5kZFl5mYGsSgteA.png" /><figcaption>Key Gemini Models available as APIs</figcaption></figure><h3>Gemini API Pricing</h3><p>At this moment of writing the article which is Feb 13, 2024, the Gemini Pro API is free to use, however my gut tells me they will soon introduce a token-based pricing as you can see in the following screenshot taken from <a href="https://ai.google.dev/pricing">their website</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*wepYPKrzogMBDWdPb2fw9A.png" /><figcaption>Gemini API pricing</figcaption></figure><h3>Getting Started with Gemini Pro and Python</h3><p>Let’s get started now with building basic LLM functionalities using Gemini Pro API and Python. We will show you how to get an API key and then use the relevant Gemini LLMs in Python.</p><h4>Getting Your API Key from Google AI Studio</h4><p>Google AI Studio is a free, web-based tool that allows you to quickly develop prompts and obtain an API key for app development. You can sign into Google AI Studio with your Google account and get your API key from <a href="https://makersuite.google.com/app/apikey"><strong>here</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qugDAChAoZv51h8m9bltoA.png" /><figcaption>Create an API key in Google AI Studie</figcaption></figure><p>Remember to save the key somewhere safe and do NOT expose it in a public platform like GitHub.</p><blockquote>Google Gemini Pro is still not accessible in all countries but expect it to be available soon in case you are not able to access it yet, or you could use a VPN. Check available regions <a href="https://ai.google.dev/available_regions&#39;">here</a></blockquote><h4>Using Gemini Pro API with Python for Text Inputs</h4><p>To start using the Gemini Pro API, we need to install the <a href="https://github.com/google/generative-ai-python"><strong>google-generativeai</strong></a> package from PyPI or GitHub</p><pre>pip install -q -U google-generativeai</pre><p>Now I have saved my API key in a YAML file so I can load it and I do not need to expose the key in my code publicly anywhere. I load up this file and load my API key into a variable as follows.</p><pre>import yaml<br><br>with open(&#39;gemini_key.yml&#39;, &#39;r&#39;) as file:<br>    api_creds = yaml.safe_load(file)<br><br>GOOGLE_API_KEY = api_creds[&#39;gemini_key&#39;]</pre><p>The next step is to create a connection to the Gemini Pro model via the API as follows where you first need to use your API to set a config and then load the model (or rather create a connection to the model on Google’s servers).</p><pre>import google.generativeai as genai<br><br>genai.configure(api_key=GOOGLE_API_KEY)<br>model = genai.GenerativeModel(&#39;gemini-pro&#39;)</pre><p>We are now ready to start using Gemini Pro! Let’s do a basic task of getting some information.</p><pre>response = model.generate_content(&quot;Explain Generative AI with 3 bullet points&quot;)<br>to_markdown(response.text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9oGRczZso_Azou-L7nmcOw.png" /></figure><p>The <strong><em>to_markdown(…)</em></strong> function makes the text output look prettier and you can get the function from the <a href="https://ai.google.dev/tutorials/python_quickstart">official docs</a> or use my <a href="https://colab.research.google.com/drive/1OLCMhdlcjXpXk6AYQp4Li4hv6YrtQsTN?usp=sharing">Colab notebook</a>.</p><p>Let’s try a more practical example now, imagine you are automating IT support across multiple regions with different languages. We will make the LLM try to detect the source language of the customer issue, translate it to english, reply back in the original language of the customer.</p><pre>it_support_queue = [<br>    &quot;I can&#39;t access my email. It keeps showing an error message. Please help.&quot;,<br>    &quot;Tengo problemas con la VPN. No puedo conectarme a la red de la empresa. ¿Pueden ayudarme, por favor?&quot;,<br>    &quot;Mon imprimante ne répond pas et n&#39;imprime plus. J&#39;ai besoin d&#39;aide pour la réparer.&quot;,<br>    &quot;Eine wichtige Software stürzt ständig ab und beeinträchtigt meine Arbeit. Können Sie das Problem beheben?&quot;,<br>    &quot;我无法访问公司的网站。每次都显示错误信息。请帮忙解决。&quot;<br>]<br><br>it_support_queue_msgs = f&quot;&quot;&quot;<br>&quot;&quot;&quot;<br>for i, msg in enumerate(it_support_queue):<br>  it_support_queue_msgs += &quot;\nMessage &quot; + str(i+1) + &quot;: &quot; + msg<br><br>prompt = f&quot;&quot;&quot;<br>Act as a customer support agent. Remember to ask for relevant information based on the customer issue to solve the problem.<br>Don&#39;t deny them help without asking for relevant information. For each support message mentioned below<br>in triple backticks, create a response as a table with the following columns:<br><br><br>  orig_msg: The original customer message<br>  orig_lang: Detected language of the customer message e.g. Spanish<br>  trans_msg: Translated customer message in English<br>  response: Response to the customer in orig_lang<br>  trans_response: Response to the customer in English<br><br><br>Messages:<br>&#39;&#39;&#39;{it_support_queue_msgs}&#39;&#39;&#39;<br>&quot;&quot;&quot;</pre><p>Now that we have a prompt ready to go into the LLM let’s execute it!</p><pre>response = model.generate_content(prompt)<br>to_markdown(response.text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_kadg9Lnx8KRYoqcciC5LQ.png" /><figcaption>Response to our prompt from Gemini Pro LLM</figcaption></figure><p>Pretty neat! I am sure with more detailed information or a RAG system, the responses can be even more relevant and useful.</p><h4>Using Gemini Pro Vision API with Python for Text and Image Inputs</h4><p>Google has released a Gemini Pro Vision multimodal LLM which can take both text and images as input and return back text as output. Remember, this is still an LLM that outputs text only. Let’s use it with a simple use-case of understanding a picture and creating a short story from it!</p><p>We start by loading the image first.</p><pre>import PIL.Image<br><br>img = PIL.Image.open(&#39;cat_pc.jpg&#39;)<br>img</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*Kx78V1ONyBBW9_p4NjeeKA.jpeg" /></figure><p>After this we load the Gemini Pro Vision model and send it the following prompt to get a response.</p><pre>model = genai.GenerativeModel(&#39;gemini-pro-vision&#39;)<br>prompt = &quot;&quot;&quot;<br>Describe the given picture first based on what you see.<br>Then create a short story based on your understanding of the picture.<br><br>Output should have both the description and the short story as two separate items <br>with relevant headings<br>&quot;&quot;&quot;<br>response = model.generate_content(contents=[prompt, img])<br>to_markdown(response.text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muYGTHhWZFedip57PoOIxA.png" /><figcaption>Response to our prompt from Gemini Pro Vision LLM</figcaption></figure><p>Overall not too bad at all! Although I have probably seen that GPT-4 with DALL-E can recognize the game as Animal Crossing, which is even more accurate. But pretty good, I would say.</p><p>You can also use Gemini Pro to build interactive chat experiences. This involves sending messages to the API and receiving responses, supporting multi-turn conversations. Feel free to check out the <a href="https://ai.google.dev/tutorials/python_quickstart">detailed API documentation</a> for some examples!</p><h3>Conclusion</h3><p>In conclusion, whether you’re a seasoned AI developer or just starting out, Google’s Gemini Pro and Python provide a pretty straightforward and powerful way to incorporate cutting-edge AI into your applications and projects. Moreover, the current availability of the Gemini Pro API for free is an invitation to explore the capabilities of AI LLMs without initial investment. While future pricing changes are anticipated, the opportunity to start building with such a powerful tool at no cost is quite a steal!</p><p>Hopefully, you got an idea of how to obtain your API key via Google AI Studio to execute your first Python script with the Gemini Pro API with a really short time to get started. Now go ahead and try leveraging it in your own problems and projects!</p><p><em>Reach out to me at </em><a href="https://www.linkedin.com/in/dipanjans/"><em>my LinkedIn</em></a><em> or </em><a href="https://djsarkar.com/"><em>my website</em></a><em> if you want to connect. I do quite a bit of AI consulting, trainings, and projects.</em></p><p>Get the complete code in a <a href="https://colab.research.google.com/drive/1OLCMhdlcjXpXk6AYQp4Li4hv6YrtQsTN?usp=sharing"><em>Google Colab notebook here</em></a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=00700244f58a" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/get-started-with-google-gemini-pro-using-python-in-5-minutes-00700244f58a">Get Started With Google Gemini Pro Using Python in 5 Minutes</a> was originally published in <a href="https://pub.towardsai.net">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three Popular ChatGPT Prompt Engineering Patterns for Life and Business Productivity]]></title>
            <link>https://pub.towardsai.net/three-popular-chatgpt-prompt-engineering-patterns-for-life-and-business-productivity-0f2e40d819b1?source=rss-6278d12b0682------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f2e40d819b1</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dipanjan (DJ) Sarkar]]></dc:creator>
            <pubDate>Tue, 13 Feb 2024 13:29:34 GMT</pubDate>
            <atom:updated>2024-02-13T13:29:34.995Z</atom:updated>
            <content:encoded><![CDATA[<h4>Generative AI and ChatGPT: Revolutionizing digital interactions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/1*vocZN1o94_jIToBkavPDPg.png" /><figcaption>Created with DALL-E 3</figcaption></figure><h3>Introduction</h3><p>In recent years, the landscape of artificial intelligence has undergone a significant transformation with the emergence of Generative AI technologies. Leading this revolution is ChatGPT, a state-of-the-art large language model (LLM) developed by OpenAI. As a large language model, ChatGPT is built on a vast dataset of language examples, enabling it to understand and generate human-like text with remarkable accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/0*JFYgNH2RbvHgX6pA" /><figcaption><strong><em>Figure 1: Generative Large Language Models in a nutshell, Source: Google AI</em></strong></figcaption></figure><p>This capability places ChatGPT at the cutting edge of natural language processing, significantly enhancing how we interact with digital systems. The benefits of such a model are manifold, including its ability to provide detailed, context-aware responses, making it a versatile tool for a wide range of applications. ChatGPT’s advanced language understanding, and generation capacities have not only increased user engagement but also opened new avenues for increased productivity and automation in personal life as well as business problems.</p><h3>Understanding Prompt Engineering</h3><p>At the heart of effectively leveraging ChatGPT lies ‘prompt engineering’ — a crucial skill that involves crafting specific inputs or prompts to guide the AI in producing the desired outputs. This technique is key in fully utilizing ChatGPT’s capabilities, enabling users to achieve more precise and relevant responses. For example, instead of asking “<em>What’s the weather like?</em>” a more engineered prompt would be “<em>Provide a current weather report for Zurich, including temperature and chance of rain.</em>” This specific prompt helps in generating a more focused and useful response.<br>Prompts can be as simple as asking a question or as complex as planning meetings based on schedules as shown in the example in Figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/0*J61zqwFa4_SMcScn" /><figcaption><strong><em>Figure 2: Planning a meeting with ChatGPT prompting</em></strong></figcaption></figure><p>With advancements in LLM capabilities, ChatGPT can now handle both text and images in its prompts as shown in the following example in Figure 3, where it can accurately detect entities as well as context in an image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*yR46khAyj_NxoJV8" /><figcaption><strong><em>Figure 3: ChatGPT prompting with images and text</em></strong></figcaption></figure><h4>The essence of prompt engineering patterns</h4><p>Prompt engineering goes beyond mere question-asking; it’s both an art and a science. It requires an understanding of the subtleties of language and the AI’s processing abilities. Prompt engineering patterns are structured, reusable approaches or templates that assist in eliciting particular types of responses or actions from ChatGPT. These patterns are crucial for enhancing both personal life and business productivity, as they streamline interactions with AI for greater efficiency and effectiveness. For instance, a pattern for gathering information might start with “List the steps involved in…” or “Summarize the key points of…”, guiding ChatGPT to structure its response in a clear, organized manner.</p><h3>Popular prompt engineering patterns</h3><p>In the following sections, we will unveil a selection of popular prompt engineering patterns that have proven to be exceptionally effective in boosting productivity in both personal and professional contexts. These patterns are not just theoretical concepts; they are practical tools that have been honed through widespread use and refinement. By exploring these patterns, we aim to unlock the true potential of ChatGPT in addressing a variety of life and business challenges.</p><p>Each pattern is designed to serve a specific purpose, whether it’s streamlining communication, enhancing information retrieval, or aiding in complex problem-solving. We will provide examples and insights into how these patterns can be applied in real-world scenarios, demonstrating their versatility and impact. From simple queries to more sophisticated interactions, these prompt engineering patterns will empower you to navigate and optimize your engagements with ChatGPT, transforming it into a valuable ally in your daily life and business operations.</p><h3>The Persona Pattern</h3><p>When it comes to enhancing interactions with AI, the Persona Pattern is an ingenious prompt engineering technique that involves adopting a specific persona to deliver tailored information or services. This pattern is particularly useful when the response needs to be customized for an audience with a specific background or set of circumstances and rules.</p><h4>Format for the Persona Pattern:</h4><ul><li>Act as Persona P: Here, you assign the AI a specific role or identity. The persona could be any professional or character like an ‘astronomer,’ ‘yoga instructor,’ or ‘financial advisor.’</li><li>Perform Task T: You define a particular task that the assigned persona should perform, ensuring that the task is relevant to the persona’s expertise or role.</li><li>Assume that I am Persona U (optional): This is an optional step where you can provide the AI with a target audience persona, which could be yourself or someone else, such as a ‘curious 6-year-old’ or a ‘stressed-out college student.’</li></ul><h4>Examples:</h4><ul><li>“Explain the solar system. Assume that I am a curious 6-year-old.”</li><li>“Act as a yoga instructor and guide me through meditation. Assume that I am a stressed-out college student.”</li><li>“Act as a computer engineer and describe how computers work. Assume that I am a senior citizen unfamiliar with modern technology.”</li></ul><h4>Real-world scenario:</h4><p>In the scenario provided in Figure 4, the task is to act as a financial advisor for someone looking to invest 10,000 CHF. The persona of the user is a novice investor, aged 32, with no prior investment expertise. The AI, adopting the persona of a financial advisor, provides a simplified yet comprehensive investment strategy that is tailored to the user’s novice status.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*xO5KNdrP9WQr_gFD" /><figcaption><em>Figure 4: persona pattern scenario: investment advice from ChatGPT as a financial advisor</em></figcaption></figure><p>Check out the full prompt pattern and the conversation with ChatGPT by <a href="https://chat.openai.com/share/9cdb6ee2-2de7-4c91-9f20-e61eb32dba87">clicking here</a>.</p><h3>The Cognitive Verifier Pattern</h3><p>The Cognitive Verifier Pattern is a sophisticated prompt engineering strategy designed to enhance the accuracy and relevance of the responses generated by AI. This pattern is especially useful in situations where the initial query may require additional context or clarification to provide a precise answer.</p><h3>Format for the Cognitive Verifier Pattern:</h3><p>To use the Cognitive Verifier Pattern, your prompt should incorporate these fundamental contextual cues:</p><ul><li>From now on, whenever a question is posed to you, follow these rules</li><li>Generate a number of additional questions that would help more accurately answer the question</li><li>Ask these questions one by one</li><li>Combine the answers to the individual questions to produce the final answer to the overall question</li><li>Let’s start with the first question</li></ul><h4>Pattern:</h4><p>When you are asked a question, follow these rules. Generate a number of additional questions that would help you more accurately answer the question. Ask these questions one by one. Combine the answers to the individual questions to produce the final answer to the overall question. Let’s start with the first question.</p><h4>Examples:</h4><p>“If tasked with suggesting a workout routine, follow these rules. Pose additional questions about any existing health conditions, my fitness level, and the type of equipment available to me. Ask these questions one by one. Integrate this information to offer a customized workout plan suitable for my conditions. Let’s start with the first question”</p><h4>Real-world scenario:</h4><p>In the scenario provided in Figure 5,the AI adopts the Cognitive Verifier Pattern to recommend a book based on the user’s preferences in genres, recent reads, and current mood. The AI asks targeted questions to discern the user’s preferences, delving deeper into the types of crime, mystery, and thriller books the user enjoys, as well as the mood they are in for reading. By sequentially clarifying these preferences, the AI combines the information to suggest “Big Little Lies” by Liane Moriarty, which aligns with the user’s taste for a tranquil yet suspenseful narrative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*sVEPj82MJajxhpbh" /><figcaption><strong><em>Figure 5: Cognitive Verifier Pattern Scenario: Book Recommender using ChatGPT</em></strong></figcaption></figure><p>Check out the full prompt pattern and the conversation with ChatGPT by <a href="https://chat.openai.com/share/9cdb6ee2-2de7-4c91-9f20-e61eb32dba87">clicking here</a>.</p><h3>The Recipe Pattern</h3><p>The Recipe Pattern is a prompt engineering strategy designed to create a clear, step-by-step guide to accomplish a specific task. This method is incredibly beneficial when you need to outline a process or sequence of actions in a logical and comprehensive manner.</p><h4>Format for the Recipe Pattern:</h4><ul><li>To effectively apply the Recipe Pattern in your prompts, you should include the following elements:</li><li>I would like to achieve T: Define the overall task or goal that you want to accomplish.</li><li>I know that I need to perform steps A, B, C: List some of the key steps that are already known, which contribute to the completion of the task.</li><li>Provide a complete sequence of steps for me: Request a detailed and ordered list of actions to reach the goal.</li><li>(Optional) Fill in any missing steps: Ask to identify and include any steps that may have been overlooked.</li><li>(Optional) Identify any unnecessary steps: Request to eliminate any steps that are not essential to the process.</li></ul><h4>Example:</h4><p><em>“I would like to purchase a house. I know that I need to perform steps like selecting a location, looking at various houses, considering house features, and making an offer. Provide a complete sequence of steps for me, fill in any missing steps, and identify any unnecessary steps.”</em></p><h4>Real-world scenario:</h4><p>In the scenario depicted in Figure 6, the task “T” is to plan an optimal 12-hour itinerary including visit points such as Vaduz, Bregenz, Lindau, Stein am Rhein, with the starting and ending point at Zurich City. The known steps “A, B, C” are the visit points themselves and the desire to spend about 30 minutes to 1 hour at each location, as well as the need to start the day at 8:00 AM and consider travel time between locations.</p><p>The Recipe Pattern will guide the AI to structure a travel itinerary that includes all necessary steps such as travel times, order of visit points based on geographical efficiency, and the duration of stay at each location. It will also include any essential activities or sights at each location and identify any points that are not feasible, such as visiting Mount Everest in the same trip, which is an unnecessary and impossible step within the given constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*ldk6-fctQt06lcDu" /><figcaption><strong><em>Figure 6: Recipe Pattern Scenario: Travel Itinerary Planner using ChatGPT</em></strong></figcaption></figure><p>Check out the full prompt pattern and the conversation with ChatGPT by<a href="https://chat.openai.com/share/8cf3e59f-c9c6-4ac8-b299-cb61eb29601b"> clicking here.</a></p><h3>Summary</h3><p>In this article, we’ve delved into the intricacies of prompt engineering, unveiling how structured inputs can significantly enhance the productivity of both individuals and businesses when interacting with generative AI like ChatGPT. From the artful Persona Pattern, which tailors responses to specific user identities, to the meticulous Cognitive Verifier Pattern, which ensures precise information gathering — each pattern serves as a key to unlocking the vast capabilities of AI. We examined the transformative power of the Recipe Pattern, guiding users through complex processes with step-by-step clarity. These patterns are not just theoretical; they are practical tools for improving efficiency and decision-making in real-world scenarios.</p><p><em>Reach out to me at </em><a href="https://www.linkedin.com/in/dipanjans/"><em>my LinkedIn</em></a><em> or </em><a href="https://djsarkar.com/"><em>my website</em></a><em> if you want to connect.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f2e40d819b1" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/three-popular-chatgpt-prompt-engineering-patterns-for-life-and-business-productivity-0f2e40d819b1">Three Popular ChatGPT Prompt Engineering Patterns for Life and Business Productivity</a> was originally published in <a href="https://pub.towardsai.net">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scalable and Blazing Fast Similarity Search With Milvus Vector Database]]></title>
            <link>https://pub.towardsai.net/scalable-and-blazing-fast-similarity-search-with-milvus-vector-database-d221706e605a?source=rss-6278d12b0682------2</link>
            <guid isPermaLink="false">https://medium.com/p/d221706e605a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Dipanjan (DJ) Sarkar]]></dc:creator>
            <pubDate>Mon, 27 Jun 2022 00:02:51 GMT</pubDate>
            <atom:updated>2022-06-27T00:02:51.220Z</atom:updated>
            <content:encoded><![CDATA[<h4>Store, index, manage and search trillions of document vectors in milliseconds!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u2jYf2wd5nxuY8fU.png" /><figcaption>Source: <a href="https://milvus.io/blog/scalable-and-blazing-fast-similarity-search-with-milvus-vector-database.md">Milvus Blog</a></figcaption></figure><h3>Introduction</h3><p>In this article, we will cover some interesting aspects relevant to vector databases and similarity search at scale. In today’s rapidly evolving world, we see new technology, new businesses, new data sources and consequently we will need to keep using new ways to store, manage and leverage this data for insights. Structured, tabular data has been stored in relational databases for decades, and Business Intelligence thrives on analyzing and extracting insights from such data. However, considering the current data landscape,</p><blockquote><a href="https://mitsloan.mit.edu/ideas-made-to-matter/tapping-power-unstructured-data">“over 80–90% of data is unstructured information like text, video, audio, web server logs, social media, and more”</a></blockquote><p>organizations have been leveraging the power of machine learning and deep learning to try and extract insights from such data as traditional query-based methods may not be enough or even possible. There is a huge, untapped potential to extract valuable insights from such data and we are only getting started!</p><blockquote>“Since most of the world’s data is unstructured, an ability to analyze and act on it presents a big opportunity.”</blockquote><blockquote>— Mikey Shulman, Head of ML, Kensho</blockquote><p>Unstructured data, as the name suggests, does not have an implicit structure, like a table of rows and columns (hence called tabular or structured data). Unlike structured data, there is no easy way to store the contents of unstructured data within a relational database. There are three main challenges with leveraging unstructured data for insights:</p><ul><li><strong>Storage:</strong> Regular relational databases are good for holding structured data. While you can use NoSQL databases to store such data, it becomes an additional overhead to process such data to extract the right representations to power AI applications at a scale</li><li><strong>Representation:</strong> Computers don’t understand text or images as we do. They only understand numbers and we need to convert unstructured data into some useful numeric representation, typically vectors or embeddings.</li><li><strong>Querying:</strong> You can’t query unstructured data directly based on definite conditional statements like SQL for structured data. Imagine, a simple example of you trying to search for similar shoes given a photo of your favorite pair of shoes! You can’t use raw pixel values for search, nor can you represent structured features like shoe shape, size, style, color, and more. Now imagine having to do this for millions of shoes!</li></ul><p>Hence, in order for computers to understand, process, and <strong><em>represent </em></strong>unstructured data, we typically convert them into dense vectors, often called embeddings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*L6Jz_fpvoIQD09Dc8286vA.png" /><figcaption>Representing Images as Dense Embedding Vectors; Source: <a href="https://zilliz.com/learn/embedding-generation">Embedding Generation</a></figcaption></figure><p>There exists a variety of methodologies especially leveraging deep learning, including convolutional neural networks (CNNs) for visual data like images and Transformers for text data which can be used to transform such unstructured data into embeddings. <a href="https://zilliz.com/">Zilliz</a> has <a href="https://zilliz.com/learn/embedding-generation"><em>an excellent article covering different embedding techiques</em></a>!</p><p>Now storing these embedding vectors is not enough. One also needs to be able to query and find out similar vectors. Why do you ask? A majority of real-world applications are powered by vector similarity search for AI-based solutions. This includes visual (image) search in Google, recommendations systems in Netflix or Amazon, text search engines in Google, multi-modal search, data de-duplication, and many more!</p><p>Storing, managing, and querying vectors at scale is not a simple task. You need specialized tools for this and vector databases are the most effective tool for the job! In this article we will cover the following aspects:</p><ul><li>Vectors and Vector Similarity Search</li><li>What is a Vector Database?</li><li>Milvus — The World’s Most Advanced Vector Database</li><li>Performing visual image search with Milvus — A use-case blueprint</li></ul><p>Let’s get started!</p><h3>Vectors and Vector Similarity Search</h3><p>Earlier, we established the necessity of representing unstructured data like images and text as vectors, since computers can only understand numbers. We typically leverage AI models, to be more specific deep learning models to convert unstructured data into numeric vectors which can be read by machines. Typically these vectors are basically a list of floating-point numbers which collectively represent the underlying item (image, text, etc.).</p><h4>Understanding Vectors</h4><p>Considering the field of natural language processing (NLP) we have many word embedding models like <a href="https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa">Word2Vec, GloVe, and FastText</a> which can help represent words as numeric vectors. With advancements over time, we have seen the rise of <a href="https://arxiv.org/abs/1706.03762">Transformer</a> models like <a href="https://jalammar.github.io/illustrated-bert/">BERT</a> which can be leveraged to learn contextual embedding vectors and better representations for entire sentences and paragraphs.</p><p>Similarly, in the field of computer vision, we have models like <a href="https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf">Convolutional Neural Networks (CNNs)</a> which can help in learning representations from visual data such as images and videos. With the rise of Transformers, we also have <a href="https://arxiv.org/abs/2010.11929">Vision Transformers</a> which can perform better than regular CNNs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JpPbwdWCAmmWOoJd.png" /><figcaption>Sample workflow for extracting insights from unstructured data; Source: <a href="https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md">Milvus Blog</a></figcaption></figure><p>The advantage with such vectors is that we can leverage them for solving real-world problems such as visual search, where you typically upload a photo and get search results including visually similar images. Google has this as a very popular feature in their search engine as depicted in the following example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*YqYjhWuDoOhbYj8QA0sJNA.png" /><figcaption>An example of Google’s Visual Image Search; Source: Created by Author</figcaption></figure><p>Such applications are powered by data vectors and vector similarity search. If you consider two points in an X-Y cartesian coordinate space. The distance between two points can be computed as a simple euclidean distance depicted by the following equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MdLqZwITzJoQlazn.png" /><figcaption>2-D Euclidean Distance; Source: <a href="https://en.wikipedia.org/wiki/Euclidean_distance">Wikipedia</a></figcaption></figure><p>Now imagine each data point is a vector having D-dimensions, you could still use euclidean distance or even other distance metrics like hamming or cosine distance to find out how close the two data points are to each other. This can help build a notion of closeness or similarity which could be used as a quantifiable metric to find similar items given a reference item using their vectors.</p><h4>Understanding Vector Similarity Search</h4><p>Vector similarity search, often known as nearest neighbor (NN) search, is basically the process of computing pairwise similarity (or distances) between a reference item (for which we want to find similar items) and a collection of existing items (typically in a database) and returning the top ‘k’ nearest neighbors which are the top ‘k’ most similar items. The key component to compute this similarity is the similarity metric which can be Euclidean distance, inner product, cosine distance, Hamming distance, etc. The smaller the distance, the more similar are the vectors.</p><p>The challenge with the exact nearest neighbor (NN) search is scalability. You need to compute N-distances (assuming N existing items) every time to get similar items. This can be super slow especially if you don’t store and index the data somewhere (like a vector database!). To speed up computation, we typically leverage approximate nearest neighbor search which is often called ANN search which ends up storing the vectors into an index. The index helps in storing these vectors in an intelligent way to enable quick retrieval of ‘approximately’ similar neighbors for a reference query item. Typical ANN indexing methodologies include:</p><ul><li><strong>Vector Transformations:</strong> This includes adding additional transformations to the vectors like dimension reduction (e.g PCA \ t-SNE), rotation, and so on</li><li><strong>Vector Encoding:</strong> This includes applying techniques based on data structures like Locality Sensitive Hashing (LSH), Quantization, Trees, etc. which can help in faster retrieval of similar items</li><li><strong>Non-Exhaustive Search Methods: </strong>This is mostly used to prevent exhaustive search and includes methods like neighborhood graphs, inverted indices, etc.</li></ul><p>This establishes the case that to build any vector similarity search application, you need a database that can help you with efficient storing, indexing, and querying (search) at scale. Enter vector databases!</p><h3>What is a Vector Database?</h3><p>Given that we now understand how vectors can be used to represent unstructured data and how vector search works, we can combine the two concepts together to build a vector database.</p><p>Vector databases are scalable data platforms to store, index, and query across embedding vectors that are generated from unstructured data (images, text, etc.) using deep learning models.</p><p>Handling a massive number of vectors for similarity search (even with indices) can be super expensive. Despite this, the best and most advanced vector databases should allow you to insert, index, and search across millions or billions of target vectors, in addition to specifying an indexing algorithm and similarity metric of your choice.</p><p>Vector databases mainly should satisfy the following key requirements considering a robust database management system to be used in the enterprise:</p><ol><li><strong>Scalable</strong>: Vector databases should be able to index and run an approximate nearest neighbor search for billions of embedding vectors</li><li><strong>Reliable</strong>: Vector databases should be able to handle internal faults without data loss and with minimal operational impact, i.e be fault-tolerant</li><li><strong>Fast</strong>: Query and write speeds are important for vector databases. For platforms such as Snapchat and Instagram, which can have hundreds or thousands of new images uploaded per second, speed becomes an incredibly important factor.</li></ol><p>Vector databases don’t just store data vectors. They are also responsible for using efficient data structures to index these vectors for fast retrieval and supporting CRUD (create, read, update and delete) operations. Vector databases should also ideally support attribute filtering which is filtering based on metadata fields which are usually scalar fields. A simple example would be retrieving similar shoes based on the image vectors for a specific brand. Here brand would be the attribute based on which filtering would be done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sB_RErQhXTfzA2rN.png" /><figcaption>The Milvus filtering mechanism; Source: <a href="https://zilliz.com/learn/attribute-filtering">Attribute Filtering</a></figcaption></figure><p>The figure above showcases how <a href="https://milvus.io/">Milvus</a>, the vector database we will talk about shortly, uses attribute filtering. <a href="https://milvus.io/">Milvus</a> introduces the concept of a bitmask to the filtering mechanism to keep similar vectors with a bitmask of 1 based on satisfying specific attribute filters. More details on this are <a href="https://zilliz.com/learn/attribute-filtering"><em>here</em></a>.</p><h3>Milvus — The World’s Most Advanced Vector Database</h3><p><a href="https://milvus.io/">Milvus</a> is an open-source vector database management platform built specifically for massive-scale vector data and streamlining machine learning operations (MLOps).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X0NnDHdsO8pIEGQ_.png" /><figcaption>Milvus; Source: <a href="https://milvus.io/blog/scalable-and-blazing-fast-similarity-search-with-milvus-vector-database.md">Milvus Blog</a></figcaption></figure><p><a href="https://zilliz.com/">Zilliz</a>, is the organization behind building <a href="https://milvus.io/">Milvus</a>, the world’s most advanced vector database, to accelerate the development of next-generation data fabric. Milvus is currently a graduation project at the <a href="https://lfaidata.foundation/">LF AI &amp; Data Foundation</a> and focuses on managing massive unstructured datasets for storage and search. The platform’s efficiency and reliability simplify the process of deploying AI models and MLOps at scale. Milvus has broad applications spanning drug discovery, computer vision, recommendation systems, chatbots, and much more.</p><h4>Key Features of Milvus</h4><p>Milvus is packed with useful features and capabilities, such as:</p><ul><li><strong>Blazing search speeds on a trillion vector datasets: </strong>Average latency of vector search and retrieval has been measured in milliseconds on a trillion vector datasets.</li><li><strong>Simplified unstructured data management</strong>: Milvus has rich APIs designed for data science workflows.</li><li><strong>Reliable, always on vector database:</strong> Milvus’ built-in replication and failover/failback features ensure data and applications can maintain business continuity always.</li><li><strong>Highly scalable and elastic:</strong> Component-level scalability makes it possible to scale up and down on demand.</li><li><strong>Hybrid search:</strong> In addition to vectors, Milvus supports data types such as Boolean, String, integers, floating-point numbers, and more. Milvus pairs scalar filtering with a powerful vector similarity search (as seen in the shoe similarity example earlier).</li><li><strong>Unified Lambda structure:</strong> Milvus combines stream and batch processing for data storage to balance timeliness and efficiency.</li><li><a href="https://milvus.io/docs/v2.0.x/timetravel_ref.md"><strong>Time Travel:</strong></a> Milvus maintains a timeline for all data insert and delete operations. It allows users to specify timestamps in a search to retrieve a data view at a specified point in time.</li><li><strong>Community-supported &amp; Industry recognized:</strong> With over 1,000 enterprise users, 10.5K+ stars on <a href="https://github.com/milvus-io/milvus">GitHub</a>, and an active open-source community, you’re not alone when you use Milvus. As a graduate project under the <a href="https://lfaidata.foundation/">LF AI &amp; Data Foundation</a>, Milvus has institutional support.</li></ul><h4>Existing Approaches to Vector Data Management and Search</h4><p>A common way to build an AI system powered by vector similarity search is to pair algorithms like Approximate Nearest Neighbor Search (ANNS) with open-source libraries such as:</p><ul><li><a href="https://ai.facebook.com/tools/faiss/"><strong>Facebook AI Similarity Search (FAISS):</strong></a><strong> </strong>This framework enables efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It supports indexing capabilities like inverted multi-index and product quantization</li><li><a href="https://github.com/spotify/annoy"><strong>Spotify’s Annoy (Approximate Nearest Neighbors Oh Yeah)</strong></a><strong>: </strong>This framework uses <a href="http://en.wikipedia.org/wiki/Locality-sensitive_hashing#Random_projection">random projections</a> and builds up a tree to enable ANNS at scale for dense vectors</li><li><a href="https://github.com/google-research/google-research/tree/master/scann"><strong>Google’s ScaNN (Scalable Nearest Neighbors):</strong></a> This framework performs an efficient vector similarity search at scale. Consists of implementations, which include search space pruning and quantization for Maximum Inner Product Search (MIPS)</li></ul><p>While each of these libraries is useful in its own way, due to several limitations, these algorithm-library combinations are not equivalent to a full-fledged vector data management system like Milvus. We will discuss some of these limitations now.</p><h4>Limitations of Existing Approaches</h4><p>Existing approaches used for managing vector data as discussed in the previous section have the following limitations:</p><ol><li><strong>Flexibility:</strong> Existing systems typically store all data in main memory, hence they cannot be run in distributed mode across multiple machines easily and are not well-suited for handling massive datasets</li><li><strong>Dynamic data handling:</strong> Data is often assumed to be static once fed into existing systems, complicating processing for dynamic data and making near real-time search impossible</li><li><strong>Advanced query processing:</strong> Most tools do not support advanced query processing (e.g., attribute filtering, hybrid search, and multi-vector queries), which is essential for building real-world similarity search engines supporting advanced filtering.</li><li><strong>Heterogeneous computing optimizations:</strong> Few platforms offer optimizations for heterogeneous system architectures on both CPUs and GPUs (excluding FAISS), leading to efficiency losses.</li></ol><p><a href="https://milvus.io/">Milvus</a> attempts to overcome all of these limitations and we will discuss this in detail in the next section.</p><h4>The Milvus Advantage —Understanding Knowhere</h4><p><a href="https://milvus.io/">Milvus</a> tries to tackle and successfully solve the limitations of existing systems build on top of inefficient vector data management and similarity search algorithms in the following ways:</p><ul><li>It enhances flexibility by offering support for a variety of application interfaces (including SDKs in Python, Java, Go, C++, and RESTful APIs)</li><li>It supports multiple vector index types (e.g., quantization-based indexes and graph-based indexes), and advanced query processing</li><li>Milvus handles dynamic vector data using a log-structured merge tree (LSM tree), keeping data insertions and deletions efficient and searches humming along in real-time</li><li>Milvus also provides optimizations for heterogeneous computing architectures on modern CPUs and GPUs, allowing developers to adjust systems for specific scenarios, datasets, and application environments</li></ul><p>Knowhere, the vector execution engine of Milvus, is an operator interface for accessing services in the upper layers of the system and vector similarity search libraries like Faiss, Hnswlib, and Annoy in the lower layers of the system. In addition, Knowhere is also in charge of heterogeneous computing. Knowhere controls on which hardware (eg. CPU or GPU) to execute index building and search requests. This is how Knowhere gets its name — knowing where to execute the operations. More types of hardware including DPU and TPU will be supported in future releases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*mfoZulvlZzwwiMhLlnr3Pw.png" /><figcaption>Knowhere architecture in Milvus; Source: <a href="https://milvus.io/blog/deep-dive-8-knowhere.md">Knowhere Deep Dive</a></figcaption></figure><p>Computation in Milvus mainly involves vector and scalar operations. Knowhere only handles the operations on vectors in Milvus. The figure above illustrates the Knowhere architecture in Milvus. The bottom-most layer is the system hardware. The third-party index libraries are on top of the hardware. Then Knowhere interacts with the index node and query node on the top via CGO. Knowhere not only further extends the functions of Faiss but also optimizes the performance and has several advantages including support for BitsetView, support for more similar metrics, support for AVX512 instruction set, automatic SIMD instruction selection, and other performance optimizations. Details can be found <a href="https://milvus.io/blog/deep-dive-8-knowhere.md"><em>here</em></a>.</p><h4>Milvus Architecture</h4><p>The following figure showcases the overall architecture of the <a href="https://milvus.io/">Milvus</a> platform. Milvus separates data flow from control flow and is divided into four layers that are independent in terms of scalability and disaster recovery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*QOJ_Zivjx-iOyblu.png" /><figcaption>Milvus Architecture; Source: <a href="https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md">Milvus Overview</a></figcaption></figure><ul><li><strong>Access layer: </strong>The access layer is composed of a group of stateless proxies and serves as the front layer of the system and endpoint to users.</li><li><strong>Coordinator service: </strong>The coordinator service is responsible for cluster topology node management, load balancing, timestamp generation, data declaration, and data management</li><li><strong>Worker nodes: </strong>The worker, or execution, node executes instructions issued by the coordinator service and the data manipulation language (DML) commands initiated by the proxy. A worker node in Milvus is similar to a data node in <a href="https://hadoop.apache.org/">Hadoop</a>, or a region server in HBase</li><li><strong>Storage:</strong> This is the cornerstone of Milvus, responsible for data persistence. The storage layer is comprised of <strong>meta store</strong>, <strong>log broker</strong> and <strong>object storage</strong></li></ul><p>Do check out more details about the architecture <a href="https://milvus.io/docs/v2.0.x/four_layers.md">here</a>!</p><h3>Performing visual image search with Milvus — A use-case blueprint</h3><p>Open-source vector databases like Milvus make it possible for any business to create their own visual image search system with a minimum number of steps. Developers can use pre-trained AI models to convert their own image datasets into vectors, and then leverage Milvus to enable searching for similar products by image. Let’s look at the following blueprint of how to design and build such a system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UbJ7NPS1RZcMQPDP.jpg" /><figcaption>Workflow for Visual Image Search; Source: <a href="https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search/quick_deploy">Reverse Image Search</a></figcaption></figure><p>In this workflow, we can use an open-source framework like <a href="https://github.com/towhee-io/towhee">towhee</a> to leverage a pre-trained model like ResNet-50 and extract vectors from images, store and index these vectors with ease in Milvus and also store a mapping of image IDs to the actual pictures in a MySQL database. Once the data is indexed we can upload any new image with ease and perform an image search at scale using Milvus. The following figure shows a sample visual image search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XLI6edo0vVckGgUn.png" /><figcaption>Sample Visual Search Example; Source: <a href="https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search/quick_deploy">Reverse Image Search</a></figcaption></figure><blockquote><a href="https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search/quick_deploy">Do check out the detailed tutorial which has been open-sourced on GitHub thanks to Milvus.</a></blockquote><h3>Conclusion</h3><p>We’ve covered a fair amount of ground in this article. We started with challenges in representing unstructured data, leveraging vectors, and vector similarity search at scale with Milvus, an open-source vector database. We discussed details on how Milvus is structured and the key components powering it and a blueprint of how to solve a real-world problem, visual image search with Milvus. Do give it a try and start solving your own real-world problems with <a href="https://milvus.io/">Milvus</a>!</p><p>The content in this article has been adapted from my article on <a href="https://milvus.io/blog/scalable-and-blazing-fast-similarity-search-with-milvus-vector-database.md"><em>the Milvus Blog</em></a>.</p><blockquote>Liked this article? Do <a href="https://www.linkedin.com/in/dipanzan/">reach out to me</a> to discuss more on it or give feedback!</blockquote><p><a href="https://www.linkedin.com/in/dipanzan/">Dipanjan Sarkar - Lead Data Scientist - SIT Academy | LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d221706e605a" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/scalable-and-blazing-fast-similarity-search-with-milvus-vector-database-d221706e605a">Scalable and Blazing Fast Similarity Search With Milvus Vector Database</a> was originally published in <a href="https://pub.towardsai.net">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adversarial Examples in Deep Learning — A Primer]]></title>
            <link>https://medium.com/data-science/adversarial-examples-in-deep-learning-a-primer-feae6153d89?source=rss-6278d12b0682------2</link>
            <guid isPermaLink="false">https://medium.com/p/feae6153d89</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Dipanjan (DJ) Sarkar]]></dc:creator>
            <pubDate>Thu, 05 Nov 2020 15:20:43 GMT</pubDate>
            <atom:updated>2021-02-01T14:42:43.527Z</atom:updated>
            <content:encoded><![CDATA[<h3>Adversarial Examples in Deep Learning — A Primer</h3><h4>Introducing adversarial examples in deep learning vision models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yCmsnioukUFf-151_ziRBQ.jpeg" /><figcaption>Source: <a href="https://pixabay.com/photos/structure-blue-rust-texture-3215624">Pixabay</a></figcaption></figure><h3>Introduction</h3><p>We have seen the advent of state-of-the-art (SOTA) deep learning models for computer vision ever since we started getting bigger and better compute (GPUs and TPUs), more data (ImageNet etc.) and easy to use open-source software and tools (TensorFlow and PyTorch). Every year (and now every few months!) we see the next SOTA deep learning model dethrone the previous model in terms of Top-k accuracy for benchmark datasets. The following figure depicts some of the latest SOTA deep learning vision models (and doesn’t depict some like Google’s BigTransfer!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R_mA26nqowS5YdZP.png" /><figcaption>SOTA Deep Learning Vision Models (Source: <a href="https://arxiv.org/abs/1905.11946">https://arxiv.org/abs/1905.11946</a>)</figcaption></figure><p>However most of these SOTA deep learning models are brought down to their knees when it tries to make predictions on a specific class of images, called as adversarial images. The whole idea of an adversarial example can be a natural example or a synthetic example. We will look at a few examples in this article to get familiar with different adversarial examples and attacks.</p><h3>Adversarial Examples</h3><p>A natural adversarial example is a natural, organic image which is tough for the model to comprehend. A synthetic adversarial example is where an attacker (a malicious user) purposely injects some noise into an image which visually remains very similar to the original image but the model ends up making a vastly different (and wrong) prediction. Let’s look at a few of these in more detail!</p><h3>Natural Adversarial Examples</h3><p>These examples, as defined in the paper <a href="https://arxiv.org/abs/1907.07174"><em>‘Natural Adversarial Examples, Hendrycks et al.’ </em></a>are <em>real-world, unmodified, and naturally occurring examples that cause classifier accuracy to significantly degrade</em>. They have introduced two new datasets of natural adversarial examples. The first dataset contains 7,500 natural adversarial examples for ImageNet classifiers and serves as a hard ImageNet classifier test set, called IMAGENET-A. The following figure shows some of these adversarial examples from the ImageNet-A dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/0*b17eCCaO1mp6q1XQ.png" /><figcaption>ResNet-50 fails horribly on examples from ImageNet-A (Source: <a href="https://arxiv.org/abs/1907.07174)">https://arxiv.org/abs/1907.07174)</a></figcaption></figure><p>You can clearly see how wrong (and silly!) are the predictions of a state-of-the-art (SOTA) ResNet-50 model on the above examples. In-fact the DenseNet-121 pre-trained model obtains an accuracy of only 2% on ImageNet-A!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D43KUyo_bOAbMTWb.png" /><figcaption>Source: <a href="https://github.com/dipanjanS/adversarial-learning-robustness">https://github.com/dipanjanS/adversarial-learning-robustness</a></figcaption></figure><p>The authors have also curated an adversarial out-of-distribution detection dataset called IMAGENET-O, which they claim is the first out-of-distribution detection dataset created for ImageNet models. The following figure shows some interesting examples of ResNet-50 inference on images from the ImageNet-O dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/0*6ORyf0d4qHTz0w7y.png" /><figcaption>ResNet-50 fails horribly on examples from ImageNet-O (Source: <a href="https://arxiv.org/abs/1907.07174)">https://arxiv.org/abs/1907.07174)</a></figcaption></figure><p>The examples are indeed interesting and showcase the limitations of SOTA pre-trained vision models on some of these images which are more complex for these models to interpret. Some of the reasons of failure can be attributed to what deep learning models are trying to focus on when making predictions for a specific image. Let’s look at some more examples to try and understand this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/0*_lbTNnyHyzArGUHc.png" /><figcaption>Natural adversarial examples in ImageNet-A (Source: <a href="https://arxiv.org/abs/1907.07174)">https://arxiv.org/abs/1907.07174)</a></figcaption></figure><p>Based on the examples showcased in the figure above, it is pretty clear that there are some specific patterns w.r.t mis-interpretations made by deep learning vision models. For instance:</p><ul><li>Candles are predicted as a jack-o’-lanterns, despite the absence of a pumpkin due to model focusing more on aspects like the flame and its illumination</li><li>Dragonfly is predicted as a skunk or a banana due to the model focusing more on color and texture</li><li>Mushroom being classified as a nail because the models learn to associate certain elements together e.g. wood — nails</li><li>Models also end up suffering from overgeneralization problems e.g. shadows to sundials</li></ul><p>The overall performance of SOTA deep learning vision models are pretty poor on these examples as depicted in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*K6VFfkX-xq8lrPbI.png" /><figcaption>SOTA deep learning vision models performance on ImageNet-A (Source: <a href="https://arxiv.org/abs/1907.07174)">https://arxiv.org/abs/1907.07174)</a></figcaption></figure><p>The sad part is that robust adversarial training methods hardly help in tacking problems associated with mis-interpreting natural adversarial examples as mentioned in the same paper by Hendrycks et al. Some of these methods include training against specific synthetic attacks like Projected Gradient Descent (PGD) and Fast Gradient Sign Method (FGSM) which we will look at in more detail in subsequent articles. Luckily these methods work well for handling malicious synthetic attacks which are usually a larger concern.</p><h3>Synthetic Adversarial Examples</h3><p>These examples basically involve artificially inducing some noise in an input image such that visually it still remains very similar to the original image, but the infused noise ends up degrading classifier accuracy. While there are a wide variety of synthetic adversarial attacks, all of them operate on a few core set of principles as depicted in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*26J3IiePR-mLW1bF.png" /><figcaption>Source: <a href="https://github.com/dipanjanS/adversarial-learning-robustness">https://github.com/dipanjanS/adversarial-learning-robustness</a></figcaption></figure><p>The focus is always to figure out a way to perfect the noise \ perturbation tensor (matrix of values) which can be super-imposed on top of the original image such that these perturbations are invisible to the human eye but ends up making the deep learning model fail in making correct predictions. The example depicted above showcases a Fast Gradient Sign Method (FGSM) attack where we add in a small multiplier to the sign of the gradients of the input image and super-impose with the image of a panda making the model fail in its prediction thinking that the image is that of a gibbon. The following graphic showcases some of the more popular types of adversarial attacks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yi1BL2H_HhyMs4Qd.png" /><figcaption>Source: <a href="https://github.com/dipanjanS/adversarial-learning-robustness">https://github.com/dipanjanS/adversarial-learning-robustness</a></figcaption></figure><h3>What’s Next?</h3><p>In the next few articles we will discuss about each of the above mentioned adversarial attack methodologies and showcase with hands-on code examples how you can fool the latest and best SOTA vision models. Stay tuned!</p><p>The content in this article has been adapted from <a href="https://blog.djsarkar.ai/adversarial-learning-attacks-1"><em>my recent blog post</em></a> on adversarial learning done by <a href="https://www.linkedin.com/in/dipanzan/">myself </a>and <a href="https://in.linkedin.com/in/sayak-paul">Sayak</a> and you can find detailed examples in <a href="https://github.com/dipanjanS/adversarial-learning-robustness">this GitHub Repository</a>.</p><blockquote>Liked this article? Do <a href="https://www.linkedin.com/in/dipanzan/">reach out to me</a> to discuss more on it or give feedback!</blockquote><p><a href="https://www.linkedin.com/in/dipanzan/">Dipanjan Sarkar - Lead Data Scientist - SIT Academy | LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=feae6153d89" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/adversarial-examples-in-deep-learning-a-primer-feae6153d89">Adversarial Examples in Deep Learning — A Primer</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Robust Production-Ready Deep Learning Vision Models in Minutes]]></title>
            <link>https://medium.com/google-developer-experts/building-robust-production-ready-deep-learning-vision-models-in-minutes-acd716f6450a?source=rss-6278d12b0682------2</link>
            <guid isPermaLink="false">https://medium.com/p/acd716f6450a</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dipanjan (DJ) Sarkar]]></dc:creator>
            <pubDate>Fri, 20 Sep 2019 11:51:10 GMT</pubDate>
            <atom:updated>2019-09-20T12:17:27.833Z</atom:updated>
            <content:encoded><![CDATA[<h4>A comprehensive hands-on guide to TensorFlow Serving</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-uxB0EnnMX-rAoTFp8xZew.png" /></figure><h3>Introduction</h3><p>Thanks to faster compute, better storage and easy to use software, deep learning based solutions are definitely seeing the light of the day coming out from the proof-of-concept tunnel into the real-world! We are seeing widespread adoption of deep learning models across diverse domains in the industry including healthcare, finance, retail, tech, logistics, food-tech, agriculture amongst many others! Considering the fact that deep learning models are resource hungry and often compute-heavy, we need to pause for a moment and think about model inference and serving times, when consumed by end-users.</p><p>Training and performing model inference on static batches of data while prototyping is necessary. However this methodology and code artifacts don’t make the cut when we want our model to be consumed in the form of a web service or API. We usually need a robust and low-latency model serving layer which should be able to serve model inference requests faster and at ease. This article tries to give a simple, yet comprehensive hands-on overview of how to leverage TensorFlow Serving to serve deep learning models for computer vision. The article will be pretty extensive and will also contain a lot of hands-on code which you can adopt for your own practices. We will be covering the following topics in this article:</p><ul><li><strong>What is Serving?</strong></li><li><strong>TensorFlow Serving Overview</strong></li><li><strong>TensorFlow Serving Architecture</strong></li><li><strong>Model Serving Methodology</strong></li><li><strong>Main Objective — Building an Apparel Classifier</strong></li><li><strong>Training a simple CNN Model</strong></li><li><strong>Fine-tuning a pre-trained ResNet-50 CNN Model</strong></li><li><strong>Saving models for TensorFlow Serving</strong></li><li><strong>Serving models with CPU Inference</strong></li><li><strong>Serving models with Docker for GPU Inference</strong></li><li><strong>Bonus: Building an Apparel Classifier API with Flask &amp; TensorFlow Serving</strong></li></ul><p>Feel free to reach out to me if you have better ways of making deployments and serving simpler and easier to understand. I’m by no means an expert but I sincerely hope you find this useful. Let’s get started!</p><h3>What is Serving?</h3><p>Serving, or to be more specific, model serving, is a technique to use or apply a model for inference after it has been trained. Typically this involves having a server-client architecture and serving or exposing our trained models for inference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8ELDpw3oWjLLo-q-.png" /><figcaption><a href="https://medium.com/tensorflow/training-and-serving-ml-models-with-tf-keras-fd975cc0fa27">Source</a></figcaption></figure><p>Considering the problem we would be focusing on in this article — image classification, we would have some specific access patterns for our models. On the end-user or client end we would have an input image which need to be tagged or classified. We would need to convert this image to a specific encoded format, wrap it in a specific JSON payload with headers and send it to a web service \ API which should typically be hosted on a server. The API call would invoke our pre-trained model to make the prediction and serve the inference result as a JSON response from the server to the client.</p><h3><strong>TensorFlow Serving Overview</strong></h3><p>There are many excellent articles for TensorFlow serving including <a href="https://www.tensorflow.org/tfx/guide/serving">the official documentation</a> which you should definitely check out. In this section I will give a brief overview of the essentials of TensorFlow Serving and why we need it. For productionizing deep learning or machine learning models, we need a robust system which can help us in making our models serve requests with speed and consistency. TensorFlow Serving is such a framework, which is a flexible, high-performance serving system for machine learning models, designed specifically for production environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*LM8nxisx4VAdC9G82OVtRg.png" /></figure><p>While you can always build your own serving pipeline and system, there are several benefits to using TensorFlow Serving.</p><ul><li>It is very easy to deploy new algorithms and experiments, while keeping the same server architecture and APIs.</li><li>Can not only be used with just TensorFlow models but can be easily extended to serve other types of models and data.</li><li>Can be used to serve multiple models and model versions simultaneously.</li><li>Efficient model lifecycle management.</li><li>Can be integrated with tools like Docker and Kubernetes for more scalability.</li></ul><h3>TensorFlow Serving Architecture</h3><p>In this section we will take a brief look into the essentials of the architecture behind TensorFlow Serving. For a detailed deep dive into the architecture, I recommend checking out <a href="https://www.tensorflow.org/tfx/serving/architecture">the official documentation</a> once again.</p><p><strong><em>Servables</em></strong> are the central abstraction in the TensorFlow Serving architecture. Typically servables are the underlying objects that clients use to perform computations like model inference. A single Servable might include one model or even multiple models. The following figure showcases the typical life of a Servable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6oeTiEJmmg4raGkCkwr4iw.png" /><figcaption>Life of a Servable (<a href="https://www.tensorflow.org/tfx/serving/architecture">Source</a>)</figcaption></figure><p>Typically there are several components in this architecture. A <strong><em>servable stream</em></strong> is the sequence of versions of a servable. <strong><em>Loaders</em></strong> manage a servable’s life cycle and has APIs for loading and unloading a servable. <strong><em>Sources</em></strong> are plugin modules that find and provide servables and can maintain state that is shared across multiple servables. <strong><em>Managers</em></strong> handle the full lifecycle of servables, including loading, unloading and serving servables. Using the standard TensorFlow Serving APIs, <strong><em>TensorFlow Serving Core</em></strong> manages the lifecycle and metrics of servables.</p><h3><strong>Model Serving Methodology</strong></h3><p>Before we talk about our main objective and train our models, let’s discuss briefly the model serving methodology we would follow, assuming we have trained a few vision-based deep learning models. The key steps to be followed are displayed in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*bZJkbwx9Vx5TR38jcmF70Q.png" /></figure><p>Thus the key steps we would be focusing on for serving models include the following:</p><ul><li><strong>Model Training:</strong> To serve any models, we would need to train the models first! In this article we will leverage the <strong>tf.keras</strong> API in TensorFlow which helps us train deep learning models easily.</li><li><strong>Exporting the models:</strong> Here we would need to export our trained models into a specific format which can be used by TF Serving. TensorFlow provides the <strong>SavedModel</strong>format as a universal format for exporting models. This creates a protobuf file in a well-defined directory hierarchy, and will also include a version number as depicted below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*z_oURX3FVEjYYjaWsCPFhA.png" /></figure><p>TensorFlow Serving allows us to select <a href="https://www.tensorflow.org/tfx/tutorials/overview">which version of a model, or “servable”</a> we want to use when we make inference requests. Each version will be exported to a different sub-directory under the given path as depicted in the figure above. TensorFlow provides a convenience function <strong>tf.saved_model.simple_save() </strong>which helps us in saving these models easily.</p><ul><li><strong>Hosting TensorFlow Serving Model Server:</strong> Here we would be using the TensorFlow Serving framework to host our saved models. We will focus on locally based TF Serving installation for CPU inference and also show how we can use a Docker container-based TF Serving instance for GPU inference. We also leverage the Flask framework on top of TF Serving to build our own custom serving API at the end of this article.</li><li><strong>Making Server Requests:</strong> Once the server up and running, we can make requests to it either via gRPC or HTTP. For both the methods, we typically create a payload message with necessary content and headers, and send it to the server. The server in turn should return a message that contains the prediction. We will use the <strong>requests</strong> module for HTTP requests.</li></ul><h3><strong>Main Objective — Building an Apparel Classifier</strong></h3><p>We will keep things simple here with regard to the key objective. We will build a simple apparel classifier by training models on the very famous <a href="https://github.com/zalandoresearch/fashion-mnist">Fashion MNIST</a> dataset based on Zalando’s article images — consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. The idea is to classify these images into an apparel category amongst 10 categories on which we will be training our models on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/0*VktuUmwUZBTk5HZ8.png" /><figcaption><a href="https://github.com/zalandoresearch/fashion-mnist">Fashion-MNIST samples</a> (by Zalando, MIT License).</figcaption></figure><p>We will build the following two deep learning CNN (Convolutional Neural Network) classifiers in this article before focusing on model serving.</p><ul><li><strong>A simple CNN trained from scratch</strong></li><li><strong>Fine-tuning a pre-trained ResNet-50 CNN</strong></li></ul><p>The intent of this article is to focus more on deployment and serving aspects hence we won’t spend a whole lot of time talking about model architectures or training and fine-tuning. Feel free to check out a brief on CNNs if necessary towards the very end of <a href="https://towardsdatascience.com/ai-powered-search-for-extra-terrestrial-intelligence-analyzing-radio-telescopic-data-c9e46741041"><em>this article</em></a><em>. </em>My testbed for this article was a <a href="https://cloud.google.com/deep-learning-vm/"><strong>Google Cloud Platform Deep Learning VM</strong></a> with a NVIDIA Tesla T4 which makes it really easy to carry out experiments on the cloud!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*hDu3dNc7AN2rOX-fpxr_wA.png" /></figure><h4>Loading Dependencies and Data</h4><p>Before we can train our deep learning models, let’s load up the necessary dependencies and our dataset to train our deep learning models.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bbb53e2170599ded4dd56847453ba6e9/href">https://medium.com/media/bbb53e2170599ded4dd56847453ba6e9/href</a></iframe><pre><strong>1.14.0</strong></pre><p>We use TensorFlow GPU version 1.14 here but this can easily be also extended to TensorFlow 2.0 considering we focus mostly on the <strong>tf.keras</strong> API. We can leverage TensorFlow itself now to load up the Fashion-MNIST dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/866fa3310a569f708b04881eaf15a972/href">https://medium.com/media/866fa3310a569f708b04881eaf15a972/href</a></iframe><pre><strong>Train_images.shape: (60000, 28, 28), of uint8<br>Test_images.shape: (10000, 28, 28), of uint8</strong></pre><p>Based on what we mentioned earlier, we have 60000 training and 10000 test images of size <strong>28x28</strong>. We will now start training our deep learning models.</p><h3><strong>Training a simple CNN Model</strong></h3><p>In this section, we will train a basic 2-layer CNN model from scratch. We do need to reshape our data before we train our model and the following code takes care of the same.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9240d89621f01b894bf593b173f3ac3/href">https://medium.com/media/b9240d89621f01b894bf593b173f3ac3/href</a></iframe><pre><strong>Train_images.shape: (60000, 28, 28, 1), of uint8<br>Test_images.shape: (10000, 28, 28, 1), of uint8</strong></pre><p>We can also take a look at how some of the images look like, as depicted in the following snapshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gNszQooPzVzoInGf7CUw5g.png" /></figure><p>We will now build our basic 2-layer CNN model architecture.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23bb8c5cc46ab9692561fdaffd45fd4b/href">https://medium.com/media/23bb8c5cc46ab9692561fdaffd45fd4b/href</a></iframe><p>Let’s train our model for 10 epochs and look at the performance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3549c827d309ab530f1359dc53f1cb2/href">https://medium.com/media/a3549c827d309ab530f1359dc53f1cb2/href</a></iframe><p>Do note that we train on 90% of the training data and validate on 10% of the training data. Performance is pretty decent on the validation set. Let’s save our model and then check the performance on the test dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a55e31a4c4595b17f67720994cfe665/href">https://medium.com/media/6a55e31a4c4595b17f67720994cfe665/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yK-JYYNV0M4UzX68ghpWLg.png" /></figure><p>Overall model performance on the test dataset gives us an <strong>f1-score</strong> of <strong>91% </strong>which is pretty neat!</p><h3><strong>Fine-tuning a pre-trained ResNet-50 CNN Model</strong></h3><p>Transfer learning is seeing unprecedented success in the world of computer vision and natural language processing with pre-trained models often outperforming training models from scratch. Here, we will use a ResNet-50 model which was pre-trained on the ImageNet dataset by fine-tuning it on the Fashion-MNIST dataset. The ResNet-50 model is a 50-convolutional block (several layers in each block) deep learning network built on the ImageNet database. This model has over 175+ layers in total and is a very deep network. ResNet stands for Residual Networks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*qdCZYPLJQIWlqgjQ.png" /><figcaption><a href="https://towardsdatascience.com/understanding-and-coding-a-resnet-in-keras-446d7ff84d33">Source</a></figcaption></figure><p>The ResNet-50 model which we will be using, consists of 5 stages, each with a Convolution and Identity block. Each convolution block has 3 convolution layers and each identity block also has 3 convolution layers. You can find some more information about this model in <a href="https://towardsdatascience.com/ai-powered-search-for-extra-terrestrial-intelligence-signal-classification-with-deep-learning-6c09de8fd57c">one of my articles</a>. The focus in this section will be taking the pre-trained ResNet-50 model and then perform complete fine-tuning of all the layers in the network. We will add the regular dense and output layers as usual.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7f34efcc9adef20d2bb242311e27138/href">https://medium.com/media/c7f34efcc9adef20d2bb242311e27138/href</a></iframe><p>This model is huge and you can see the evidence based on the number of trainable parameters! Before we can train the model, we need to convert our grayscale images into images with three channels since the ResNet model was trained on color images. Besides that, the minimal dimensions acceptable by the ResNet model is 32x32 so we need to resize our images.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/808e8b58451dea86d472aebb3297ae09/href">https://medium.com/media/808e8b58451dea86d472aebb3297ae09/href</a></iframe><pre><strong>Train_images.shape: (60000, 32, 32, 3), of float32<br>Test_images.shape: (10000, 32, 32, 3), of float32</strong></pre><p>Let’s train our model for 10 epochs now similar to our previous model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/793560aca53ceb07c3b06b58b36ef1d6/href">https://medium.com/media/793560aca53ceb07c3b06b58b36ef1d6/href</a></iframe><p>Do note that as in the previous model, we train on 90% of the training data and validate on 10% of the training data. Performance looks to be much better on the validation set. Let’s save our model and then check the performance on the test dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ad5753eb43ebcd463ec1cc8516f48e0/href">https://medium.com/media/8ad5753eb43ebcd463ec1cc8516f48e0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PWoAIZQuUi22PD3JgAUSSA.png" /></figure><p>Overall we get an <strong>f1-score</strong> of <strong>92%</strong> which is better than our first model! Most tutorials would end here but, you could say that ours will begin here, since the steps needed to enable model serving start now!</p><h3><strong>Saving models for TensorFlow Serving</strong></h3><p>We had briefly discussed this in the model serving methodology. To serve models using TensorFlow Serving, we need to save them into the <strong>SavedModel </strong>format. Thanks to the very nifty <strong>tf.saved_model.simple_save(…)</strong> function, we can do this within a few lines.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4fd03af27b651f43d995136364bed7d5/href">https://medium.com/media/4fd03af27b651f43d995136364bed7d5/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*QGJf4-ve-ruuvz2MQrEbbA.png" /></figure><p>Before we move on to setting up TensorFlow Serving, we can leverage TensorFlow’s <strong>SavedModel</strong> command line interface (CLI) tool, <strong>saved_model_cli</strong>, which is useful to quickly inspect the input and output specifications of our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*Is6CMnuvwxUQcfQ8vH3oaQ.png" /></figure><p>The above output shows details pertaining to our second model including input and output specifications.</p><h3><strong>Serving models with CPU Inference</strong></h3><p>In this section, we will show how to serve our saved models using TensorFlow serving leveraging our CPUs. We will do a local installation in our system, however I recommend you to use the <a href="https://www.tensorflow.org/tfx/serving/docker"><em>Docker installation based setup</em></a> for TF Serving which is easier to use and maintain since you only have to pull in the container using the following command and you don’t need to setup any configurations or dependencies.</p><pre><strong>docker pull tensorflow/serving</strong></pre><p>However, to show the different options out here, we will also show how you can setup TF Serving locally.</p><h4>Install TensorFlow Serving</h4><p>Here we showcase the necessary steps to install TF Serving locally. Do note this only consists of the CPU version. To start with we add the source to obtain TF Serving.</p><pre><strong>!echo &quot;deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal&quot; \<br>| sudo tee /etc/apt/sources.list.d/tensorflow-serving.list &amp;&amp; \<br>curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg \<br>| sudo apt-key add -</strong></pre><p>Next, you can remove the existing version of TF Serving if present using the following command.</p><pre><strong>!sudo apt-get -y remove tensorflow-model-server</strong></pre><pre>Reading package lists... Done<br>Building dependency tree       <br>Reading state information... Done<br>The following packages will be REMOVED:<br>  tensorflow-model-server<br>0 upgraded, 0 newly installed, 1 to remove and 39 not upgraded.<br>After this operation, 0 B of additional disk space will be used.<br>(Reading database ... 124893 files and directories currently installed.)<br>Removing tensorflow-model-server (1.13.0) ...</pre><p>Next, we install TF Serving using the following command</p><pre><strong>!sudo apt-get update &amp;&amp; sudo apt-get install tensorflow-model-server</strong></pre><pre>Hit:1 <a href="http://packages.cloud.google.com/apt">http://packages.cloud.google.com/apt</a> cloud-sdk-stretch InRelease<br>Get:2 <a href="http://security.debian.org/">http://security.debian.org</a> stretch/updates InRelease          <br>...<br>...<br>Preparing to unpack .../tensorflow-model-server_1.14.0_all.deb<br>Unpacking tensorflow-model-server (1.14.0) ...<br>Setting up tensorflow-model-server (1.14.0) ...</pre><h4>Start Serving Models with TensorFlow Serving</h4><p>We are now ready to start serving models with TF Serving. We need to setup a model serving config file if we want to customize the number of models we want to serve and other advanced configurations which you can read about <a href="https://www.tensorflow.org/tfx/serving/serving_config"><em>here</em></a>. Our configuration file is simple and depicted in the following snippet which is stored in <strong>models.conf</strong>. Do note that we want to serve both our models, hence the config file makes things easier.</p><pre><strong>model_config_list: {<br>  config: {<br>    name:  &quot;fashion_model_serving&quot;,<br>    base_path:  &quot;/home/jupyter/tensorflow_serving/tf_saved_models&quot;,<br>    model_platform: &quot;tensorflow&quot;,<br>    model_version_policy {<br>      specific {<br>        versions: 1<br>        versions: 2<br>      }<br>    }<br>  }<br>}</strong></pre><p>You can run the following code directly in Jupyter Notebooks to start the server as a background process or you can run it from the terminal also.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21b570c8f0d4b89a23f84e7e8655d5a5/href">https://medium.com/media/21b570c8f0d4b89a23f84e7e8655d5a5/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GUA230bkdVqgSJVDpAhWYQ.png" /></figure><p>You can see based on the server log messages that our server has successfully loaded both the models and they are ready for serving.</p><h4>Serving Model Inference Requests</h4><p>Now that our models are ready for serving, let’s first start with serving model inference requests using our first model, the basic CNN.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e175ee076da5cab9bcfa2e29438be94/href">https://medium.com/media/7e175ee076da5cab9bcfa2e29438be94/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*hdt6Gs2epGPhZg1xegQeIA.png" /></figure><p>Looks like TF Serving is able to serve model inference requests properly! Do note that we have to pre-process our image and create the proper payload before sending the request to the server.</p><p>Let’s take a look at serving model inference requests for the same sample of images using our second model, the ResNet-50 CNN.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cc7c0cc9b342abc3ef4f33d29d51c3e/href">https://medium.com/media/2cc7c0cc9b342abc3ef4f33d29d51c3e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*Mr0SQvgvESm0TnoejjZ0uw.png" /></figure><p>Looks like our model is serving requests and the predictions are better than our previous model!</p><h4>Model Warmups</h4><p>While productionizing and serving models, an important thing to remember is that the TensorFlow runtime has components that are lazily initialized, which can cause high latency for the first request(s) sent to a model after it is loaded. This latency can be several orders of magnitude higher than that of a single inference request. Hence it is good to warmup the models by sending a few sample records as an initial request after loading the model. You can do this at model load time by checking out <a href="https://www.tensorflow.org/tfx/serving/saved_model_warmup">the following documentation</a>.</p><p>Here, we will follow a simple approach of sending a sample request to each of our models to warm it up after it is loaded. For this, we save some sample data in a file which we can load and use later for warming up our models as showcased in the following code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d84bd498cca4fd6010395d784161bd04/href">https://medium.com/media/d84bd498cca4fd6010395d784161bd04/href</a></iframe><pre><strong>Model 1 warmup complete<br>Model 2 warmup complete</strong></pre><h4>Benchmarking Model Serving Requests</h4><p>Let’s take a look at how long it takes to serve model inference request for a bunch of images. For this, we will consider 10000 images from our test dataset. Do note we are sending only a single request and seeing the inference time of the whole batch. We will look at multiple requests in the next section. Let’s take a look at how our first model performs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14138e7b2f3e69d1ec9f44076c0f43b6/href">https://medium.com/media/14138e7b2f3e69d1ec9f44076c0f43b6/href</a></iframe><pre><strong>CPU times: user 7.64 s, sys: 612 ms, total: 8.26 s<br>Wall time: 9.76 s</strong></pre><pre><strong>Out[116]: 10000</strong></pre><p>We were able to serve the request of inference for 10000 images in around 9.8 seconds which is pretty good considering the model performed the inference using a CPU. Let’s check out the performance of our second model now.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9256d812687bc66d7a68acaf8ea8f768/href">https://medium.com/media/9256d812687bc66d7a68acaf8ea8f768/href</a></iframe><pre><strong>CPU times: user 21.6 s, sys: 1.38 s, total: 23 s<br>Wall time: 36.4 s</strong></pre><pre><strong>Out[122]: 10000</strong></pre><p>Time to serve the request considering CPU inference was 36 seconds which is not too bad!</p><h3>Serving models with GPU Inference</h3><p>In this section, we will show how to serve our saved models using TensorFlow serving leveraging our GPU. The idea is, if you have a GPU, use it! We will leverage Docker to setup our TensorFlow Serving system.</p><h4>Pulling TF Serving GPU image</h4><p>Assuming you have docker installed in your system or on the cloud. You can using the following code to pull in the latest version of TF Serving on GPUs</p><pre><strong>!docker pull tensorflow/serving:latest-gpu</strong></pre><pre>latest-gpu: Pulling from tensorflow/serving<br>Digest: sha256:fd54edb56a7b<br>Status: Image is up to date for tensorflow/serving:latest-gpu<br>docker.io/tensorflow/serving:latest-gpu</pre><p>You can check if the image is present in your system using the following command</p><pre><strong>!docker images</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*yfT9kQpUmyYn77m9eXH9TQ.png" /></figure><h4>Start Serving Models with Docker TensorFlow Serving GPU</h4><p>We are now ready to start serving models with TF Serving. We will do that by running the docker image we just downloaded. You can do it directly from Jupyter Notebooks also using the following code. In practice it is better to run it from the terminal.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af6db689861d9ec9af1d59a2d56885c6/href">https://medium.com/media/af6db689861d9ec9af1d59a2d56885c6/href</a></iframe><p>You can then use the following command in Docker to check if the container is up and running.</p><pre><strong>!docker ps -all</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*Wxwn1b7iNRUAaJuKiM3OuA.png" /></figure><p>Finally, you can check the logs in Docker to verify everything is working perfectly.</p><pre><strong>!docker logs 7d4b091ccefa | tail -n 15</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*sJel5ip2F0s_nr3o-7sPqQ.png" /></figure><p>This verifies the fact that TF Serving will be using the GPU on our system for inference!</p><h4>Model Warmup</h4><p>We can leverage our previously implemented code to warmup our models. Here we will focus more on our complex 2nd CNN model so we warmup Model 2.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f747b01813c7cf66d0c8fb4606f29c7d/href">https://medium.com/media/f747b01813c7cf66d0c8fb4606f29c7d/href</a></iframe><pre><strong>Model 2 warmup complete</strong></pre><h4>Benchmarking Model Serving Requests</h4><p>Let’s take all our 10000 test images and send a single request to check model serving time for inference using our GPU. Do note that we only focus on our 2nd model here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8268155211366c9db25fe5618b7a10aa/href">https://medium.com/media/8268155211366c9db25fe5618b7a10aa/href</a></iframe><pre><strong>CPU times: user 23.5 s, sys: 1.87 s, total: 25.3 s<br>Wall time: 31.3 s</strong></pre><pre><strong>Out[155]: 10000</strong></pre><p>Total inference time is a bit over 30 seconds for 10000 images, pretty decent! Let’s now perform a real benchmarking test. Consider 10000 separate requests for inference of a single apparel image to be classified each time. How much time would it take for TF Serving to serve these 10000 requests?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa29c786e59ceab56a128f6fd5a40e47/href">https://medium.com/media/aa29c786e59ceab56a128f6fd5a40e47/href</a></iframe><pre><strong>100%|██████████| 10000/10000 [01:55&lt;00:00, 86.56it/s]</strong></pre><pre><strong>CPU times: user 39.1 s, sys: 2.2 s, total: 41.3 s<br>Wall time: 1min 55s</strong></pre><pre><strong>Out[157]: 10000</strong></pre><p>It takes a total of <strong>115 seconds</strong> to serve <strong>10000 requests</strong>. Which means TF Serving serves <strong>each request</strong> in roughly <strong>11.5 milliseconds</strong>. Pretty good!</p><p>Let’s try an interesting comparison now. We will use the regular <strong>model.predict(…)</strong> API call from <strong>tf.keras</strong> to see how long it takes to serve 10000 requests.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3da5b0a13aaf16b86121e9c52b721b4/href">https://medium.com/media/e3da5b0a13aaf16b86121e9c52b721b4/href</a></iframe><pre><strong>100%|██████████| 10000/10000 [03:04&lt;00:00, 54.12it/s]</strong></pre><pre><strong>CPU times: user 3min 8s, sys: 17.2 s, total: 3min 25s<br>Wall time: 3min 4s</strong></pre><pre><strong>Out[159]: 10000</strong></pre><p>It takes a total of 184 seconds to serve 10000 requests. Which means using the native model prediction API, we are able to serve <strong>each request</strong> in roughly <strong>18.4 milliseconds</strong>.</p><blockquote>This showcases the need and importance of leveraging TF Serving especially when productionizing models!</blockquote><h3><strong>Bonus: Building an Apparel Classifier API with Flask &amp; TensorFlow Serving</strong></h3><p>While TF Serving is extremely useful and provides us with a high-performance system to serve inference requests. Considering an end-to-end perspective, you might have noticed that model serving is not just dumping some data as requests to the server. We need to access the image data, pre-process it and then send it in an appropriate format to TF Serving. Also once we get back the response, we need to access the class probabilities, get the class with the maximum probability and then get the corresponding apparel class label.</p><p>The best way to put all these steps together is to leverage a robust framework like Flask to build a web service / API on top of TF Serving to accept images from the real-world, perform necessary pre-processing, call TF Serving, post-process the response and then send the final JSON response to the end-user. Do note that we can even dockerize and deploy the Flask API on Kubernetes or use a WSGI server like <a href="https://gunicorn.org/">Gunicorn</a> to scale and improve performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5voGRt_Wk_JCtExZY0_P9g.png" /></figure><h4>Create API with Flask</h4><p>We will start by creating our own apparel API leveraging Flask, you will find the code files in <a href="https://github.com/dipanjanS/data_science_for_all/tree/master/gde_tf_serve_vision/tensorflow_serving"><strong><em>my GitHub repository</em></strong></a>, however we will also cover the code here for the sake of completeness and ease of understanding.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1223836264de908bb8a4268849298a1/href">https://medium.com/media/e1223836264de908bb8a4268849298a1/href</a></iframe><p>We store this file as <strong>app.py</strong> in our server which forms the base of our API.</p><h4>Start Docker Container for TF Serving</h4><p>Up next, check and restart the docker container for TF Serving if it’s not already up and running.</p><pre><strong>!docker start 7d4b091ccefa<br>!docker ps -all</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*i9FFvA7hcbo9XUWkYx-5uQ.png" /></figure><h4>Start our Apparel Classifier Web Service</h4><p>Now, we need to start our web service. In production, it is recommended you do NOT use the default web server provided by Flask but a better production-ready WSGI server, just like Gunicorn. We start our web service using the following command from the terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*m_cdBBSFmF1mHc0bJsLdzQ.png" /></figure><p>We leverage multiple workers to serve more requests as needed. Let’s now check if our API is live using the liveness test endpoint.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ca14a52d2084a6a73b91e10c1123313/href">https://medium.com/media/7ca14a52d2084a6a73b91e10c1123313/href</a></iframe><pre><strong>(200, &#39;API Live!&#39;)</strong></pre><h4>Serve Sample Apparel Classification with Web Service</h4><p>Let’s now take a sample real-world image and try to use our web service for performing classification. The image is depicted with the following code.</p><pre><strong>plt.imshow(cv2.cvtColor(cv2.imread(&#39;sneaker.jpg&#39;),    <br>                        cv2.COLOR_BGR2RGB))</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*40zEMCzhCyc1vmWpoNTSdg.png" /></figure><p>So this is clearly the image of a Sneaker. Let’s leverage our API to serve the model prediction. Do remember we are encoding any input image to the Base64 format and then decoding it and pre-processing the same in the server side before performing model inference.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c20b7eecfd6c9a42c12752692c5819e9/href">https://medium.com/media/c20b7eecfd6c9a42c12752692c5819e9/href</a></iframe><pre><strong>{&#39;apparel_type&#39;: &#39;Sneaker&#39;}</strong></pre><p>We finally get the right apparel category in the form of a JSON response. Things are working out exactly the way we want!</p><h4>Benchmark our Web Service</h4><p>Considering web server latency, image processing, model inference and serving, let’s check out how much time it takes to process 10000 requests now.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0979e37f5c86e2c3bf285206da660bc5/href">https://medium.com/media/0979e37f5c86e2c3bf285206da660bc5/href</a></iframe><pre><strong>100%|██████████| 10000/10000 [05:26&lt;00:00, 30.66it/s]</strong></pre><pre><strong>CPU times: user 1min, sys: 3.17 s, total: 1min 4s<br>Wall time: 5min 26s</strong></pre><pre><strong>Out [58]: 10000<br>Inference time per image: 32.599999999999994 ms</strong></pre><p>We are able to serve each request in 32.6 ms which is not too bad! Can you improve on this even more? Feel free to let us know your thoughts!</p><h3>Conclusion</h3><p>I hope this slightly lengthy, yet comprehensive article gives you an idea about how building models and prototyping on notebooks is very different from actually productionizing models in the real-world. Always think of the complete end-to-end picture even when training your models. This helps you envision and implement your own inference system much faster once your models are trained. Hope this also gives you an idea that while TensorFlow Serving might seem complex, once you get started with it, you can train, save and serve your models pretty easily and it works well with frameworks like Docker, Flask, Kubernetes and so on. Feel free to let us know your own tips and tricks and methods to deploy deep learning models in production.</p><p>All the code used in this article is available on <a href="https://github.com/dipanjanS"><strong><em>my GitHub</em></strong></a> in <a href="https://github.com/dipanjanS/data_science_for_all/tree/master/gde_tf_serve_vision/tensorflow_serving"><strong><em>this repository</em></strong></a><strong><em> </em></strong>as Jupyter notebooks and necessary files\assets.</p><p>I am a <a href="https://google-developers.appspot.com/community/experts/directory/profile/profile-dipanjan_sarkar"><em>Google Developer Expert in Machine Learning</em></a> and I look forward to sharing more interesting aspects of machine learning and deep learning over time. Feel free to check out my <a href="https://medium.com/@dipanzan.sarkar"><strong>Medium</strong></a> and <a href="https://www.linkedin.com/in/dipanzan"><strong>LinkedIn</strong></a> for updates on interesting content!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acd716f6450a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/building-robust-production-ready-deep-learning-vision-models-in-minutes-acd716f6450a">Building Robust Production-Ready Deep Learning Vision Models in Minutes</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Powered Search for Extra-terrestrial Intelligence — Deep Learning Signal Classifiers]]></title>
            <link>https://medium.com/data-science/ai-powered-search-for-extra-terrestrial-intelligence-signal-classification-with-deep-learning-6c09de8fd57c?source=rss-6278d12b0682------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c09de8fd57c</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Dipanjan (DJ) Sarkar]]></dc:creator>
            <pubDate>Wed, 21 Aug 2019 16:29:08 GMT</pubDate>
            <atom:updated>2019-08-21T17:36:57.416Z</atom:updated>
            <content:encoded><![CDATA[<h4>AI FOR SOCIAL GOOD SERIES — PART 2.2</h4><h3>AI Powered Search for Extra-terrestrial Intelligence — Deep Learning Signal Classifiers</h3><h4>Classifying Radio-Telescope Signals from SETI with Deep Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xHKwf4Vl4GXuh1HIW0Wnjg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@wizwow?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Donald Giannatti</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Welcome (or welcome back!) to the AI for social good series! In the second part, of this two-part series of articles, we will look at how Artificial intelligence (AI) coupled with the power of open-source tools and techniques like deep learning can help us further the quest for finding extra-terrestrial intelligence!</p><p>In <a href="https://towardsdatascience.com/ai-powered-search-for-extra-terrestrial-intelligence-analyzing-radio-telescopic-data-c9e46741041"><strong><em>the first part</em></strong></a><strong><em> </em></strong>of this two-part series, we formulated our key objective and motivation behind doing this project. Briefly, we were looking at different radio-telescope signals simulated from <a href="https://www.seti.org/"><strong><em>SETI (Search for Extra-terrestrial Intelligence) Institute</em></strong></a><strong><em> </em></strong>data. We leveraged techniques to process, analyze and visualize radio signals as spectrograms, which are basically visual representations of the raw signal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*JCcoBc0pwYuX6UHt.png" /><figcaption>Signal observed at the Allen Telescope Array from the Cassini satellite while orbiting Saturn on September 3, 2014 (Source: <a href="https://medium.com/@gadamc/using-artificial-intelligence-to-search-for-extraterrestrial-intelligence-ec19169e01af">https://medium.com/@gadamc/using-artificial-intelligence-to-search-for-extraterrestrial-intelligence-ec19169e01af</a>)</figcaption></figure><p>The key focus for us in this article will be to try and build a robust radio signal classifier using deep learning for a total of seven different types of signals!</p><h3>Loading SETI Signal Data</h3><p>Like we discussed in the previous article, the simulated SETI radio-signal dataset isavailable in <a href="https://www.kaggle.com/tentotheminus9/seti-data/home"><strong>Kaggle</strong></a>. Remember, the processed dataset is available in the <strong>primary_small</strong> folder. After unzipping its contents this is how the directory structure looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*OxHxn-VzZklSAcxh8EVL6Q.png" /></figure><p>We have a total of 7 different signal classes to classify, each class has a total of 800 samples for training, 100 for validation and test respectively. Considering noise has been added to the simulated signal data coupled with the fact that we have less number of samples per class makes this a tough problem to solve! Before we get started, let’s load the necessary dependencies we will be using for building models. We will leverage the <strong>tf.keras</strong> API from TensorFlow here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38213c29438a418ceb3039c0e65fc7b6/href">https://medium.com/media/38213c29438a418ceb3039c0e65fc7b6/href</a></iframe><pre><strong>./data/train<br>./data/valid<br>./data/test</strong></pre><h3>Visualize Sample SETI Signals</h3><p>Just to recap from our last article and take a peek at the different types of signals we are dealing with, we can visualize their spectrograms using the following code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd9272910317607210d3c199a67a5562/href">https://medium.com/media/fd9272910317607210d3c199a67a5562/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*F9HPFJKz0jSUJaCQICiTxQ.png" /></figure><p>Things look to be in order with regard to the different signal samples we are dealing with!</p><h3>Data Generators and Image Augmentation</h3><p>Since we have a low number of training samples per class, one strategy to get more data is to generate new data using image augmentation. The idea behind image augmentation is exactly as the name sounds. We load in existing images from our training dataset and apply some image transformation operations to them, such as rotation, shearing, translation, zooming, flipping and so on, to produce new, altered versions of existing images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*1eyXxnBCHPU5GiZP.png" /></figure><p>Due to these random transformations, we don’t get the same images each time. We need to be careful with augmentation operations though based on the problem we are solving so that we don’t end up distorting the source images too much.</p><p>We will be applying some basic transformations to all our training data but keep our validation and test dataset as it is except just scaling the data. Let’s build our data generators now.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4a22f630d06cec834c02659ed839885/href">https://medium.com/media/d4a22f630d06cec834c02659ed839885/href</a></iframe><p>We can now build a sample data generator just to get an idea of how the data generator coupled with image augmentation works.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02381c6e7bab92fa9e0f99ce64a455e9/href">https://medium.com/media/02381c6e7bab92fa9e0f99ce64a455e9/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*UllYGM0zLsERk5eDl90jQw.png" /></figure><p>The labels are one-hot encoded given that we have a total of 7 classes so each label is a one-hot encoded vector of size 7.</p><h3>Deep Transfer Learning with CNNs</h3><p>The idea of transfer learning is not a new concept and is something which is very useful when we are working with less data. Given that we have a pre-trained model, which was trained previously on a lot of data, we can use this model on a new problem with less data and should ideally get a model which performs better and converges faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*2g7__HctYE18eKSk.png" /></figure><p>There are a wide variety of pre-trained CNN models which have been trained on the ImageNet dataset having a lot of images belonging to a total of 1000 classes. The idea is that these models should act as effective feature extractors for images and can also be fine-tuned to the specific task we perform. Pre-trained models can be frozen completely where we don’t change the layer weights at all when we train on the new dataset or we can fine-tune (partially or completely) the model weights as we train on the new dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*zyLkMjftbFW7Z546.png" /></figure><p><strong><em>In our scenario, we will try out partial and complete fine-tuning of our pre-trained models</em></strong>.</p><h3>Pre-trained CNN Models</h3><p>One of the fundamental requirements for transfer learning is the presence of models that perform well on source tasks. Luckily, the deep learning world believes in sharing. Many of the state-of-the art deep learning architectures have been openly shared by their respective teams. Pre-trained models are usually shared in the form of the millions of parameters/weights the model achieved while being trained to a stable state. Pre-trained models are available for everyone to use through different means.</p><p>Pre-trained models are available in TensorFlow which you can access easily using its API. We will be showing how to do that in this article. You can also access pre-trained models from the web since most of them have been open-sourced.</p><p><strong><em>For computer vision</em></strong>, you can leverage some popular models including,</p><ul><li><a href="https://www.kaggle.com/keras/vgg16/home">VGG-16</a></li><li><a href="https://www.kaggle.com/keras/vgg19/home">VGG-19</a></li><li><a href="https://arxiv.org/abs/1512.00567">Inception V3</a></li><li><a href="https://arxiv.org/abs/1610.02357">XCeption</a></li><li><a href="https://www.kaggle.com/keras/resnet50/home">ResNet-50</a></li></ul><p>The models we will be using in our article will be VGG-19 and ResNet-50.</p><h3>VGG-19 model</h3><p>The VGG-19 model is a 19-layer (convolution and fully connected) deep learning network built on the ImageNet database, which is built for the purpose of image recognition and classification. This model was built by Karen Simonyan and Andrew Zisserman and is mentioned in their paper titled <a href="https://arxiv.org/pdf/1409.1556.pdf"><em>‘Very Deep Convolutional Networks for Large-Scale Image Recognition’</em></a><em>. </em>I recommend all interested readers to go and read up on the excellent literature in this paper. The architecture of the VGG-19 model is depicted in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*wMgNnJpJHjDgLWb_.png" /></figure><h3>ResNet-50 model</h3><p>The ResNet-50 model is a 50-convolutional block (several layers in each block) deep learning network built on the ImageNet database. This model has over 175+ layers in total and is a very deep network. ResNet stands for Residual Networks. The following figure shows the typical architecture of ResNet-34.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XKfCkMhillH_7FgpqQnvdQ.png" /></figure><p>In general, deep convolutional neural networks have led to major breakthroughs in image classification accuracy. However, as we go deeper; the training of neural network becomes difficult. The reason for this often is because of the vanishing gradient problem . Basially as the gradient is back-propagated to the shallower layers (closer to the input), repeated tensor operations make the gradient really small. Hence, the accuracy starts saturating and then degrades also. Residual Learning tries to solve these problems with residual blocks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/1*QBTeyaDNIjR6HmJBQ8gldw.png" /></figure><p>Leveraging skip connections, we can allow the network to learn the identity function (depicted in the above figure), which allows the network to pass the input through the residual block without passing through the other weight layers. This helps in tackling the problems of vanishing gradients and also keeping a focus on the high-level features which sometimes get lost with multiple levels of max-pooling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*uqAup0yF8ohawYvr.png" /><figcaption>Source: <a href="https://towardsdatascience.com/understanding-and-coding-a-resnet-in-keras-446d7ff84d33">https://towardsdatascience.com/understanding-and-coding-a-resnet-in-keras-446d7ff84d33</a></figcaption></figure><p>The ResNet-50 model which we will be using, consists of 5 stages, each with a Convolution and Identity block. Each convolution block has 3 convolution layers and each identity block also has 3 convolution layers.</p><h3>Deep Transfer Learning with VGG-19</h3><p>The focus here will be to take the pre-trained VGG-19 model and then perform both partial and complete fine-tuning of all the layers in the network. We will add the regular dense and output layers in the model for our downstream classification task.</p><h4>Partial Fine-tuning</h4><p>We will start our model training by taking the VGG-19 model and fine-tune the last two blocks of the model. The first task here is the build the model architecture and also specify which blocks \ layers we want to fine-tune.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/723eb1e30d4d764f6f0ae3c13a27c885/href">https://medium.com/media/723eb1e30d4d764f6f0ae3c13a27c885/href</a></iframe><p>Now we will train the model for 100 epochs. I save the model after each epoch because I have a lot of space. I wouldn’t recommend to do this practically unless you have a lot of storage. You can always leverage a callback like <strong>ModelCheckpoint </strong>to focus on storing only the best model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2b730c4b8e1e86399aef5f64a136232/href">https://medium.com/media/d2b730c4b8e1e86399aef5f64a136232/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71c00fe46c7a1b918d333867949244f9/href">https://medium.com/media/71c00fe46c7a1b918d333867949244f9/href</a></iframe><p>We can view the overall model learning curves using the following code snippet.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce9893da4801e252414118e7caf33d76/href">https://medium.com/media/ce9893da4801e252414118e7caf33d76/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*Req-0C3SSE23zTref6NKeQ.png" /></figure><p>Looks decent but definitely a lot of fluctuations with regard to the loss and accuracy over time for the validation dataset.</p><h4>Complete Fine-tuning</h4><p>For our next training process, we will take the VGG-19 model and fine-tune all the blocks and add in our own dense and output layers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0ec85bf347f8d37c99cb840f63812a7/href">https://medium.com/media/b0ec85bf347f8d37c99cb840f63812a7/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07ded056f7243bb782dfc990b7a9df2a/href">https://medium.com/media/07ded056f7243bb782dfc990b7a9df2a/href</a></iframe><p>The learning curves for the training process are depicted in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*t3q1VBuO7gT0WB0G1hP0Fw.png" /></figure><p>Looks to be more stable as the epochs increase with regard to the validation accuracy and loss.</p><h3>Deep Transfer Learning with ResNet-50</h3><p>The focus in this section will be taking the pre-trained ResNet-50 model and then perform complete fine-tuning of all the layers in the network. We will add the regular dense and output layers as usual.</p><h4>Complete Fine-tuning</h4><p>For our training process, we will load the pre-trained ResNet-50 model and fine-tune the entire network for 500 epochs. Let’s start by building the model architecture.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af814eed4054c3a9ebeddf32285e7d7a/href">https://medium.com/media/af814eed4054c3a9ebeddf32285e7d7a/href</a></iframe><p>Let’s now train the model for a total of 500 epochs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15d3e653dda46453dfd1349ef748342c/href">https://medium.com/media/15d3e653dda46453dfd1349ef748342c/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cbab2882f8ff5e8beca19630643e001e/href">https://medium.com/media/cbab2882f8ff5e8beca19630643e001e/href</a></iframe><p>The learning curves for our trained model can be observed in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*dV_A1cz-Km6snJuaSQbtLA.png" /></figure><h3>Evaluating Model Performance on Test Data</h3><p>It is now time to put our trained models to the test. We will do so by making predictions on the test dataset and evaluating model performance based on relevant classification metrics for our multi-class classification problem.</p><h4>Load Test Dataset</h4><p>We start by loading out test dataset and labels leveraging our data generator we had built previously.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94184b365e0307e814d2b3aa5f8a3cf6/href">https://medium.com/media/94184b365e0307e814d2b3aa5f8a3cf6/href</a></iframe><pre><strong>Found 700 images belonging to 7 classes.</strong></pre><pre><strong>((700, 192, 192, 3), (700,))</strong></pre><h4>Build Model Performance Evaluation Function</h4><p>We will now build a basic classification model performance evaluation function, which we will use to test the performance of each of our three models.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c2ca8875b8a3909390f9bf478b3fce03/href">https://medium.com/media/c2ca8875b8a3909390f9bf478b3fce03/href</a></iframe><p>We are now ready to test our models’ performance on the test dataset</p><h4>Model 1 — Partial fine-tuned VGG-19</h4><p>Here we evaluate the performance of our partially fine-tuned VGG-19 model on the test dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8bf1a34c8e53ba9e76e8dbed9ac77338/href">https://medium.com/media/8bf1a34c8e53ba9e76e8dbed9ac77338/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7UpPP6OpYyWOEusSQK2aDw.png" /></figure><p>An overall accuracy \ f1-score of <strong>86%</strong> on the test dataset which is pretty decent!</p><h4>Model 2 — Complete fine-tuned VGG-19</h4><p>Here we evaluate the performance of our completely fine-tuned VGG-19 model on the test dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f7f7785b259d272f2591d658df588b1/href">https://medium.com/media/2f7f7785b259d272f2591d658df588b1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jQKdp1U_bRXE1ZvGY0vqOg.png" /></figure><p>An overall accuracy \ f1-score of <strong>85%</strong> on the test dataset which is slightly lesser than the partially fine-tuned model.</p><h4>Model 3 — Complete fine-tuned ResNet-50</h4><p>Here we evaluate the performance of our completely fine-tuned ResNet-50 model on the test dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8492136c861ea2414b87ef426c3c5336/href">https://medium.com/media/8492136c861ea2414b87ef426c3c5336/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*08zlKkhRGqz8IU4WIV26uw.png" /></figure><p>We get an overall accuracy \ f1-score of <strong>88% </strong>which is definitely the best model performance yet on the test dataset! Looks like the ResNet-50 model performed the best, given the fact that we did train it for 500 epochs.</p><h4>Best Model Predictions on Sample Test Data</h4><p>We can now use our best model to make predictions on sample radio-signal spectrograms.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b87243bd9c11e5b63f6b9b8ebef0e04e/href">https://medium.com/media/b87243bd9c11e5b63f6b9b8ebef0e04e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*iYGr8gMYrXJj9kwwZygUtA.png" /></figure><h3>Conclusion</h3><p>This brings us to the end of our two-part series on leveraging deep learning to further the search of extra-terrestrial intelligence. You saw how we converted radio-signal data into spectrograms and then leveraged the power of transfer learning to build classifiers which performed pretty well even will really low number of training data samples per class.</p><p>The code used in this article is available on <a href="https://github.com/dipanjanS"><strong>my GitHub</strong></a><strong> </strong>in this <a href="https://github.com/dipanjanS/data_science_for_all"><strong>repository</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c09de8fd57c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/ai-powered-search-for-extra-terrestrial-intelligence-signal-classification-with-deep-learning-6c09de8fd57c">AI Powered Search for Extra-terrestrial Intelligence — Deep Learning Signal Classifiers</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Powered Search for Extra-terrestrial Intelligence — Analyzing Radio Telescopic Data]]></title>
            <link>https://medium.com/data-science/ai-powered-search-for-extra-terrestrial-intelligence-analyzing-radio-telescopic-data-c9e46741041?source=rss-6278d12b0682------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9e46741041</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Dipanjan (DJ) Sarkar]]></dc:creator>
            <pubDate>Fri, 16 Aug 2019 12:58:56 GMT</pubDate>
            <atom:updated>2019-08-16T13:46:44.979Z</atom:updated>
            <content:encoded><![CDATA[<h4>AI for Social Good Series — Part 2.1</h4><h3>AI Powered Search for Extra-terrestrial Intelligence — Analyzing Radio Telescopic Data</h3><h4>Understanding Radio-Telescope Signal Data from SETI</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XYiiR_zgnZEdGvfK88hj3Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@gorkhe1980?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ambir Tolang</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>In this two-part series of articles, we will look at how Artificial intelligence (AI) coupled with the power of open-source tools and frameworks can be used to solve a very interesting problem in a non-conventional domain — the quest for finding extra-terrestrial intelligence!</p><p>Perhaps many of you are already familiar with the<a href="https://www.seti.org/"><strong><em> SETI (Search for Extra-terrestrial Intelligence) Institute</em></strong></a><strong><em>,</em></strong> which focuses on trying to find out the existence of extra-terrestrial intelligence out in the universe and as its mission suggests, <em>“exploring, understanding and explaining the origin and nature of life in the universe and the evolution of intelligence”</em>. Recently, I had come across an open dataset from SETI on Kaggle which talks about SETI focusing on several public initiatives tied around leveraging AI on their datasets in the form of competitions which had occurred in the past. While the competition is inactive now, a subset of this data is still available for analysis and modeling which is the main focus of our articles in this series.</p><p>We will be looking at analyzing data sourced by SETI as a part of their past initiative, ML4SETI — Machine Learning 4 the Search for Extra Terrestrial Intelligence. This data for this initiative is simulation data based on radio signals The goal here was for data scientists to find a robust signal classification algorithm for use in the mission to find E.T. radio communication. The key focus here being able to classify different types \ classes of signals accurately.</p><h3>Motivation &amp; Significance</h3><p>The SETI Institute is dedicatedly researching and working on methods to improve their search for extra-terrestrial intelligence. One of their key instruments accelerating this search is known as the Allen Telescope Array (ATA), which is situated at the Hat Creek Observatory, located in the Cascade Mountains just north of Lassen Peak, in California</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*byuia87qmrtrgthJtPsvZQ.jpeg" /><figcaption>The Allen Telescoping Array (<a href="https://www.seti.org/ata">https://www.seti.org/ata</a>)</figcaption></figure><p>The Allen Telescope Array helps in radio-signal based searches and is speeding up SETI targeted searching by a factor of at least 100. The motivation for applying AI or Deep Learning here can be maifold.</p><ul><li>Deep Learning Models optimized to distinguish between different signals may decrease search time</li><li>Efficiency and maybe even new and better ways to find out extra-terrestrial intelligence as they observe signals from star systems</li></ul><p>While the data we will be using here is simulation data based on radio signals, they are very much in line with the real data captured by radio-telescopic devices at SETI. We will take a brief look at this now.</p><h3>Understanding SETI Data</h3><p>The information presented here was based on insights which I gathered from researching on details about SETI data when they had this competition in collaboration with IBM a while back. Details of the challenge are presented <a href="https://github.com/setiQuest/ML4SETI"><em>here</em></a> for folks who might be interested and a lot of the background information about the setup and data are mentioned in <a href="https://medium.com/@gadamc/using-artificial-intelligence-to-search-for-extraterrestrial-intelligence-ec19169e01af"><em>this useful article</em></a>.</p><h4>Allen Telescope Array Architecture</h4><p>The Allen Telescope Array (ATA) consists of several relatively small dishes (antennas) extending over about 1 km. This provides a very high quality beam shape (the spot in the sky to which the telescope is most sensitive). The ATA is actually optimized to cover frequencies between 1000 MHz — 15000 MHz. The ATA combines the signals from different dishes, using a process called <em>“beamforming”. </em>Typically using this process, the ATA observes radio signals from very small windows of the sky about specific stellar systems. On a high level, the ATA has four main conceptual systems as explained in the <a href="https://www.seti.org/ata-technical-overview"><em>official page</em></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*2dL3Kl1r6rQfc2-e_qpxpg.png" /><figcaption>ATA architecture (<a href="https://www.seti.org/ata-technical-overview">https://www.seti.org/ata-technical-overview</a>)</figcaption></figure><ul><li>The antenna collects the radiation from space;</li><li>The signal path brings the radiation from the feed (which is located at the antenna focus) back to the user</li><li>The monitor and command systems allow the dishes to be accurately moved, and the signal path controlled</li><li>The site includes the overall antenna configuration, as well as other infrastructure.</li></ul><p>The ATA will permit remote users to access and use the instrument via a secure Internet connection.</p><h4>Radio-Telescope Time Series Signals</h4><p>From a usage perspective, three separate beams may be observed simultaneously and are used together to make decisions about the likelihood of observing intelligent signals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*fJKzNnR79JevnDHh.png" /></figure><p>The software that controls the data acquisition system, analyzes this radio signal time-series data in real-time and writes data out to disk. It is called SonATA (<strong>S</strong>ETI <strong>on</strong> the <strong>ATA</strong>).</p><p>To find relevant signals, the SonATA software calculates the signal power as a function of both frequency and time, and focuses on signals with power greater than the average noise power that persist for more than a few seconds. The best way of representation of the power as a function of frequency and time is through <a href="https://en.wikipedia.org/wiki/Spectrogram">spectrograms</a>, or “waterfall plots”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*lFAl2qkkuiyE6dzluqxZHQ.png" /></figure><p>A sample spectrogram of real radio-signal data from the ATA is depicted in the following figure. This is a classic example of what is called as a <em>“narrowband” </em>signal, which is what SonATA primarily searches for in the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*NJ9ShcUVS5T-UcH6.png" /><figcaption>Signal observed at the Allen Telescope Array from the Cassini satellite while orbiting Saturn on September 3, 2014 (Source: <a href="https://medium.com/@gadamc/using-artificial-intelligence-to-search-for-extraterrestrial-intelligence-ec19169e01af">https://medium.com/@gadamc/using-artificial-intelligence-to-search-for-extraterrestrial-intelligence-ec19169e01af</a>)</figcaption></figure><p>The power of the signal is represented on a black &amp; white scale. We have time on the y-axis and frequency (Hz) on the x-axis. You can clearly see a signal starting at approximately 8429245830 Hz and drifting up to 8429245940 Hz over the 175 second observation.</p><h4>The Need for Analyzing Radio-Signal Data</h4><p>The reason SETI is capturing this data and searching for specific patterns in the signals is because this is the kind of signal we use to communicate with our satellites in space. Hence there is a hope that a seemingly advanced extra-terrestrial (E.T.) civilization might transmit a signal to us if they were trying to get our attention!</p><h3>Getting Simulated SETI Data</h3><p>Now that we have enough background information on SETI and its data. Let’s try and get the dataset which we will be using to train our deep learning models. The competition ML4SETI is long over unfortunately but fortunately for us, a subset of the data is still available. Feel free to check out the ML4SETI <a href="https://github.com/setiQuest/ML4SETI/blob/master/GettingStarted.md"><strong><em>‘Getting Started’</em></strong></a> page to get an idea of how to get the dataset. If you feel like diving right into the data without reading the background information, just head over to <a href="https://nbviewer.jupyter.org/github/setiQuest/ML4SETI/blob/master/tutorials/Step_1_Get_Data.ipynb#Direct-Data-URLs-if-you-are-working-from-outside-of-IBM-Data-Science-Experience"><strong><em>this Jupyter Notebook</em></strong></a>! An easier way to get this dataset is from <a href="https://www.kaggle.com/tentotheminus9/seti-data/home"><strong>Kaggle</strong></a><strong>.</strong></p><p>Since there were challenges of working with real data, a set of simulated signals were built to approximate this real signal data. Typically there are a number of signal classes that SETI Institute researchers often observe. For the purpose of this dataset there were a total of seven different classes.</p><ul><li><strong>brightpixel</strong></li><li><strong>narrowband</strong></li><li><strong>narrowbanddrd</strong></li><li><strong>noise</strong></li><li><strong>squarepulsednarrowband</strong></li><li><strong>squiggle</strong></li><li><strong>squigglesquarepulsednarrowband</strong></li></ul><p>The class names are basically descriptive how they look like in spectrograms.</p><h3>Main Objective</h3><p>It is easy to define our main objective now that we have all our background information and our data. Give a total of seven different radio-signal classes including noise and a total of 1000 samples per class, leverage deep learning models to build an accurate classifier. While we will be building the deep learning models in the second part of this series, in the remaining sections of this article, we will be diving deeper into the dataset to gain a better understanding of what we are dealing with.</p><h3>Analyzing Simulated SETI Data</h3><p>We used the small version of the SETI data <strong>primary_small_v3</strong> in its raw format just to show how to load and process the data. Remember to process the signal data you will be needing the open-source <strong>ibmseti</strong> package. You can install it using <strong>pip install ibmseti</strong>. Let’s load up the raw dataset now before some basic processing and analysis.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c1fbf39be744a698f1a2005722969f7/href">https://medium.com/media/7c1fbf39be744a698f1a2005722969f7/href</a></iframe><pre><strong>7000</strong></pre><p>This tells us we have a total of 7000 signal samples in our dataset. Let’s try processing and visualizing one of these signals.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55ee407c63fa9c823ae6c99f27419a25/href">https://medium.com/media/55ee407c63fa9c823ae6c99f27419a25/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*-EjYb1zBYA5B0OXVXiuyvg.png" /></figure><p>Thus based on the visuals above, we have successfully processed a narrowband signal into a spectrogram before visualizing it. Let’s look at the total number of signal samples per-class.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39a1d5b2fd260e5f57eaa1d8531442c9/href">https://medium.com/media/39a1d5b2fd260e5f57eaa1d8531442c9/href</a></iframe><pre><strong>brightpixel                       1000<br>narrowbanddrd                     1000<br>squiggle                          1000<br>narrowband                        1000<br>squigglesquarepulsednarrowband    1000<br>noise                             1000<br>squarepulsednarrowband            1000<br>Name: signal_classification, dtype: int64</strong></pre><p>Like we mentioned before, we have a total of 1000 samples per class, which is frankly not a lot! More on that later. Let’s process and visualize some sample signals with spectrograms.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89eab9315b9ab4751a22c1c92c253e16/href">https://medium.com/media/89eab9315b9ab4751a22c1c92c253e16/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*t4orL52Zr7KGvuc7yZ3x8g.png" /></figure><p>This kind of gives us an idea of how the different signals look like. You can see that these signals are not always very distinguishable and hence makes this a tougher classification challenge.</p><h3>Loading Processed SETI Data</h3><p>Now, for the remainder of this article and the next one in this series, you can manually leverage the <strong>ibmseti</strong> package and process each signal into a spectrogram or just download and use the already processed spectrogram files from the dataset available in <a href="https://www.kaggle.com/tentotheminus9/seti-data/home"><strong>Kaggle</strong></a>. Remember, the processed dataset is available in the <strong>primary_small</strong> folder. After unzipping its contents this is how the directory structure looks like.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75fcc887ac4f6a488cca86269901c395/href">https://medium.com/media/75fcc887ac4f6a488cca86269901c395/href</a></iframe><p>We have a total of 7 classes and 800 training samples, 100 validation samples and 100 test samples per class. Let’s look at some sample processed spectrogram signal files now.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13380e5af4d76f8093adb568d55cfe2d/href">https://medium.com/media/13380e5af4d76f8093adb568d55cfe2d/href</a></iframe><h4>Loading and Visualizing Spectrograms with Keras</h4><p>The Keras framework has some excellent utilities to work with image files containing these spectrograms. Following snippet showcases this on a sample spectrogram.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8e2e0ec5f7f2f02961ab8bdd0cbd259/href">https://medium.com/media/a8e2e0ec5f7f2f02961ab8bdd0cbd259/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*eKonflqvrFmZp5zpWJtoPQ.png" /></figure><h4>Loading and Visualizing Spectrograms with OpenCV</h4><p>There are a lot of OpenCV fans out there who love using the framework for all their image processing needs. You can use that too for working with spectrograms. Do remember that OpenCV loads these images by default in the BGR format and not RGB. We showcase the same in the following code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd17415e4758b886cb80094d022dfb1b/href">https://medium.com/media/cd17415e4758b886cb80094d022dfb1b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*FyRFF3L3tqfYMSgmYeoE6A.png" /></figure><h3>Visualizing Processed SETI Sample Data</h3><p>Let’s now visualize the seven different sample signals which we selected from the processed dataset we will be using later for classification.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e911133fadc4cbd6ac849b7b5ecc225/href">https://medium.com/media/1e911133fadc4cbd6ac849b7b5ecc225/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*KNrHVFDjKpSC2IYSKh3pDg.png" /></figure><p>This shows us the different types of radio-signals in our dataset which are processed and ready to be used.</p><h3>Understanding the Radio-Signal Simulation Data</h3><p>All simulation signals here were generated by a sum of the signal and a noise background. The following background information about signals, has been collated from <a href="https://medium.com/@gadamc/using-artificial-intelligence-to-search-for-extraterrestrial-intelligence-ec19169e01af"><em>this wonderful article</em></a> to understand about each signal in more detail</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*_RhZSoQmLet3jZIqvNKA1g.png" /></figure><h3>Deep Learning on SETI Data</h3><p>Given that we have spectrogram images for our seven signal classes, we can leverage vision-based deep learning models to build robust image classifiers. In the next article in this series, we will be leveraging convolutional neural networks to see how we can build accurate classifiers to distinguish between these signal classes!</p><h3>Brief on Convolutional Neural Networks</h3><p>The most popular deep learning models leveraged for computer vision problems are convolutional neural networks (CNNs)!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rO65zGl2k7xVBRhJZggQSA.gif" /><figcaption>Source: becominghuman.ai</figcaption></figure><p>CNNs typically consist of multiple convolution and pooling layers which help the deep learning model in automatically extracting relevant features from visual data like images. Due to this multi-layered architecture, CNNs learn a robust hierarchy of features, which are spatial, rotation, and translation invariant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NKL76WYQwH5LuqyaQTjBNw.png" /></figure><p>The key operations in a CNN model are depicted in the figure above. Any image can be represented as a tensor of pixel values. The convolution layers help in extracting features from this image (forms feature maps). Shallower layers (closer to the input data) in the network learn very generic features like edges, corners and so on. Deeper layers in the network (closer to the output layer) learn very specific features pertaining to the input image. The following graphic helps summarize the key aspects of any CNN model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nCQeDMjKoTGst1RiCDo9Fw.png" /></figure><p>We will be leveraging the power of transfer learning, where we use pre-trained deep learning models (which have already been trained on a lot of data). More on that in the next article!</p><h3>Next Steps</h3><p>In the next article we will build deep learning classifiers based on CNNs to classify SETI signals accurately. Stay tuned!</p><p>The code used in this article is available on <a href="https://github.com/dipanjanS"><strong>my GitHub</strong></a><strong> </strong>in this <a href="https://github.com/dipanjanS/data_science_for_all"><strong>repository</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9e46741041" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/ai-powered-search-for-extra-terrestrial-intelligence-analyzing-radio-telescopic-data-c9e46741041">AI Powered Search for Extra-terrestrial Intelligence — Analyzing Radio Telescopic Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interpreting Deep Learning Models for Computer Vision]]></title>
            <link>https://medium.com/google-developer-experts/interpreting-deep-learning-models-for-computer-vision-f95683e23c1d?source=rss-6278d12b0682------2</link>
            <guid isPermaLink="false">https://medium.com/p/f95683e23c1d</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Dipanjan (DJ) Sarkar]]></dc:creator>
            <pubDate>Thu, 15 Aug 2019 11:22:12 GMT</pubDate>
            <atom:updated>2019-08-15T11:45:46.936Z</atom:updated>
            <content:encoded><![CDATA[<h4>Explainable Artificial Intelligence</h4><h4>Interpreting Convolutional Neural Network Models built with TensorFlow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kXSdiGvBk0Dq-2Wm5HUaww.png" /><figcaption>What does a deep learning model really see?</figcaption></figure><h3>Introduction</h3><p>Artificial Intelligence (AI) is no longer a field restricted only to research papers and academia. Businesses and organizations across diverse domains in the industry are building large-scale applications powered by AI. The questions to think about here would be, <em>“Do we trust decisions made by AI models?” </em>and<em> “How does a machine learning or deep learning model make its decisions?”. </em>Interpreting machine learning or deep learning models has always been a task often overlooked in the entire data science lifecycle since data scientists or machine learning engineers would be more involved with actually pushing things out to production or getting a model up and running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*rzDyIYmGwSu60U-9w9ztBA.png" /></figure><p>However, unless we are building a machine learning model for fun, accuracy is not the only thing which counts! Business stakeholders and consumers will often ask the tough questions of fairness, accountability and transparency of any machine learning model being used to solve real-world problems!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*Fb4sZSoSxxtQ14yy4aaN8g.png" /></figure><p>There is a whole branch of research for explainable artificial intelligence (XAI) now! While the scope of this article is not to cover XAI, if you are interested you can always read my <a href="https://towardsdatascience.com/human-interpretable-machine-learning-part-1-the-need-and-importance-of-model-interpretation-2ed758f5f476"><em>series on XAI</em></a>.</p><p>In this article, we will look at concepts, techniques and tools to interpret deep learning models used in computer vision, to be more specific — convolutional neural networks (CNNs). We will take a hands-on approach and implement our deep learning models using Keras and TensorFlow 2.0 and leverage open-source tools to interpret decisions made by these models! In short, the purpose of the article is to find out — <strong><em>what do deep learning models really see?</em></strong></p><h3>Convolutional Neural Networks</h3><p>The most popular deep learning models leveraged for computer vision problems are convolutional neural networks (CNNs)!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rO65zGl2k7xVBRhJZggQSA.gif" /><figcaption>Source: becominghuman.ai</figcaption></figure><p>CNNs typically consist of multiple convolution and pooling layers which help the deep learning model in automatically extracting relevant features from visual data like images. Due to this multi-layered architecture, CNNs learn a robust hierarchy of features, which are spatial, rotation, and translation invariant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OArfWBpCOwHsTHM4Ltfz-Q.png" /></figure><p>The key operations in a CNN model are depicted in the figure above. Any image can be represented as a tensor of pixel values. The convolution layers help in extracting features from this image (forms feature maps). Shallower layers (closer to the input data) in the network learn very generic features like edges, corners and so on. Deeper layers in the network (closer to the output layer) learn very specific features pertaining to the input image. The following graphic helps summarize the key aspects of any CNN model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YXejorVVsRBPqgrtPr1TbA.png" /></figure><p>Since we are only concerned with understanding how CNN models perceive images, we won’t be training any CNN models from scratch here. Rather, we will be leveraging the power of transfer learning and pre-trained CNN models in our examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/0*uHebnJGZ-hSb8iy-.png" /></figure><p>A pre-trained model like VGG-16 has already been pre-trained on a huge dataset (ImageNet) with a lot of diverse image categories. Considering this fact, the model should have already learned a robust hierarchy of features. Hence, the model, having learned a good representation of features for over a million images belonging to 1,000 different categories, can act as a good feature extractor for new images suitable for computer vision problems.</p><h3>Interpreting CNN Models — What does a deep learning model really see?</h3><p>Here’s the interesting part, can we really unbox the opacity presented to us by a seemingly black-box CNN model and try and understand what’s really going on under the hood and what does the model really see when it looks at an image? There are a wide variety of techniques and tools for interpreting decisions made by vision-based deep learning models. Some of the major techniques covered in this article are depicted as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_e8cWSTcpGrSs9FeOB105A.png" /></figure><p>Let’s look at each of these techniques and interpret some deep learning CNN-based models built with Keras and TensorFlow.</p><h4>SHAP Gradient Explainer</h4><p>This technique tries to combine a multitude of ideas from Integrated Gradients, SHapley Additive exPlanations (SHAP) and SmoothGrad. This technique tries to explain model decisions using expected gradients (an extension of integrated gradients). This is a feature attribution method designed for differentiable models based on an extension of Shapley values to infinite player games. We will use the <a href="https://github.com/slundberg/shap"><strong>shap</strong></a> framework here for this technique.</p><p>Integrated gradients values are a bit different from SHAP values, and require a single reference value to integrate from. However in SHAP Gradient Explainer, expected gradients reformulates the integral as an expectation and combines that expectation with sampling reference values from the background dataset. Thus this technique uses an entire dataset as the background distribution versus just a single reference value. Let’s try and implement this on some sample images. To get started, we load up some basic dependencies and model visualization function utilities.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60ce93f771f020cd5d1a5619e787fd5c/href">https://medium.com/media/60ce93f771f020cd5d1a5619e787fd5c/href</a></iframe><p>The next step is to load a pre-trained VGG-16 model, which was trained previously on the Imagenet dataset. We can do that easily with the following code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1989b6652e37331e9ef17ae970b3cb0/href">https://medium.com/media/c1989b6652e37331e9ef17ae970b3cb0/href</a></iframe><p>Once our CNN model is loaded, we will now load a small dataset of images which can be used as a background distribution and we will use four sample images for model interpretation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c511290584e600c28df5175d8962197/href">https://medium.com/media/5c511290584e600c28df5175d8962197/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hXzusiyX6FBvuiB38Riwcg.png" /></figure><p>We have four different types of images including a picture of one of my cats! Let’s first look at our model’s prediction for each of these images.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c51fd94bcd2175fd3ce8e8afd2531d05/href">https://medium.com/media/c51fd94bcd2175fd3ce8e8afd2531d05/href</a></iframe><pre><strong>[[&#39;n02999410&#39;, &#39;chain&#39;], <br> [&#39;n01622779&#39;, &#39;great_grey_owl&#39;], <br> [&#39;n03180011&#39;, &#39;desktop_computer&#39;], <br> [&#39;n02124075&#39;, &#39;Egyptian_cat&#39;]]</strong></pre><p>Let’s start by trying to visualize what the model sees in the 7th layer of the neural network (typically one of the shallower layers in the model).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/415b9b810117ea7dfba5cc0784b256cd/href">https://medium.com/media/415b9b810117ea7dfba5cc0784b256cd/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cmhi2hUPQMHEFdTespBkeg.png" /></figure><p>This gives us some good perspective into the top two predictions made by the model for each image and why did it take such decisions. Let’s take a look at one of the deeper layers in the VGG-16 model and visualize the 14th layer’s decisions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d97a5e4360cc97cda5fe84bf6adcb580/href">https://medium.com/media/d97a5e4360cc97cda5fe84bf6adcb580/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*26nDfDTQpj99fd3lNMyroQ.png" /></figure><p>Now you can see the model gets stronger and more confident with the prediction decision based on the shap value intensities and also aspects like why the model predicts a screen vs. a desktop_computer where it also looks at the keyboard. Predicting my cat as a tabby because of specific features like the nose, whiskers, facial patterns and so on!</p><h3>Interpreting CNN Models built with TensorFlow 2.0</h3><p>For the remaining four techniques, we will leverage a pre-trained model using TensorFlow 2.0 and use the popular open-source framework <a href="https://github.com/sicara/tf-explain"><strong>tf-explain</strong></a>. The idea here is to look at different model intepretation techniques for CNNs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*alMLiG5RUVx1xfK7.png" /></figure><h4>Load Pre-trained CNN Model</h4><p>Let’s load one of the most complex pre-trained CNN models out there, <a href="https://arxiv.org/abs/1610.02357"><strong><em>the Xception model</em></strong></a> which claims to be slightly better than the Inception V3 model. Let’s start by loading the necessary dependencies and our pre-trained model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6589cf90ff6e3d2f71c446e7e97970d7/href">https://medium.com/media/6589cf90ff6e3d2f71c446e7e97970d7/href</a></iframe><p>You can see from the model architecture snapshot above that this model has a total of 14 block with multiple layers in each block. Definitely one of the deeper CNN models!</p><h4>Model Predictions on Sample Image</h4><p>We will reuse the sample image of my cat and make the top-5 predictions with our Xception model. Let’s load our image first before making predictions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35ba65a2de2a8b86efc60113f0dc0a8c/href">https://medium.com/media/35ba65a2de2a8b86efc60113f0dc0a8c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*DlNrmJpfcy4ajc_UObvwXg.png" /></figure><p>Let’s now making the top-5 predictions on this image using our Xception model. We will pre-process the image before inference.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23412bd4cab6a0c8248bbc24bf745452/href">https://medium.com/media/23412bd4cab6a0c8248bbc24bf745452/href</a></iframe><pre><strong>[[(&#39;n02124075&#39;, &#39;Egyptian_cat&#39;, 0.80723596),<br>  (&#39;n02123159&#39;, &#39;tiger_cat&#39;, 0.09508163),<br>  (&#39;n02123045&#39;, &#39;tabby&#39;, 0.042587988),<br>  (&#39;n02127052&#39;, &#39;lynx&#39;, 0.00547999),<br>  (&#39;n02971356&#39;, &#39;carton&#39;, 0.0014547487)]]</strong></pre><p>Interesting predictions, at least the top 3 here definitely are relevant!</p><h4>Activation Layer Visualizations</h4><p>This technique is typically used to visualize how a given input comes out of specific activation layers. The key idea is to explore which feature maps are getting activated in the model and visualize them. Usually this is done by looking at each specific layer. The following code showcases activation layer visualizations for one of the layers in Block 2 of the CNN model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ecf2a227ee23e4e789812aaacfc6fc09/href">https://medium.com/media/ecf2a227ee23e4e789812aaacfc6fc09/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*vsf6v-K9GAa7sIUH0WkGng.png" /></figure><p>This kind of gives us an idea of which feature maps are getting activated and what parts of the image they typically focus on.</p><h4>Occlusion Sensitivity</h4><p>The idea of interpretation using occlusion sensitivity is quite intuitive. We basically try to visualize how parts of the image affects our neural network model’s confidence by occluding (hiding) parts iteratively. This is done by systematically occluding different portions of the input image with a grey square, and monitoring the output of the classifier.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ffb665cc9390fe01a406a8e673769e1/href">https://medium.com/media/4ffb665cc9390fe01a406a8e673769e1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*7rfjCGmmPDvtJOCAEIeDvw.png" /></figure><p>Ideally specific patches of the image should be highlighted in red\yellow like a heatmap but for my cat image it kind of highlighted the overall image in a red hue, the reason for this could be because of the zoomed image of a cat. However the left side of the image has a higher intensity focusing more on the shape of the cat to an extent rather than the texture of the image.</p><h4>GradCAM</h4><p>This is perhaps one of the most popular and effective methods for interpreting CNN models. Using GradCAM, we try and visualize how parts of the image affects neural network’s output by looking into the class activation maps (CAM). Class activation maps are a simple technique to get the discriminative image regions used by a CNN to identify a specific class in the image. In other words, a class activation map (CAM) lets us see which regions in the image were relevant to this class.</p><ul><li>The output of grad-CAM will be pixels that contribute to the maximization of a target function. If for example you are interested in what maximizes category number 285, then zero out all the other categories.</li><li>Compute the gradients of the target function, with respect to the convolutional layer outputs. This can be done efficiently with backpropagation</li></ul><p>Given an image and a class of interest (e.g., ‘tiger cat’ or any other type of differentiable output) as input, we forward propagate the image through the CNN part of the model and then through task-specific computations to obtain a raw score for the category. The gradients are set to zero for all classes except the desired class (tiger cat), which is set to 1. This signal is then backpropagated to the rectified convolutional feature maps of interest, which we combine to compute the coarse Grad-CAM localization (blue heatmap) which represents where the model has to look to make the particular decision.</p><p>Let’s look at GradCAM visualizations for specific blocks in our CNN model. We start by visualizing one of the layers from block 1 (shallower layer).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50685238c90a571a2cdc5900275f1e9a/href">https://medium.com/media/50685238c90a571a2cdc5900275f1e9a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*QAojvSa7XMgefRPgT1dYLQ.png" /></figure><p>Like we expected, this being one of the shallow layers, we see higher level features like edges and corners being activated in the network. Let’s now visualize GradCAM outputs from one of the deeper layers in the network in Block 6.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba2c7d0ab879e4447b4e45130206bf5e/href">https://medium.com/media/ba2c7d0ab879e4447b4e45130206bf5e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*1O9PKtoYUaXaQARoShpUug.png" /></figure><p>Things definitely start to get more interesting, we can clearly see that when the model predicts the cat as <strong>tabby</strong>, it is focusing on both the textures and also the overall shape and structure of the cat versus when it predicts the cat as an <strong>Egyptian_cat</strong>. Finally, let’s take a look at one of the deepest layers in the model from Block 14.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8128068b50bc47c8cd4c5870a3b4c476/href">https://medium.com/media/8128068b50bc47c8cd4c5870a3b4c476/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*4bes_FSAhwyBMEDr5XBmWw.png" /></figure><p>Very interesting to observe that for the tabby cat label prediction, the model is also looking at the region surrounding the cat which is basically focusing on the overall shape \ structure of the cat and also some aspects of the cat’s facial structure!</p><h4>SmoothGrad</h4><p>This technique helps us visualize stabilized gradients on the inputs towards the decision. The key objective is to identify pixels that strongly influence the final decision. A starting point for this strategy is the gradient of the class score function with respect to the input image. This gradient can be interpreted as a sensitivity map, and there are several techniques that elaborate on this basic idea.</p><p><em>SmoothGrad</em> is a simple method that can help visually sharpen gradient-based sensitivity maps. The core idea is to take an image of interest, sample similar images by adding noise to the image, then take the average of the resulting sensitivity maps for each sampled image.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0196d5db94f2f866eab2d96bba1144b7/href">https://medium.com/media/0196d5db94f2f866eab2d96bba1144b7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*LDTZ87VXrum_-8i5Wh1_1A.png" /></figure><p>For the tabby cat focus is definitely on key points on the face including patches and stripes which are very distinguishing characteristics.</p><h3>Conclusion</h3><p>This should give you a good idea of how you can not only leverage pre-trained complex CNN models to predict on new images but to even try and make an attempt to visualize what the neural network models are really seeing! The list of techniques here are not exhaustive but definitely cover some of the most popular and widely used methods to interpret CNN models. I recommend you to try these out with your own data and models!</p><p>All the code used in this article is available on <a href="https://github.com/dipanjanS"><strong><em>my GitHub</em></strong></a> in <a href="https://github.com/dipanjanS/data_science_for_all/tree/master/gde_cnn_interpretation_xai"><strong><em>this repository</em></strong></a><strong><em> </em></strong>as Jupyter notebooks.</p><p>I am a <a href="https://google-developers.appspot.com/community/experts/directory/profile/profile-dipanjan_sarkar"><em>Google Developer Expert in Machine Learning</em></a> and I look forward to sharing more interesting aspects of machine learning and deep learning over time. Feel free to check out my <a href="https://medium.com/@dipanzan.sarkar"><strong>Medium</strong></a> and <a href="https://www.linkedin.com/in/dipanzan"><strong>LinkedIn</strong></a> for updates on interesting content!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f95683e23c1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/interpreting-deep-learning-models-for-computer-vision-f95683e23c1d">Interpreting Deep Learning Models for Computer Vision</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting Malaria with Deep Learning]]></title>
            <link>https://medium.com/data-science/detecting-malaria-with-deep-learning-9e45c1e34b60?source=rss-6278d12b0682------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e45c1e34b60</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Dipanjan (DJ) Sarkar]]></dc:creator>
            <pubDate>Tue, 23 Apr 2019 13:08:11 GMT</pubDate>
            <atom:updated>2019-04-26T19:31:09.549Z</atom:updated>
            <content:encoded><![CDATA[<h4>AI for Social Good Series— Part 1</h4><h4>AI for Social Good — A Healthcare Case Study</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Aa35cz76rGh6PiDb2UEs8w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/OgvqXGL7XO4?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Hal Gatewood</a> on <a href="https://unsplash.com/collections/345758/science?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><blockquote>The content for this article has been adapted from my <a href="https://opensource.com/article/19/4/detecting-malaria-deep-learning"><strong>own article</strong></a> published previously in <a href="https://opensource.com/">opensource.com</a></blockquote><h3>Introduction</h3><p>Welcome to the AI for Social Good Series, where we will be focusing on different aspects of how Artificial Intelligence (AI) coupled with popular open-source tools, technologies and frameworks are being used for development and betterment of our society. <em>“Health is Wealth”</em> is perhaps a cliched quote yet very true! In this particular article, we will look at how AI can be leveraged for detecting malaria, a deadly disease and the promise of building a low-cost, yet effective and accurate open-source solution. The intent of the article is two-fold — understanding the motivation and importance of the deadly disease malaria and the effectiveness of deep learning in detecting malaria. We will be covering the following major topics in this article.</p><ul><li><strong>Motivation for this project</strong></li><li><strong>Methods for Malaria Detection</strong></li><li><strong>Deep Learning for Malaria Detection</strong></li><li><strong>Convolutional Neural Networks (CNNs) trained from scratch</strong></li><li><strong>Transfer Learning with Pre-trained Models</strong></li></ul><p>Now before we begin, I’d like to point out that I am neither a doctor nor a healthcare researcher and I’m nowhere near to being as qualified as they are. I do have interests though in applying AI for healthcare research. The intent of this article is not to dive into the hype that AI would be replacing jobs and taking over the world, but to showcase how AI can be useful in assisting with malaria detection, diagnosis and reducing manual labor with low-cost effective and accurate open-source solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WHN7O5OiuyZx9I5HPTE4sg.png" /><figcaption>Python and TensorFlow — A great combo to build open-source deep learning solutions</figcaption></figure><p>Thanks to the power of Python and deep learning frameworks like TensorFlow, we can build robust, scalable and effective deep learning solutions. The added benefit of these tools being open-source and free, enable us to build solutions which can be really cost effective and be adopted and used by everyone easily. Let’s get started!</p><h3>Motivation</h3><p>Malaria is a deadly, infectious mosquito-borne disease caused by Plasmodium parasites. These parasites are transmitted by the bites of infected female Anopheles mosquitoes. While we won’t get into details about the disease, there are five main types of malaria. Let’s now look at the significance of how deadly this disease can be in the following plot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wReAbo_Aap_gRyl9.png" /><figcaption>Malaria Estimated Risk Heath Map (Source: treated.com)</figcaption></figure><p>It is pretty clear that malaria is prevalent across the globe especially in tropical regions. The motivation for this project is however based on the nature and fatality of this disease. Initially if an infected mosquito bites you, parasites carried by the mosquito will get in your blood and start destroying oxygen-carrying RBCs (red blood cells). Typically the first symptoms of malaria are similar to the flu or a virus when you usually start feeling sick within a few days or weeks after the mosquito bite. However these deadly parasites can live in your body for over a year without any problems! Thus, a delay in the right treatment can lead to complications and even death. Hence early and effective testing and detection of malaria can save lives.</p><p>The World Health Organization (WHO) has released several crucial facts on malaria which you can check out <a href="https://www.who.int/features/factfiles/malaria/en/"><em>here</em></a>. In short, nearly half the world’s population is at risk from malaria and there are over 200 million malaria cases and approximately 400,000 deaths due to malaria every year. This gives us all the more motivation to make malaria detection and diagnosis fast, easy and effective.</p><h3>Methods for Malaria Detection</h3><p>There are several methods and tests which can be used for malaria detection and diagnosis. The original paper on which our data and analysis is based on, <a href="https://peerj.com/articles/4568/"><em>‘ Pre-trained convolutional neural networks as feature extractors toward improved Malaria parasite detection in thin blood smear images’</em></a> by S Rajaraman et. al. introduces us briefly to some of these methods. These include but are not limited to, thick and thin blood smear examinations, polymerase chain reaction (PCR) and rapid diagnostic tests (RDT). While we won’t cover all the methods here in detail, an important point to remember is that the latter two tests are alternative methods typically used an an alternative particularly where good quality microscopy services cannot be readily provided.</p><p>We will discuss briefly about a standard malaria diagnosis, based on a typical blood-smear workflow, thanks to <a href="https://blog.insightdatascience.com/https-blog-insightdatascience-com-malaria-hero-a47d3d5fc4bb">this wonderful article</a> by Carlos Ariza on Insight Data Science, which I got to know from Adrian Rosebrock’s <a href="https://www.pyimagesearch.com/2018/12/03/deep-learning-and-medical-image-analysis-with-keras/">excellent article on malaria detection</a> on <a href="https://www.pyimagesearch.com/"><em>pyimagesearch</em></a>, so my heartfelt thanks to both of them for such excellent resources, giving me more perspective in this domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*VWxRC2BePykk3xVVEDzbdg.png" /><figcaption>A blood smear workflow for Malaria detection (<a href="https://blog.insightdatascience.com/https-blog-insightdatascience-com-malaria-hero-a47d3d5fc4bb">Source</a>)</figcaption></figure><p>Based on the guidelines from the WHO protocol, this procedure involves intensive examination of the blood smear at a 100X magnification, where people manually count red blood cells that contain parasites out of 5000 cells. In fact the paper by Rajaraman et. al. which we mentioned previously, talks about the exact same thing and I quote the following exerpt from the paper to make things clearer.</p><blockquote>Thick blood smears assist in detecting the presence of parasites while thin blood smears assist in identifying the species of the parasite causing the infection (Centers for Disease Control and Prevention, 2012). The diagnostic accuracy heavily depends on human expertise and can be adversely impacted by the inter-observer variability and the liability imposed by large-scale diagnoses in disease-endemic/resource-constrained regions (Mitiku, Mengistu &amp; Gelaw, 2003). Alternative techniques such as polymerase chain reaction (PCR) and rapid diagnostic tests (RDT) are used; however, PCR analysis is limited in its performance (Hommelsheim et al., 2014) and RDTs are less cost-effective in disease-endemic regions (Hawkes, Katsuva &amp; Masumbuko, 2009).</blockquote><p>Thus, malaria detection is definitely an intensive manual process which can perhaps be automated using deep learning which forms the basis of this article.</p><h3>Deep Learning for Malaria Detection</h3><p>With regular manual diagnosis of blood smears, it is an intensive manual process requiring proper expertise in classifying and counting the parasitized and uninfected cells. Typically this may not scale well and might cause problems if we do not have the right expertise in specific regions around the world. Some advancements have been made in leveraging state-of-the-art (SOTA) image processing and analysis techniques to extract hand-engineered features and build machine learning based classification models. However these models are not scalable with more data being available for training and given the fact that hand-engineered features take a lot of time.</p><p>Deep Learning models, or to be more specific, Convolutional Neural Networks (CNNs) have proven to be really effective in a wide variety of computer vision tasks. While we assume that you have some knowledge on CNNs, in case you don’t, feel free to dive deeper into them by checking out <a href="http://cs231n.github.io/convolutional-networks/">this article here</a>. Briefly, The key layers in a CNN model include convolution and pooling layers as depicted in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*-XHx36NZBaazD8Gk.png" /><figcaption>A typical CNN architeture (Source: deeplearning.net)</figcaption></figure><p>Convolution layers learn spatial hierarchical patterns from the data, which are also translation invariant. Thus they are able to learn different aspects of images. For example, the first convolution layer will learn small and local patterns such as edges and corners, a second convolution layer will learn larger patterns based on the features from the first layers, and so on. This allows CNNs to automate feature engineering and learn effective features which generalize well on new data points. Pooling layers help with downsampling and dimension reduction.</p><p>Thus, CNNs help us with automated and scalable feature engineering. Also, plugging in dense layers at the end of our model enables us to perform tasks like image classification. Automated malaria detection using deep learning models like CNNs could be very effective, cheap and scalable especially with the advent of transfer learning and pre-trained models which work quite well even with constraints like less data.</p><p>The paper by Rajaraman et al. , ‘<a href="https://lhncbc.nlm.nih.gov/system/files/pub9752.pdf"><em>Pre-trained convolutional neural networks as feature extractors toward improved parasite detection in thin blood smear images</em></a><em>’ </em>leverages a total of six pre-trained models on the data mentioned in their paper to obtain an impressive accuracy of 95.9% in detecting malaria vs. non-infected samples. Our focus would be to try out some simple CNN models from scratch and a couple of pre-trained models using transfer learning to see the kind of results we get on the same dataset! We will be using open-source tools and frameworks which include Python and TensorFlow to build our models.</p><h3>Dataset Details</h3><p>Let’s talk about the dataset we would be using in our analysis. We are lucky to have researchers at the Lister Hill National Center for Biomedical Communications (LHNCBC), part of National Library of Medicine (NLM) who have carefully collected and annotated this dataset of healthy and infected blood smear images. You can download these images from <a href="https://ceb.nlm.nih.gov/repositories/malaria-datasets/"><strong><em>the official website</em></strong></a>.</p><p>In fact they have developed a mobile application that runs on a standard Android smartphone attached to a conventional light microscope (Poostchi et al., 2018). Giemsa-stained thin blood smear slides from 150 P. falciparum-infected and 50 healthy patients were collected and photographed at Chittagong Medical College Hospital, Bangladesh. The smartphone’s built-in camera acquired images of slides for each microscopic field of view. The images were manually annotated by an expert slide reader at the Mahidol-Oxford Tropical Medicine Research Unit in Bangkok, Thailand. Let’s briefly check out our dataset structure. We install some basic dependencies first based on the OS being used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*YJkziNUn29mvPMNz4lvbSg.png" /></figure><p>I am using a Debian based system on the cloud having a GPU so I can run my models faster! Install the tree dependency in case you don’t have it so we can view our directory structure (<strong>sudo apt install tree</strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*qQK0HV0OJo6NQayQZ1s7Jg.png" /></figure><p>Looks like we have two folders which contain images of cells which are infected and healthy. We can get further detail of the total number of images using the following code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f403f63c8eb9fba705cf4cb3393a3c0/href">https://medium.com/media/7f403f63c8eb9fba705cf4cb3393a3c0/href</a></iframe><p>Looks like we have a balanced dataset of <strong>13779 </strong><em>malaria </em>and <em>non-malaria</em> (uninfected) cell images. Let’s build a dataframe from this which will be of use to us shortly as we start building our datasets.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75e7115cc09e7b2cf24f7a8c3e9fe542/href">https://medium.com/media/75e7115cc09e7b2cf24f7a8c3e9fe542/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*53ikUzpkTO9nUcnN0I7I7g.png" /></figure><h3>Build and Explore Image Datasets</h3><p>To build deep learning models we need training data but we also need to test the model’s performance on unseen data. We will use a 60:10:30 split for train, validatation and test datasets respectively. We will leverage the train and validation datasets during training and check the performance of the model on the test dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2009695c82b2b89ef5217a1ec9cdc68/href">https://medium.com/media/a2009695c82b2b89ef5217a1ec9cdc68/href</a></iframe><p>Now obviously the images will not be of equal dimensions given blood smears and cell images will vary based on the human, the test method and the orientation in which the photo was taken. Let’s get some summary statistics of our training dataset to decide optimal image dimensions (remember we don’t touch the test dataset at all!).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b36783625df3eb99bb393183cd9bceae/href">https://medium.com/media/b36783625df3eb99bb393183cd9bceae/href</a></iframe><p>We apply parallel processing to speed up the image read operations and based on the summary statistics, we have decided to resize each image to <strong>125x125</strong> pixels. Let’s load up all our images and resize them to these fixed dimensions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/897ad276a1f5a0a6e1bd106d3e89b0df/href">https://medium.com/media/897ad276a1f5a0a6e1bd106d3e89b0df/href</a></iframe><p>We leverage parallel processing again to speed up computations pertaining to image load and resizing. Finally we get our image tensors of desired dimensions as depicted in the preceding output. We can now view some sample cell images to get an idea of how our data looks like.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74429e0f7548211444e133bde100cd07/href">https://medium.com/media/74429e0f7548211444e133bde100cd07/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*PffWf_ixyqDfpUHazzugvA.png" /></figure><p>Based on the sample images above, we can notice some subtle differences between malaria and healthy cell images. We will basically make our deep learning models try and learn these patterns during model training. We setup some basic configuration settings before we start training our models.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2f34f7a85da4eef5896f3d19ac27c3b/href">https://medium.com/media/a2f34f7a85da4eef5896f3d19ac27c3b/href</a></iframe><p>We fix our image dimensions, batch size, epochs and encode our categorical class labels. The alpha version of TensorFlow 2.0 was released on March, 2019 just a couple of weeks before this article was written and it gives us a perfect excuse to try it out!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ef7c89b04e699563534446a3405fbd9/href">https://medium.com/media/6ef7c89b04e699563534446a3405fbd9/href</a></iframe><h3>Deep Learning Model Training Phase</h3><p>In the model training phase, we will build several deep learning models and train them on our training data and compare their performance on the validation data. We will then save these models and use them later on again in the model evaluation phase.</p><h4>Model 1: CNN from Scratch</h4><p>Our first malaria detection model will be building and training a basic convolutional neural network (CNN) from scratch. First let’s define our model architecture.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4deaa9df9c62b486c8b1e21dc21b1ed0/href">https://medium.com/media/4deaa9df9c62b486c8b1e21dc21b1ed0/href</a></iframe><p>Based on the architecture in the preceding code, our CNN model has three convolution and pooling layers followed by two dense layers and dropout for regularization. Let’s train our model now!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1800d21cefbab63b9350238dee7e1a0d/href">https://medium.com/media/1800d21cefbab63b9350238dee7e1a0d/href</a></iframe><p>We get a <strong><em>validation accuracy</em></strong> of <strong>95.6%</strong> which is pretty good, though our model looks to be overfitting slightly looking at our <strong><em>training accuracy</em></strong> which is <strong>99.9%</strong>. We can get a clear perspective on this by plotting the training and validation accuracy and loss curves.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90f5321beb73b2a6879366cdba38c933/href">https://medium.com/media/90f5321beb73b2a6879366cdba38c933/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ZD-X4VKd0QllI5id2JTjpw.png" /><figcaption>Learning Curves for Basic CNN</figcaption></figure><p>Thus we can see after the fifth epoch, things don’t seem to improve a whole lot overall. Let’s save this model for future evaluation.</p><pre><strong>model.save(&#39;basic_cnn.h5&#39;)</strong></pre><h4>Deep Transfer Learning</h4><p>Just like humans have an inherent capability of being able to transfer knowledge across tasks, transfer learning enables us to utilize knowledge from previously learned tasks and apply them to newer, related ones even in the context of machine learning or deep learning. A comprehensive coverage of transfer learning is available in <a href="https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a"><em>my article</em></a> and <a href="https://github.com/dipanjanS/hands-on-transfer-learning-with-python"><em>my book</em></a> for readers interested in doing a deep-dive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*Ww3AMxZeoiB84GVSRBr4Bw.png" /><figcaption>Ideas for deep transfer learning</figcaption></figure><p>For the purpose of this article, the idea is, can we leverage a pre-trained deep learning model (which was trained on a large dataset — like <a href="http://image-net.org/index"><em>ImageNet</em></a>) to solve the problem of malaria detection by applying and transferring its knowledge in the context of our problem?</p><p>We will apply the two most popular strategies for deep transfer learning.</p><ul><li><strong>Pre-trained Model as a Feature Extractor</strong></li><li><strong>Pre-trained Model with Fine-tuning</strong></li></ul><p>We will be using the pre-trained VGG-19 deep learning model, developed by the Visual Geometry Group (VGG) at the University of Oxford, for our experiments. A pre-trained model like the VGG-19 is an already pre-trained model on a huge dataset (ImageNet) with a lot of diverse image categories. Considering this fact, the model should have learned a robust hierarchy of features, which are spatial, rotation, and translation invariant with regard to features learned by CNN models. Hence, the model, having learned a good representation of features for over a million images, can act as a good feature extractor for new images suitable for computer vision problems just like malaria detection! Let’s briefly discuss the VGG-19 model architecture before unleashing the power of transfer learning on our problem.</p><h4>Understanding the VGG-19 model</h4><p>The VGG-19 model is a 19-layer (convolution and fully connected) deep learning network built on the ImageNet database, which is built for the purpose of image recognition and classification. This model was built by Karen Simonyan and Andrew Zisserman and is mentioned in their paper titled <a href="https://arxiv.org/pdf/1409.1556.pdf"><em>‘Very Deep Convolutional Networks for Large-Scale Image Recognition’</em></a><em>. </em>I recommend all interested readers to go and read up on the excellent literature in this paper. The architecture of the VGG-19 model is depicted in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*SYxBT4hxo65twqBNA2-BmQ.png" /><figcaption>VGG-19 Model Architecture</figcaption></figure><p>You can clearly see that we have a total of <strong>16</strong> convolution layers using <strong>3 x 3</strong>convolution filters along with max pooling layers for downsampling and a total of two fully connected hidden layers of <strong>4096</strong> units in each layer followed by a dense layer of <strong>1000</strong> units, where each unit represents one of the image categories in the ImageNet database. We do not need the last three layers since we will be using our own fully connected dense layers to predict malaria. We are more concerned with the first five blocks, so that we can leverage the VGG model as an effective feature extractor.</p><p>For one of the models, we will use it as a simple feature extractor by freezing all the five convolution blocks to make sure their weights don’t get updated after each epoch. For the last model, we will apply fine-tuning to the VGG model, where we will unfreeze the last two blocks (Block 4 and Block 5) so that their weights get updated in each iteration (per batch of data) as we train our own model.</p><h4>Model 2: Pre-trained Model as a Feature Extractor</h4><p>For building this model, we will leverage TensorFlow to load up the VGG-19 model, and freeze the convolution blocks so that we can use it as an image feature extractor. We will plugin our own dense layers at the end for performing the classification task.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7b5fac359ecce2096be78f7d5116529/href">https://medium.com/media/d7b5fac359ecce2096be78f7d5116529/href</a></iframe><p>Thus it is quite evident from the preceding output that we have a lot of layers in our model and we will be using the frozen layers of the VGG-19 model as feature extractors only. You can use the following code to verify how many layers in our model are indeed trainable and how many total layers are present in our network.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca553f6951b71139460f672719865668/href">https://medium.com/media/ca553f6951b71139460f672719865668/href</a></iframe><p>We will now train our model using similar configurations and callbacks which we used in our previous model. Refer to <a href="https://nbviewer.jupyter.org/github/dipanjanS/data_science_for_all/tree/master/os_malaria_detection/"><strong><em>my GitHub repository</em></strong></a> for the complete code to train the model. We observe the following plots showing the model’s accuracy and loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*SK1X07uZsU7ygZvuINYM_A.png" /><figcaption>Learning Curves for frozen pre-trained CNN</figcaption></figure><p>This shows us that our model is not overfitting as much as our basic CNN model but the performance is not really better and in fact is sligtly lesser than our basic CNN model. Let’s save this model now for future evaluation.</p><pre><strong>model.save(&#39;vgg_frozen.h5&#39;)</strong></pre><h4>Model 3: Fine-tuned Pre-trained Model with Image Augmentation</h4><p>In our final model, we will fine-tune the weights of the layers present in the last two blocks of our pre-trained VGG-19 model. Besides that, we will also introduce the concept of image augmentation. The idea behind image augmentation is exactly as the name sounds. We load in existing images from our training dataset and apply some image transformation operations to them, such as rotation, shearing, translation, zooming, and so on, to produce new, altered versions of existing images. Due to these random transformations, we don’t get the same images each time. We will leverage an excellent utility called <strong>ImageDataGenerator</strong> in <strong>tf.keras</strong> that can help us build image augmentors.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5364145f406cc543c0f7b36b372b0c8b/href">https://medium.com/media/5364145f406cc543c0f7b36b372b0c8b/href</a></iframe><p>We do not apply any transformations on our validation dataset except scaling the images (which is mandatory), since we will be using it to evaluate our model performance per epoch. For detailed explanation of image augmentation in the context of transfer learning feel free to check out <a href="https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a"><em>my article</em></a> if needed. Let&#39;s take a look at some sample results from a batch of image augmentation transforms.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f9edb6356d73b7fe18af0900da6d16ae/href">https://medium.com/media/f9edb6356d73b7fe18af0900da6d16ae/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*i6FikxlgMiwpjqjQdwoBCw.png" /><figcaption>Sample Augmented Images</figcaption></figure><p>You can clearly see the slight variations of our images in the preceding output. We will now build our deep learning model making sure the last two blocks of the VGG-19 model is trainable.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99d85a8c1f602a7d6e17018138efbbdc/href">https://medium.com/media/99d85a8c1f602a7d6e17018138efbbdc/href</a></iframe><p>We reduce the learning rate in our model since we don’t want to make to large weight updates to the pre-trained layers when fine-tuning. The training process of this model will be slightly different since we are using data generators and hence we will be leveraging the <strong>fit_generator(…) </strong>function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1433a83255036a58e174a8d3e295b948/href">https://medium.com/media/1433a83255036a58e174a8d3e295b948/href</a></iframe><p>This looks to be our best model yet giving us a <strong><em>validation accuracy</em></strong> of almost <strong>96.5%</strong> and based on the training accuracy, it doesn’t look like our model is overfitting as much as our first model. This can be verified with the following learning curves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*xReuJs1_wpMo7qWFk2ovow.png" /><figcaption>Learning Curves for fine-tuned pre-trained CNN</figcaption></figure><p>Let’s save this model now so that we can use it for model evaluation on our test dataset shortly.</p><pre><strong>model.save(&#39;vgg_finetuned.h5&#39;)</strong></pre><p>This completes our model training phase and we are now ready to test the performance of our models on the actual test dataset!</p><h3>Deep Learning Model Performance Evaluation Phase</h3><p>We will now evaluate the three different models that we just built in the training phase by making predictions with them on the data from our test dataset, because just validation is not enough! We have also built a nifty utility module called <strong>model_evaluation_utils</strong>, which we will be using to evaluate the performance of our deep learning models with relevant classification metrics. The first step here is to obviously scale our test data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9ed01fcaa1ed6fc8ee21df9f258a636/href">https://medium.com/media/b9ed01fcaa1ed6fc8ee21df9f258a636/href</a></iframe><p>The next step involves loading up our saved deep learning models and making predictions on the test data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21964efc8f4340c18fd6d76cbec3d1ee/href">https://medium.com/media/21964efc8f4340c18fd6d76cbec3d1ee/href</a></iframe><p>The final step is to leverage our <strong>model_evaluation_utils</strong> module and check the performance of each model with relevant classification metrics.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2cc367d2a03d5351704f052a1141bb3/href">https://medium.com/media/e2cc367d2a03d5351704f052a1141bb3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*aNv8fw460qptKmB6dyGhyQ.png" /></figure><p>Looks like our third model performs the best out of all our three models on the test dataset giving a model <strong><em>accuracy</em></strong> as well as<strong><em> f1-score</em></strong> of <strong>96%</strong> which is pretty good and quite comparable to the more complex models mentioned in the research paper and articles we mentioned earlier!</p><h3>Conclusion</h3><p>We looked at an interesting real-world medical imaging case study of malaria detection in this article. Malaria detection by itself is not an easy procedure and the availability of the right personnel across the globe is also a serious concern. We looked at easy to build open-source techniques leveraging AI which can give us state-of-the-art accuracy in detecting malaria thus enabling AI for social good. I encourage everyone to check out the articles and research papers mentioned in this article, without which it would have been impossible for me to conceptualize and write this article. Let’s hope for more adoption of open-source AI capabilities across healthcare making it cheaper and accessible for everyone across the world!</p><blockquote>This article has been adapted from my <a href="https://opensource.com/article/19/4/detecting-malaria-deep-learning"><strong>own article</strong></a> published previously in <a href="https://opensource.com/">opensource.com</a></blockquote><p>If you are interested in running or adopting all the code used in this article, it is available on <a href="https://nbviewer.jupyter.org/github/dipanjanS/data_science_for_all/tree/master/os_malaria_detection/"><strong><em>my GitHub repository</em></strong></a>.</p><p>Remember to download the data from the <a href="https://ceb.nlm.nih.gov/repositories/malaria-datasets/"><strong><em>official website</em></strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e45c1e34b60" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/detecting-malaria-with-deep-learning-9e45c1e34b60">Detecting Malaria with Deep Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>