<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eddie Smolyansky on Medium]]></title>
        <description><![CDATA[Stories by Eddie Smolyansky on Medium]]></description>
        <link>https://medium.com/@eddiesmo?source=rss-2d85974b345f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tJ79i_Lb0Y3XqFWCQeExzA.jpeg</url>
            <title>Stories by Eddie Smolyansky on Medium</title>
            <link>https://medium.com/@eddiesmo?source=rss-2d85974b345f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:36:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eddiesmo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Announcing Premium accounts, multi-origin graphs and more]]></title>
            <link>https://medium.com/connectedpapers/announcing-premium-accounts-multi-origin-graphs-and-more-92e8dbd66848?source=rss-2d85974b345f------2</link>
            <guid isPermaLink="false">https://medium.com/p/92e8dbd66848</guid>
            <category><![CDATA[premium]]></category>
            <category><![CDATA[launch]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[features]]></category>
            <dc:creator><![CDATA[Eddie Smolyansky]]></dc:creator>
            <pubDate>Wed, 27 Jul 2022 18:43:52 GMT</pubDate>
            <atom:updated>2022-07-27T18:43:52.180Z</atom:updated>
            <content:encoded><![CDATA[<p>In 2020, we <a href="https://medium.com/connectedpapers/announcing-connected-papers-a-visual-tool-for-researchers-to-find-and-explore-academic-papers-89146a54c7d4">launched Connected Papers</a> as a side project that we’ve worked on to solve our own pains with literature search. It was an immediate success with researchers and the scientific community, and grew <a href="https://www.connectedpapers.com/testimonials">wildly beyond our expectations</a>.</p><p>Today we are building over 50 graphs every minute, with more than <strong>two million researchers</strong> using Connected Papers in the past year. It is crazy that all of this growth has happened by word of mouth — we don’t do any marketing.</p><p>For over two years, we have provided Connected Papers completely for free, and while this has been an absolute privilege and a wonderful ride, it has also taken a toll — in terms of the time and personal money our small team put in to keep improving the service and handle the growing server costs.</p><h3>Introducing Premium accounts</h3><p>Starting August 1st, Connected Papers will have two types of accounts: Free and Premium.</p><p>The Free plan will include all the features that already exist in Connected Papers, but will be limited to 5 graphs per month. This is enough to cover the research needs of the majority of our users.</p><p>The Premium plans (Academic and Business) are aimed at active researchers who want to perform more searches and unlock an unlimited amount of graphs every month. We have two plans:</p><ul><li><strong>Academic:</strong> for researchers in academic institutes, non-profit organizations and for personal use.</li><li><strong>Business</strong>: for individuals and organizations who use Connected Papers in a commercial context.</li></ul><p>You can learn more about our plans on the <a href="https://www.connectedpapers.com/pricing">Pricing page</a>.</p><h3>Unveiling multi-origin graphs</h3><p>In addition, we are announcing a new core way to use Connected Papers — multi-origin graphs!</p><p>This has been by far the most requested feature from the community. It’s taken us a long time to implement, because it’s very tricky to get right.</p><p>The way it works is — from any graph, you can select any node and add it as a second origin via the right side panel button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nbf1rqhVU3Hnx8QWFYEgWw.png" /><figcaption>Add a paper as a graph origin</figcaption></figure><p>This will construct a new graph, where all the nodes (papers) in it are selected by similarity to <strong><em>both</em></strong> origin papers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*6J1VMwqwHalcsEUKuR2LCw.png" /><figcaption>A graph where papers are selected and arranged by similarity to <em>both origins.</em></figcaption></figure><p>You can then add a third origin, a fourth, and so on.</p><p><strong>In essence, every time you add an origin you are <em>refining your search</em>. You are telling the algorithm: “I want more papers like these, please”.</strong></p><p>Multi-Origin graphs will be available in all plans: Free and Premium.</p><h3>User accounts and support for mobile</h3><p>Over the past few months, we have also released several important features to Connected Papers.</p><p>In April, we announced <a href="https://twitter.com/ConnectedPapers/status/1518961093648953344/photo/1">Connected Papers for mobile browsers</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZRG4gKqhKw8RFN5bbwFfFg.png" /><figcaption>Connected Papers now on mobile!</figcaption></figure><p>and in June we <a href="https://twitter.com/ConnectedPapers/status/1533839698141270021">announced</a> User Accounts, along with a way to save papers to a list and the ability to view your previous graphs history. This is just the beginning, with more features and improvements on the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/1*3MXd4elKhSLYHjxYTQbOxA.png" /><figcaption>User Accounts</figcaption></figure><h3>Looking forward to the future</h3><p>With the launch of Premium accounts we are trying out a new business model that would hopefully allow us to continue to follow our passion and develop more and better tools for scientists and researchers.</p><p>We are very excited about the future. We hope you are, too!</p><p><strong>Thank you!<br></strong>The Connected Papers team<br>Alex, Itay, Sahar and Eddie</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92e8dbd66848" width="1" height="1" alt=""><hr><p><a href="https://medium.com/connectedpapers/announcing-premium-accounts-multi-origin-graphs-and-more-92e8dbd66848">Announcing Premium accounts, multi-origin graphs and more</a> was originally published in <a href="https://medium.com/connectedpapers">Connected Papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connected Papers partners with arXiv]]></title>
            <link>https://medium.com/connectedpapers/connected-papers-partners-with-arxiv-8ce8122f6b4c?source=rss-2d85974b345f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ce8122f6b4c</guid>
            <category><![CDATA[science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[academia]]></category>
            <category><![CDATA[partnerships]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Eddie Smolyansky]]></dc:creator>
            <pubDate>Wed, 03 Feb 2021 20:33:24 GMT</pubDate>
            <atom:updated>2021-02-06T15:33:53.929Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re excited to announce a collaboration that we’ve been working on with the wonderful team at arxiv.org!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_g8pZzIJOohbu2XTqazRPQ.png" /></figure><p><strong>Starting today, every paper page on arXiv.org will link to a graph of Connected Papers.</strong></p><p>The link can be found under the <em>Related Papers</em><strong><em> </em></strong>tab<strong> </strong>in the section below the paper description.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*8lWjgRb4Q4pSb-gHl84DUA.png" /></figure><p>And this is how it works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9SLWQGHeV0kZwEx9ehrlMw.gif" /></figure><p>Clicking the link will take you directly to the Connected Papers graph which originates from that specific paper, where you can visually explore similar papers and widen or deepen your literature review.</p><p>For those unfamiliar with these tools, here’s a quick rundown:</p><h4>Connected Papers: Visual discovery of academic papers</h4><p><a href="https://www.connectedpapers.com/"><strong>Connected Papers</strong></a> is a free visual tool that helps researchers and applied scientists find and explore papers relevant to their field of work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PjEUPSfaYtWgUjATRZD3yA.png" /><figcaption>A Connected Papers <a href="https://www.connectedpapers.com/main/faae66791d209449cf4ebe8b819e63098ca17d5c/The-Coevolution-of-Galaxies-and-Supermassive-Black-Holes-Insights-from-Surveys-of-the-Contemporary-Universe/graph">graph</a> for “The Coevolution of Galaxies and Supermassive Black Holes…”</figcaption></figure><p>You can use Connected Papers to:</p><ul><li>Get a visual overview of a new academic field</li><li>Create a bibliography for your thesis</li><li>Discover the most relevant prior and derivative works</li><li>Or simply explore paper-space!</li></ul><h4>arXivLabs</h4><p><a href="https://labs.arxiv.org/">arXivLabs</a> is a recent offering by arXiv.org which aims to enhance and improve the browsing experience on the website. It’s a new framework for enabling the arXiv community to contribute to arXiv and develop tools that can benefit the scientific community.</p><p>We’d like to thank the arXiv team for their work and strongly encourage more tools and products to integrate with them to improve the academic ecosystem.</p><h4>Our mission: Improving work with academic papers</h4><p>Connected Papers started as a side project to solve some of our own problems in working with academic papers. It is developed in our free time as a service to the scientific community. We still have many ideas, features and products we’d like to explore, and as our user base grows we are excited about the future.</p><p>Recently we’ve <a href="https://twitter.com/ConnectedPapers/status/1347616683641466880">introduced a graph history feature</a> and <a href="https://twitter.com/ConnectedPapers/status/1329078023488823296">released a special version of our website for conferences</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IgCAFhvVhK9CIwvYqM5TLA.gif" /><figcaption>New feature: Graph History</figcaption></figure><p>In addition, we’ve added <a href="https://twitter.com/ConnectedPapers/status/1329832306719281157">links to external resources</a> in the paper details panel and <a href="https://twitter.com/ConnectedPapers/status/1330915427472584706">partnered with Papers with Code</a>.</p><p>To stay up to date with our work, follow us on Twitter <a href="https://twitter.com/ConnectedPapers">@connectedpapers</a>.</p><h4>Call for sponsors</h4><p>We’d like to thank our sponsor <a href="https://dagshub.com/">DAGsHub</a> for supporting us with this development. Check out their recent blog: <a href="https://dagshub.com/blog/datasets-should-behave-like-git-repositories/">Datasets should behave like git repositories</a>.</p><p>If you’d like to be our sponsor, feel free to reach out at sponsors@connectedpapers.com</p><p>Thanks!<br>The Connected Papers team</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ce8122f6b4c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/connectedpapers/connected-papers-partners-with-arxiv-8ce8122f6b4c">Connected Papers partners with arXiv</a> was originally published in <a href="https://medium.com/connectedpapers">Connected Papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connected Papers — Post Launch Community Update]]></title>
            <link>https://medium.com/connectedpapers/connected-papers-post-launch-community-update-64d952423d56?source=rss-2d85974b345f------2</link>
            <guid isPermaLink="false">https://medium.com/p/64d952423d56</guid>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[academia]]></category>
            <category><![CDATA[launch]]></category>
            <dc:creator><![CDATA[Eddie Smolyansky]]></dc:creator>
            <pubDate>Mon, 31 Aug 2020 14:01:23 GMT</pubDate>
            <atom:updated>2020-09-04T04:33:05.796Z</atom:updated>
            <content:encoded><![CDATA[<h3>Connected Papers — Post Launch Community Update</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VfViSZcoG3LRfaToDfCmjg.png" /><figcaption>Connected Papers <a href="https://www.connectedpapers.com/main/7bba95b3d145564025e26b49ca67f13f884f8560/Superintelligence-Paths-Dangers-Strategies/graph">graph</a> for Superintelligence by Nick Bostrom</figcaption></figure><p>Connected Papers <a href="https://medium.com/connectedpapers/announcing-connected-papers-a-visual-tool-for-researchers-to-find-and-explore-academic-papers-89146a54c7d4?sk=eb6c686826e03958504008fedeffea18">launched in June</a> and the response has been absolutely amazing. What started as a side project on the weekends ended up servicing over 200,000 users in just its first month out of beta. Talk about stress-testing…</p><p>In this post we’ll discuss what we’ve been up to since launch and our plans for the future.</p><h3>Community reception</h3><p>Frankly, we have been blown away.</p><p>We received hundreds of love letters in emails and on <a href="https://twitter.com/ConnectedPapers">Twitter</a>. Our user base is extremely global. We’ve had blog posts written about us in <a href="http://drsearchio.blogspot.com/2020/06/connected-papers.html?m=1">Hebrew</a>, <a href="https://www.youtube.com/watch?v=mHMsrXUxduQ">Turkish</a>, <a href="https://mp.weixin.qq.com/s/rF-ZF6jj4MgKgfQ6PUU8lw">Chinese</a>, <a href="https://universoabierto.org/2020/07/07/connected-papers-herramienta-que-permite-descubrir-documentos-relevantes-en-un-campo-de-trabajo/">Spanish</a>, <a href="http://blog.csuc.cat/?p=20341">Catalan</a> and <a href="https://www.servicescape.com/blog/connected-papers-a-game-changing-tool-for-researchers?s=09">more</a> <a href="https://medium.com/a-academic-librarians-thoughts-on-open-access/4-ways-to-find-review-papers-systematic-reviews-meta-analysis-and-other-rich-sources-of-82898aebb6e7">than</a> <a href="https://stevenhill.org.uk/connecting-journal-articles/">a</a> <a href="https://www.linkedin.com/feed/update/urn:li:activity:6674551399281418240/">few</a> <a href="https://irenevigueguix.wordpress.com/2020/06/05/connected-papers-a-visual-tool-to-explore-related-papers-for-your-research/">in</a> <a href="https://zengo.com/zengo-connected-papers/">English</a>.</p><p>We just want to say — we love you back and we will keep working to make Connected Papers the best experience for researchers world-wide. <br>And we will try to keep it free, always.</p><h3>Funding</h3><p>When we launched Connected Papers we were planning to fund the service ourselves, as an altruistic side project. Given the huge demand, it became obvious that our personal budgets were not going to be enough.</p><p>During the last couple of months we’ve worked behind the scenes to secure funding for the servers and operations. We have partnered with two companies who support us financially, and we joined the “Microsoft for Startups” program which provides us with cloud credits.</p><p>We have also added a donation button on the website for individuals who are capable and willing to support us. The money goes directly to fund the servers and future development of the website. Thank you!</p><h3>What’s new?</h3><ul><li><strong>More powerful servers:</strong><br>We have migrated our servers to Azure cloud, and our devops infrastructure has been seriously upgraded — we are using more powerful machines, and graphs are now built twice as fast as they did at launch! We have also increased the number of graphs each user can build concurrently from 3 to 5.</li><li><strong>Deepened partnership with Semantic Scholar:</strong><br>We became official data partners of Semantic Scholar, whose wonderful team is assembling the biggest academic database in the world. Connected Papers is making extensive use of both their data and search APIs, and we are extremely grateful for their contribution to open science.</li><li><strong>New Features:<br></strong>We have received hundreds of feedback messages from the community and have prepared an internal roadmap for new features and improvements. This month, we have added an option to <a href="https://twitter.com/ConnectedPapers/status/1293197316480999427">Download the Connected Papers Lists</a> and added a button to <a href="https://twitter.com/ConnectedPapers/status/1297996092366348288">Report errors in paper metadata</a>.</li></ul><p>Check out the <a href="https://www.notion.so/Connected-Papers-Release-Notes-f758d434dfcc45e2b8ed3d0452739982">full release notes</a> or follow us on <a href="https://twitter.com/ConnectedPapers">Twitter </a>to get real-time updates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*d8Ju9ZqQ0yIhZ7S_TKN0mA.gif" /><figcaption>New Feature: Download Connected Paper Lists</figcaption></figure><h3>Meet our sponsors</h3><p>We have been extremely lucky to be in contact with two innovative companies who wish to support our project and the research community. We are very proud to have them as our first sponsors!</p><ul><li><a href="https://dagshub.com/">DAGsHub</a>: A platform for open source data science. They make it easy to share and collaborate on data, code, models and experiments. This team shares a lot of its values with us and is building next generation infrastructure for the machine learning community.</li><li><a href="https://zengo.com/research/">ZenGo X</a>: The team building a crypto wallet that is both secure and user friendly. Their research is focused on applied cryptography and distributed systems and they have been using Connected Papers from its very beginnings. Their co-founder, Omer Shlomovits, wrote a wonderful <a href="https://zengo.com/zengo-connected-papers/">blog post</a> about how Connected Papers impacts their research.</li></ul><p>Would you like to support Connected Papers and become one of our sponsors? Drop us a line at <a href="mailto:sponsors@connectedpapers.com">sponsors@connectedpapers.com</a>.</p><h3>What’s next?</h3><p>We have big plans and small plans. In the next few months we will continue to release updates to the core functionality and beyond.</p><p>Our plans include:</p><ul><li>A Connected Papers browser extension</li><li>A way to highlight the connection between each paper and the origin of a graph</li><li>A feature to allow users to view historical versions of the graphs for each paper and an option to re-run graphs with the latest database updates (useful for new papers that just came out)</li></ul><p>In the long run we want to continue building products that propel the research experience into the 21st century, and allow researchers to access academic papers in an intuitive, global, open, collaborative environment.</p><p>Thank you to everyone who’s supported us on this journey so far. We are just getting started.</p><p>The <a href="https://www.connectedpapers.com/">Connected Papers</a> team: <br>Alex, Itay and Eddie</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64d952423d56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/connectedpapers/connected-papers-post-launch-community-update-64d952423d56">Connected Papers — Post Launch Community Update</a> was originally published in <a href="https://medium.com/connectedpapers">Connected Papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connected Papers — a visual tool for researchers to find and explore academic papers]]></title>
            <link>https://medium.com/connectedpapers/announcing-connected-papers-a-visual-tool-for-researchers-to-find-and-explore-academic-papers-89146a54c7d4?source=rss-2d85974b345f------2</link>
            <guid isPermaLink="false">https://medium.com/p/89146a54c7d4</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[academia]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Eddie Smolyansky]]></dc:creator>
            <pubDate>Tue, 02 Jun 2020 13:58:37 GMT</pubDate>
            <atom:updated>2023-05-29T18:57:39.599Z</atom:updated>
            <content:encoded><![CDATA[<h3>Announcing Connected Papers — a visual tool for researchers to find and explore academic papers</h3><p>After a long beta, today we are really excited to release <a href="http://connectedpapers.com">Connected Papers</a> to the public. To use it, <strong>simply enter a paper of interest and we will generate a graph that shows that section of paper-space and its interconnections.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1jR_0VZk92Yh074MCqTHSg.png" /><figcaption>connectedpapers.com. Discover your next papers through an interactive graph.</figcaption></figure><p>To create each graph, we analyze an order of ~50,000 papers and select the few dozen with the strongest connections to the origin paper (more on that below).</p><p>Each node (circle) in the graph represents a paper which can be previewed on the right side panel. By clicking the “Build a graph” button, that paper becomes the origin to a new graph that’s created in the background. Every such hop can refine or expand the search to reveal more relevant papers. Congratulations, now you’re t̶h̶i̶n̶k̶i̶n̶g̶ ̶w̶i̶t̶h̶ ̶p̶o̶r̶t̶a̶l̶s exploring paper-space.</p><h3>The problem</h3><p>Almost every research project in academia or industry involves phases of literature review. I’ve written before about the <a href="https://towardsdatascience.com/finding-and-managing-research-papers-a-survey-of-tools-and-products-9151810d1b4d">pains of finding and managing academic papers</a>.</p><p>Many times we find an interesting paper, and we’d like to:</p><ul><li>Find different methods and approaches to the same subject</li><li>Track down the state of the art research in the field</li><li>Identify seminal works and background reading</li><li>Explore and immerse ourselves in the topic and become aware of the trends and dynamics in the literature</li></ul><p>Previously, the best ways to do this were to browse reference lists, or hope to find good keywords in textual search engines and databases.</p><h3>Introducing Connected Papers</h3><p>Connected Papers started as a side project between friends. We’ve felt the pains of academic literature review and exploration for years and kept thinking about how to solve it.</p><p>For the past year we’ve been meeting on weekends and prototyping a tool that would allow a very different type of search process for academic papers. When we saw how much it improved our own research and development workflows — and got increasingly more requests from friends and colleagues to use it — we committed to release it to the public.</p><p>You know… for science.</p><h4>So how does it work?</h4><p>Connected Papers is not a citation tree. Those have been done before. <br>In our graph, papers are arranged according to their similarity. That means that even papers that do not directly cite each other can be strongly connected and positioned close to each other in the graph.</p><p>To get a bit technical, our similarity is based primarily on the concepts of <em>co-citation</em> and <em>bibliographic coupling</em> (aka co-reference). According to this measure, two papers that have highly overlapping citations and references are presumed to have a higher chance of treating a related subject matter.</p><h4>Reading the graph</h4><p>The Connected Papers graph is designed to make the important and relevant papers pop out immediately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*Hb7qC0321enN7-aVWAcukQ.png" /><figcaption>Graph view</figcaption></figure><p>With our layout algorithm, similar papers cluster together in space and are connected by stronger lines (edges). Popular papers (that are frequently cited) are represented by bigger circles (nodes) and more recent papers are represented by a darker color.</p><p>So for example, finding an important new paper in your field is as easy as identifying the dark large node at the center of a big cluster.</p><h4>List view</h4><p>In some cases it is convenient to work with just a list of connected papers. For these occasions, we’ve built the List view which you can access by clicking “Expand” at the top of the left panel. Here you can view additional paper details as well as sort and filter them according to various properties.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qoIeKAvfbW-CLHFKWN8K_w.png" /><figcaption>List view</figcaption></figure><h4>Prior and derivative works</h4><p>The Prior works feature lists the top common ancestral papers for the connected papers in the graph. It usually includes <strong>seminal works</strong> in the field that heavily influenced the next generation.</p><p>Meanwhile, the Derivative works feature is the opposite: it shows a list of common descendants of the papers in the graph. It usually includes relevant <strong>state of the art</strong> papers or <strong>systematic reviews and meta-analyses in</strong> the field.</p><p>We have found these features to be especially useful when we have a paper from one era of research and we would like to be directed to the preceding and succeeding generations of research on the same topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X7H_7dR8mzH2xnM7uN7FJw.png" /><figcaption>Derivative works</figcaption></figure><h3>Spread the word and join the discussion</h3><p><strong>Please share </strong><a href="https://www.connectedpapers.com/"><strong>Connected Papers</strong></a><strong> in your scientific community!</strong></p><p>We are very eager to see how the broader academic community adopts and responds to this tool. We welcome all forms of feedback and would love to brainstorm together about how it can further evolve and improve.</p><p>You can follow us on <a href="https://twitter.com/ConnectedPapers">twitter </a>or drop us a line at <a href="mailto:hello@connectedpapers.com">hello@connectedpapers.com</a>.</p><p>We’d like to extend a special thank you to all of our beta testers and friends who helped us reach this release.</p><p>— Eddie, Alex, Itay</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89146a54c7d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/connectedpapers/announcing-connected-papers-a-visual-tool-for-researchers-to-find-and-explore-academic-papers-89146a54c7d4">Connected Papers — a visual tool for researchers to find and explore academic papers</a> was originally published in <a href="https://medium.com/connectedpapers">Connected Papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finding and managing research papers: a survey of tools and products]]></title>
            <link>https://medium.com/data-science/finding-and-managing-research-papers-a-survey-of-tools-and-products-9151810d1b4d?source=rss-2d85974b345f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9151810d1b4d</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[arxiv]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[research]]></category>
            <dc:creator><![CDATA[Eddie Smolyansky]]></dc:creator>
            <pubDate>Mon, 12 Nov 2018 11:04:38 GMT</pubDate>
            <atom:updated>2020-06-02T14:10:23.757Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>At the end of this post you can find a tl;dr with my suggestions for the most useful tools to improve your workflow with scientific papers.</em></strong></p><p><strong>Major update (2020): <br>We have released a tool for visually finding and exploring academic papers. See our </strong><a href="https://medium.com/connectedpapers/announcing-connected-papers-a-visual-tool-for-researchers-to-find-and-explore-academic-papers-89146a54c7d4?sk=eb6c686826e03958504008fedeffea18"><strong>launching blog post</strong></a><strong> for </strong><a href="https://www.connectedpapers.com/"><strong>Connected Papers</strong></a><strong>!</strong></p><p>As researchers, especially in (overly) prolific fields like Deep Learning, we often find ourselves overwhelmed by the huge amount of papers to read and keep track of in our work. I think one big reason for this is insufficient use of existing tools and services that aim to make our life easier. Another reason is the lack of a really <strong>good</strong> product which meets all our needs under one interface, but that is a topic for another post.</p><p>Lately I’ve been getting into a new subfield of ML and got extremely frustrated with the process of prioritizing, reading and managing the relevant papers… I ended up looking for tools to help me deal with this overload and want to share with you the products and services that I’ve found. The goal is to improve the workflow and quality of life of anyone who works with scientific papers.</p><p>I will focus mainly on consumption of papers (as opposed to writing) and cover:</p><ol><li>Reference Managers (AKA paper library)</li><li>Social platforms to share knowledge</li><li>Automatic paper analysis to gain additional metadata (keywords, relevant datasets, important citations…)</li></ol><h3>Reference Managers (AKA paper library)</h3><p>These are platforms where you can <strong>create and organize lists of all your past and future reading, add personal notes and share with a small group</strong>. The libraries are synced to the cloud which means your papers should be available anywhere. Think <a href="https://www.goodreads.com/">goodreads</a>, but for papers. Choose one of the following:</p><ol><li><a href="https://www.mendeley.com/">Mendeley</a>: It’s not the best looking product, but it has a freemium business model and supports multiple platforms including web, PC, Mac, and mobile. In addition to general paper notes you can annotate and highlight the PDFs directly. You pay for additional cloud storage (necessary after a few hundred papers).</li><li><a href="https://paperpile.com">Paperpile</a>: paid subscription (no free version), but looks and feels modern. Very easy to import your library there from other services. The library is synced to your own Google Drive, which is a plus. At the moment only works on a chrome browser.</li><li><a href="https://www.zotero.org/">Zotero</a>: A freemium and open source implementation where you pay for additional cloud storage. Similar to Mendeley but less versatile.</li></ol><p>There are <a href="https://en.wikipedia.org/wiki/Comparison_of_reference_management_software">more</a> <a href="https://guides.library.pdx.edu/c.php?g=474937&amp;p=3249933">options</a>, but these are the ones I have tried and they are all fine. If I had to choose one it would be Mendeley for its platform versatility and being freemium.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ryq3u3ZhRH6xT84BCWV89A.png" /><figcaption>Mendeley’s Interface</figcaption></figure><h3>ArXiv Enhancers</h3><p>ArXiv has <a href="https://en.wikipedia.org/wiki/ArXiv#History">been around</a> since 1991, and generally changed very little over the last decade, while the publication volume has increased dramatically [1]. It is natural that today we have different requirements and needs from our primary repository of papers. <strong>We want algorithms that perform paper analysis, we want to find code which implements the papers, we want a social layer via which we can share information, and perhaps we don’t want to squint at a double-columned pdf.</strong></p><p>Searching the internet for existing solutions, I found many such tools:</p><h4>Social layers</h4><ol><li><a href="http://www.shortscience.org/">Shortscience</a>: A platform for sharing paper summaries; Currently over 1000 summaries and growing. Works for any paper with a DOI (so, more than arXiv).</li><li><a href="https://openreview.net/">OpenReview</a>: A transparent paper review process which is open to public reviews as well, currently available only for selected conferences such as NIPS and ICLR. In addition to the official reviews, recently many papers there are seeing active conversations with responses from the original authors.</li><li><a href="https://scirate.com/">Scirate</a>: Adds a like (ehh, “scite”) button over a clone of arXiv. Adds a comment section. Mostly inactive.</li></ol><h4>Find code implementation of papers</h4><ol><li><a href="https://paperswithcode.com">Papers With Code</a>: Automatically connects papers to github repositories that implement them and sorts by github stars. There can be multiple, unmerged entries for each paper.</li><li><a href="https://github.com/zziz/pwc">Github pwc</a>: A minimalistic approach which automatically(?) connects papers to only one code implementation, displayed as a simple table.</li><li><a href="http://www.gitxiv.com">GitXiv</a>: Collaboratively curated feed of projects. Each project is conveniently presented as arXiv + Github + Links + Discussion. Unfortunately this project is <a href="https://github.com/samim23/GitXiv">no longer maintained</a>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xLRgZggnQk79zDnHjHd4-g.png" /><figcaption>Some links from <a href="https://github.com/zziz/pwc">Github pwc</a></figcaption></figure><h4>Other</h4><ol><li><a href="http://arxiv-sanity.com">arXiv-sanity</a>: Gives a makeover to arXiv with exposed abstracts, paper previews and very basic social and library features. A valiant attempt at tying many of the ideas above together, built in spare-time by Andrej Karpathy. The ideas are all there, but in my opinion the implementation isn’t good enough to become a go-to tool for researchers, and the project has not been very active in the past year.</li><li><a href="https://www.arxiv-vanity.com/">arXiv-vanity</a> : Renders academic papers from <a href="https://arxiv.org/">arXiv</a> as responsive web pages so you don’t have to squint at a PDF.</li></ol><h3>Paper search and analysis</h3><ol><li><a href="https://scholar.google.co.il/">Google scholar</a>: Today’s go-to place to search for papers, view paper statistics as well as citations and references, set up alerts for new papers by following an author or a paper, and keep a basic library with automatic recommendations.</li><li><a href="https://dimsum.eu-gb.containers.appdomain.cloud/">IBM Science Summarizer</a>: Summaries are generated by analyzing the content of papers, as well as their structure, sections, paragraphs, and key terms. It doesn’t always work well, but it’s continually improving and is great for skimming papers quickly.</li><li><a href="https://www.semanticscholar.org/">Semantic scholar</a>: Semantic analysis of papers with external material aggregation. Features include: expose citations and references and measure their impact, show paper figures, automatically generate keywords (topics), analyze authors, find additional resources on the internet (e.g. related youtube videos) and suggest recommended papers. <br>A great new effort supported by <a href="https://allenai.org/">AI2</a>. Lately they’ve done a small integration with Paperswithcode mentioned above and with arXiv itself (!).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QHHuPQGuKodxkN92m5POeg.png" /><figcaption>Semantic Scholar: author profile page</figcaption></figure><h3>Tools for authors</h3><ol><li><a href="https://www.overleaf.com/">Overleaf</a>: Collaborative, online LaTeX editor. Think Google docs for writing papers. Very well implemented.</li><li><a href="https://www.authorea.com">Authorea</a>: A 21st century approach to collaboratively writing papers online, aiming to mostly drop LaTeX in favor of a modern WYSIWYG editor. Supports inline code and data for reproducibility, inline public comments, and other features that make perfect sense.</li><li><a href="https://codeocean.com/">Code ocean</a>: A cloud-based computational reproducibility platform. My understanding is that you upload your research as a Jupyter environment code, run it online and reproduce the same graphs /output that the authors get. Here’s an <a href="https://codeocean.com/2018/05/18/deep-supervised-learning-for-hyperspectral-data-classification-through-convolutional-neural-networks/metadata">example</a> (press Run at the top right).</li></ol><h3><strong><em>Tl;dr — my recommendations</em></strong></h3><ul><li><strong>Manage your reading library:</strong> <a href="https://mendeley.com">Mendeley</a></li><li><strong>Read and write paper reviews:</strong> <a href="http://shortscience.org">shortscience</a> and <a href="https://openreview.net">openreview</a></li><li><strong>Match papers to github repositories:</strong> <a href="https://paperswithcode.com">paperswithcode</a> and <a href="https://github.com/zziz/pwc">pwc</a></li><li><strong>Paper and author analysis:</strong> <a href="https://semanticscholar.org">Semantic scholar</a></li><li><strong>Write papers:</strong> <a href="https://www.overleaf.com/">Overleaf</a></li></ul><p>I hope this post has introduced you to at least one service that will improve your workflow.<br>Please, if you know any helpful tools which haven’t been mentioned in this post, share them below for everyone’s benefit.</p><p>[1] By October 2016 the submission rate had grown to more than 10,000 per month. <a href="https://en.wikipedia.org/wiki/ArXiv">https://en.wikipedia.org/wiki/ArXiv</a>,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/0*kbyDBaypvjdmg2Zt.png" /><figcaption>arXiv submission by Topic, from their <a href="https://arxiv.org/help/stats/2017_by_area/index">statistics page</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9151810d1b4d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/finding-and-managing-research-papers-a-survey-of-tools-and-products-9151810d1b4d">Finding and managing research papers: a survey of tools and products</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Meta-analysis of DAVIS-2017 Video Object Segmentation Challenge]]></title>
            <link>https://medium.com/@eddiesmo/a-meta-analysis-of-davis-2017-video-object-segmentation-challenge-c438790b3b56?source=rss-2d85974b345f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c438790b3b56</guid>
            <category><![CDATA[analysis]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Eddie Smolyansky]]></dc:creator>
            <pubDate>Tue, 12 Sep 2017 14:54:55 GMT</pubDate>
            <atom:updated>2017-09-17T14:46:05.220Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*8eIcVM-M506P4bA0Y7pbag.png" /><figcaption>DAVIS-2017: a multi-instance Video Object Segmentation challenge</figcaption></figure><p>In the previous post: <a href="https://medium.com/@eddiesmo/video-object-segmentation-the-basics-758e77321914">Video Object Segmentation — The Basics</a>, we’ve gone through the problem definition of Video Object Segmentation, its metrics and nuances. Then we’ve covered the two main approaches that emerged to deal with the DAVIS-2016 video object segmentation dataset: <em>MaskTrack</em> and <em>OSVOS</em>. In this post we’ll see how these algorithms evolved to handle the more challenging DAVIS-2017 dataset.</p><p>In terms of accuracy, there was a significant leap in performance in 2017. For reference: OSVOS, the state of the art for 2016, got a Region Similarity score of ~46 on the 2017 challenge (with our best implementation) while this year’s winner achieved the impressive score of 67.9!</p><p>Breaking down the top 9 published works in the DAVIS-2017 challenge</p><p>Out of <strong>22 participating teams</strong>, here are the <a href="http://davischallenge.org/challenge2017/publications.html">top 9 with published results</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4MwgTRZjmd9Ueh_tW89rtg.png" /><figcaption>Breaking down the top 9 published works in the DAVIS-2017 challenge</figcaption></figure><p>Looking at the table, we can already see some trends forming in 2017:</p><ul><li>All of the leading works are based on either MaskTrack or OSVOS.</li><li>On the DAVIS-2017 challenge, MaskTrack has won.</li><li>Lucid Data Dreaming augmentations are becoming popular (read more below).</li><li>About half of the works have upgraded their base network to RESNET.</li><li>Almost everyone used some form of temporal component, leveraging the tendency of consecutive video frames to be similar.</li><li>About half of the works made use of a semantic component, employing semantic segmentation or detection (bounding box) networks in their solution.</li></ul><p>Before jumping to any further conclusions, let’s take a closer look at some of the leading works in the following sections:</p><h4>Lucid Data Dreaming for Object Tracking</h4><p>An important work from the authors of the original MaskTrack, <a href="https://arxiv.org/abs/1703.09554">Lucid Data Dreaming</a> aims to “change the mindset regarding how many training samples and general “objectness” knowledge is required to approach this problem”.</p><ul><li>The authors generate “in-domain” training data from DAVIS-2017 dataset and the first frame annotation for each video. For the per-video fine-tuning, they synthesize 2500 augmentations from the single annotated frame, which represent plausible future video frames.</li><li>They achieve these extreme augmentations by “cutting-out the foreground object, in-painting the background, perturbing both foreground and background, and finally recomposing the scene. This process is applied twice with randomly sampled transform parameters, resulting in a pair of frames (Iτ−1, Iτ ) with ground-truth pixel-level mask annotations (Mτ−1, Mτ ), optical flow Fτ , and occlusion regions, as the undergoing transformations are known.”</li><li>On DAVIS-2016, they show that forgoing pretraining on imagenet hinders their results by a mere 2~5%, so using ONLY the first frame annotation as training data actually achieves competitive results! I for one was surprised by this result.</li><li>They use the DeepLabV2 architecture, tweaked to a process a bigger amount of input channels: [3(rgb)+n(amount of instance masks)+1(optical flow)+1(semantic segmentation)], trained for each video separately.</li><li>Another benefit of lucid dreaming is the ability to fine-tune the postprocessing CRF per video.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tM9Hix2XW_5NLHAKrajuJA.png" /><figcaption>Lucid Data Dreaming pipeline</figcaption></figure><p>Note: since some of the other DAVIS contestants, including the #1 winners cite Lucid Dreaming, in my opinion this work deserves the honorary first place.</p><h4>Video Object Segmentation with Re-identification</h4><p><a href="https://arxiv.org/abs/1708.00197">The winning work</a> of the DAVIS-2017 challenge based itself on MaskTrack and Lucid Dream for short-term object tracking, while adding a re-identification process that recovers lost instances in the long-term.</p><ul><li>They’ve made improvements to the MaskTrack method:<br>- The base net is now RESNET.<br>- Instead of feeding the entire image to the mask propagation network, they are only giving it a cropped, size-normalized patch corresponding to the object’s bounding box, thus improving the tracking performance for small objects.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9N9hoeqkWxoP20DZbRe_bg.png" /><figcaption>The improved MaskTrack architecture</figcaption></figure><ul><li>Re-identification process. For every frame:<br>- Detect candidate bounding boxes with Faster-RCNN, then compare them to the known instances to check if they should be “recovered” using [<a href="http://openaccess.thecvf.com/content_cvpr_2017/papers/Xiao_Joint_Detection_and_CVPR_2017_paper.pdf">Joint detection and identification feature learning for person search</a>], retrained for the general object case.<br>- The recovered instance is then forward and back propagated into the past and future video frames.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uGu4BZnLoB79p4cD56-Xmw.png" /><figcaption>The re-identification process, with forward and backward propagation</figcaption></figure><h4>Multiple-Instance Video Segmentation with Sequence-Specific Object Proposals</h4><p>This was the highest ranked team based on the OSVOS method. <a href="http://davischallenge.org/challenge2017/papers/DAVIS-Challenge-4th-Team.pdf">Their idea</a> was to generate “object proposals” from various sources and then combine them in a post-processing step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*XOUo0jtrATfT5s8eipW0uw.png" /></figure><ul><li>An improved OSVOS is used to find “object proposals”.<br>- Added a small improvement to the balanced-cross-entropy loss function to better handle convergence for small objects (which appear in the DAVIS-2017 dataset)<br>- The online training is performed on Lucid-Dreaming-style augmentations of the first annotated frame</li><li>Additional (semantic) proposals are acquired from an instance-aware image segmentation algorithm</li><li>Makes use of “combinatorial grouping” to find connected components of the suggested proposals given by OSVOS and decide which ones to keep or discard according to distance-based criteria.</li><li>Performs tracking and deals with occlusions using a Semantic Proposal Tracker which is based on <a href="https://web.engr.oregonstate.edu/~lif/SegTrack2/segtrack2_cameraready.pdf">these</a> <a href="https://web.engr.oregonstate.edu/~lif/painless_occlusion.pdf">two</a> papers.</li></ul><h4>Online Adaptation of Convolutional Neural Networks for the 2017 DAVIS Challenge</h4><p>The only team (so far) to release their code, <a href="https://arxiv.org/pdf/1706.09364.pdf">onAVOS</a> took the online fine-tuning idea to the next level by segmenting future frames with a model fine-tuned on previously segmented frames.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wwog6Xfs1jPdsmfi_pSXyA.png" /><figcaption>onAVOS training pipeline</figcaption></figure><ul><li>They switch the base net to RESNET (based on <a href="https://arxiv.org/abs/1611.10080">Wider or Deeper</a>), pre-trained on PASCAL-VOC</li><li>For the online training, they generate augmentation of the first frame in a Lucid Data Dreaming style.</li><li>As each frame is segmented, foreground pixels with high confidence predictions are taken as further positive training examples, while pixels far away from the last assumed object position are taken as negative examples. Then an additional round of fine-tuning is performed on the newly acquired data.</li><li>In other words, the second frame is segmented by fine-tuning on the first frame annotation, then the third frame is segmented by training on the second frame’s prediction as well and the segmentation model keeps being updated with every new frame.</li><li>The final transfer learning chain is long and looks like this: <br>base net (imagenet) → objectness net (pascal voc) → domain specific (DAVIS) → test net (video) → online adapted test net (rolling fine-tune on high confidence pixels)</li></ul><p>Looking at the code, the authors seem to have paid a lot of attention to implementation details such as supporting low memory GPUs and providing multiple running configurations.</p><h3>Conclusions</h3><p>Going over the works of the 2017 video segmentation challenge, some common strategies have emerged:</p><ul><li>Combine MaskTrack, which is good at tracking objects in the short term but can sometimes lose the object of interest, with a re-identification mechanism that restarts the tracking.</li><li>Combine OSVOS, which is good at segmenting the object of interest but has difficulty handling extreme appearance changes, with a rolling fine-tune mechanism that updates the underlying model as the video progresses.</li><li>Upgrade OSVOS with the addition of an instance separation mechanism, such as a semantic instance segmentation network.</li><li>Upgrade to a modern base network (e.g. Resnet) and compensate for the small amount of data with extreme in-domain augmentations (e.g. Lucid Data Dreaming).</li></ul><h4>What’s next?</h4><p>This year we have seen two main approaches being expanded upon and improved. <strong>Next year, will we see completely new architectures?</strong></p><p>One promising direction could be the use of <strong>memory modules</strong>: unlike other video tasks, segmentation is a dense problem and thus the method of using a CNN to extract sparse features from each frame and then feed them as input to a separate RNN network does not work well. We have seen <a href="https://arxiv.org/abs/1611.05435">some</a> <a href="https://lear.inrialpes.fr/people/alahari/papers/tokmakov17a.pdf">interesting</a> <a href="https://arxiv.org/abs/1612.08871">papers</a> which suggest convolutional memory modules for unsupervised video segmentation. Will we see them applied to the semi-supervised case as well?</p><p>While video object segmentation has come a long way in the last couple of years, it is still far from “solved”. For the technology to become useful in real-life scenarios, great improvements need to be made in accuracy, generality and most importantly — in <strong>run-time</strong> <strong>speed</strong>. Several of the leading solutions of the 2017 competition can take hours to segment a single 5 second video, and none claim to do it fast.</p><p>We hope that our release of the more simple and focused <a href="https://github.com/ilchemla/gygo-dataset"><strong>GyGO dataset</strong></a><strong> </strong>will encourage attempts at more efficient, perhaps even <strong>real-time</strong> solutions.</p><p>See you at next year’s challenge!</p><p><a href="https://medium.com/@eddiesmo/video-object-segmentation-the-basics-758e77321914">Go to part 1: Video Object Segmentation — The Basics…</a></p><h3>References</h3><p>The main papers described and analyzed in this post are cited below.</p><ol><li><strong>Lucid data dreaming for object tracking.<br></strong>A. Khoreva, R. Benenson, E. Ilg, T. Brox, and B. Schiele.<br>arXiv preprint, arXiv:1703.09554, 2017Papers 1–8</li><li><strong>Video Object Segmentation with Re-identification<br></strong>X. Li, Y. Qi, Z. Wang, K. Chen, Z. Liu, J. Shi, P. Luo, C. Change Loy, X. Tang, <em>The 2017 DAVIS Challenge on Video Object Segmentation — CVPR Workshops, 2017</em></li><li><strong>Instance Re-Identification Flow for Video Object Segmentation<br></strong>T.-N. Le, K.-T. Nguyen, M.-H. Nguyen-Phan, T.-V. Ton, T.-A. Nguyen (2), X.-S. Trinh, Q.-H. Dinh, V.-T. Nguyen, A.-D. Duong, A. Sugimoto, T. V. Nguyen, M.-T. Tran, <em>The 2017 DAVIS Challenge on Video Object Segmentation — CVPR Workshops, 2017</em></li><li><strong>Online Adaptation of Convolutional Neural Networks for Video Object Segmentation</strong><br>Paul Voigtlaender and Bastian Leibe: BMVC 2017</li><li>The rest of the DAVIS-2017 contestant papers can be found here:<br><a href="http://davischallenge.org/challenge2017/publications.html">http://davischallenge.org/challenge2017/publications.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c438790b3b56" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Basics of Video Object Segmentation]]></title>
            <link>https://medium.com/@eddiesmo/video-object-segmentation-the-basics-758e77321914?source=rss-2d85974b345f------2</link>
            <guid isPermaLink="false">https://medium.com/p/758e77321914</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[introduction]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[dataset]]></category>
            <dc:creator><![CDATA[Eddie Smolyansky]]></dc:creator>
            <pubDate>Tue, 12 Sep 2017 14:51:47 GMT</pubDate>
            <atom:updated>2018-06-13T05:48:03.837Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ucVBQWZ2mNgw-OcVilJlcA.png" /><figcaption>Annotated ground truth frames from the DAVIS-2016 video object segmentation dataset</figcaption></figure><p>[Update: the article has also been translated to <a href="https://www.jiqizhixin.com/articles/2017-10-06">Chinese</a>.]</p><p>This is the first in a <a href="https://medium.com/@eddiesmo/a-meta-analysis-of-davis-2017-video-object-segmentation-challenge-c438790b3b56">two part</a> series about the state of the art in algorithms for Video Object Segmentation. The first part will be an introduction to the problem and it’s “classic” solutions. We will briefly cover:</p><ol><li>The problem, the datasets, the challenge</li><li>A new dataset that we’re announcing today!</li><li>The two main approaches from 2016: MaskTrack and OSVOS. These are the algorithms upon which all other works are based.</li></ol><p>In the <a href="https://medium.com/@eddiesmo/a-meta-analysis-of-davis-2017-video-object-segmentation-challenge-c438790b3b56">second part</a>, which is more advanced, I will present a comparison table of all the published approaches to the DAVIS-2017 Video Object Segmentation challenge, summarize and highlight selected works and point to some emerging trends and directions.</p><p>The posts assume familiarity with some concepts in computer vision and deep learning, but are quite accessible. I hope to make a good introduction to this computer vision challenge and bring newcomers quickly up to speed.</p><h3>Introduction</h3><p>There are three classic tasks related to objects in computer vision: classification, detection and segmentation. While classification aims to answer the “what?”, the goal of the latter two is to also answer the “where?”, and segmentation specifically aims to do it at the pixel level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TwcMmXXuumsDRvgaY2OCQA.png" /><figcaption>Classical computer vision tasks (image from Stanford’s cs231n course slides)</figcaption></figure><p>In 2016 we have seen semantic segmentation mature and perhaps even begin to saturate existing datasets. Meanwhile, 2017 has been somewhat of a breakout year for video related tasks: action classification, action (temporal) segmentation, semantic segmentation, etc. In these posts we will focus on Video Object Segmentation.</p><h3>The problem, the datasets, the challenge</h3><p>Assuming reader familiarity with <em>semantic segmentation</em>, the task of <em>video object segmentation</em> basically introduces two differences:</p><ul><li>We are segmenting <em>general, NON-semantic</em> objects.</li><li>A temporal component has been added: our task is to find the pixels corresponding to the object(s) of interest in each consecutive frame of a video.</li></ul><p><strong>This can also be thought of as a pixel-level object tracking problem.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*glryPZvHJzGa8SDXZ4YuXA.png" /><figcaption>Segmentation: sub-divisions of the space. An example dataset is given for each leaf in the graph.</figcaption></figure><p>In the video formulation, we can split the problem into two subcategories:</p><ul><li>Unsupervised (aka video saliency detection): The task is to find and segment the main object in the video. This means the algorithm should decide by itself what the “main” object is.</li><li><strong>Semi-supervised, which we’ll cover in these posts</strong>: given the ground truth segmentation mask of (only) the first frame as input, segment the annotated object in every consecutive frame.</li></ul><p>The semi-supervised case can be extended to multiple object segmentation as can be seen in the <a href="http://davischallenge.org/challenge2017/index.html">DAVIS-2017 challenge</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qooRKoB2wPNKvGs-C_MI-A.png" /><figcaption>The main difference between annotations of DAVIS-2016 (left) and DAVIS-2017 (right): multi-instance segmentation</figcaption></figure><p>→ <a href="http://davischallenge.org/images/DAVIS-2017-TrainVal.mp4">Press here to view a video of the DAVIS-2017 dataset and challenge</a> ←</p><p>As you can see, DAVIS is a dataset with pixel-perfect ground truth annotations. It aims to recreate real-life video scenarios such as camera shake, background clutter, occlusions and other complexities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*y5IuhAwd4Elznm9JRT9YVg.png" /><figcaption>DAVIS-2016 complexity attributes</figcaption></figure><p>There are two primary <strong>metrics to measure segmentation</strong> success:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/1*pz2lQMS4d5L--u1_8Lx0QQ.png" /><figcaption>Region Similarity is the intersection-over-union between mask M and ground truth G</figcaption></figure><ul><li><em>Region Similarity:</em> defined as the intersection-over-union between the estimated segmentation and the ground truth mask.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/1*od0EOZzVKjfa_V9nmPEVTA.png" /><figcaption>Contour Accuracy is the F-measure for the contour based precision and recall</figcaption></figure><ul><li><em>Contour Accuracy: </em>interprets the masks as a set of closed contours and computes the contour-based F-measure which is a function of precision and recall.</li></ul><p>Intuitively — <em>Region Similarity</em> measures the amount of mislabeled pixels, while <em>Contour Accuracy </em>measures the precision of the segmentation boundaries.</p><h3>Announcing a new dataset! GyGO: E-commerce Video Object Segmentation by Visualead</h3><p>GyGO, which we will release in parts over the following weeks, is a dataset that is focused on a specific, simple use case for video object segmentation: E-commerce. It will consist of about 150 short videos.</p><p><a href="https://github.com/ilchemla/gygo-dataset">https://github.com/ilchemla/gygo-dataset</a></p><p>On the one hand, the sequences are quite simple in that they are virtually devoid of occlusions, fast motions or many of the other complexity inducing attributes mentioned above. On the other hand, the objects in these videos have more varied categories than the DAVIS-2016 dataset, in which many of the sequences contain known semantic classes (humans, cars, etc). The GyGO dataset specializes in smartphone captured videos and its frames are relatively sparse (the annotated video speed is ~5 fps).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F4RQff9bfJsk%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D4RQff9bfJsk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F4RQff9bfJsk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c75efa2c96955d46a51937b37c4b58da/href">https://medium.com/media/c75efa2c96955d46a51937b37c4b58da/href</a></iframe><p>We release it publicly with two goals in mind:</p><ol><li>There is a severe lack of data in the space of video object segmentation at the moment. With only hundreds of annotated videos, we believe every contribution has the potential to increase performance. In our analysis we have shown that a joint training on the GyGO and DAVIS datasets yields improved inference results.</li><li>To promote a more open, sharing culture and encourage other researchers to join our efforts :) The DAVIS dataset and the research ecosystem that grew it have been massively useful for us. We hope the community will find our datasets useful as well.</li></ol><h3>The two main approaches to DAVIS-2016</h3><p>With the release of the DAVIS-2016 dataset for single object video segmentation, two main leading approaches emerged: MaskTrack and OSVOS. Looking at the contestants of the DAVIS-2017 challenge, it seems that every single team decided to build their solution on top of one of these two approaches — giving them the status of instant classics. Lets see how they work:</p><h4><strong>One Shot Video Object Segmentation</strong></h4><p>The concept behind OSVOS is simple yet powerful:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9EIR2BoEEJDs22sVGCTOpA.png" /><figcaption>OSVOS training pipeline</figcaption></figure><ol><li>Take a net (say VGG-16) pre-trained for classification for example, on imagenet.</li><li>Convert it to a fully convolutional network, à la <a href="https://arxiv.org/abs/1605.06211">FCN</a>, thus preserving spatial information:<br>- Remove the FC layers in the end.<br>- Insert a new loss: pixel-wise sigmoid balanced cross entropy (previously used by <a href="https://arxiv.org/abs/1504.06375">HED</a>). Now each pixel is separately classified into foreground or background.</li><li>Train the new fully convolutional network on the DAVIS-2016 training set.</li><li><strong>One-shot training: <br></strong>At inference time, given a new input video for segmentation and a ground-truth annotation for the first frame (remember, this is a semi-supervised problem), create a new model, initialized with the weights trained in [3] and fine-tuned on the first frame.</li></ol><p>The result of this process is a unique, one-time-use model for every new video that is overfitted for that specific video according to the first frame annotation. Because for most videos the appearance of the object and the background does not change drastically, this model produces good results. Naturally, this model would work less well for another random video sequence.</p><p>Note: the OSVOS method segments the frames independently — there is no use of temporal information in the video.</p><h4><strong>MaskTrack (Learning Video Object Segmentation from Static Images)</strong></h4><p>While OSVOS works on each frame of the video independently, MaskTrack also takes into consideration the temporal information contained within it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8nJO91nrPKdjbMg7E9hSCQ.png" /><figcaption>MaskTrack Mask propagation module</figcaption></figure><ol><li>For each frame, feed the predicted mask of the previous frame as additional input to the network: the input is now 4 channels (RGB+previous mask). Initialize this process with the ground truth annotation given for the first frame.</li><li>The net, originally based on DeepLab VGG-16 (but modular), is trained from scratch on a combination of semantic segmentation and image saliency datasets. The input for the previous mask channel is artificially synthesized by small transformations of the ground truth annotation of each still image.</li><li>Add an identical second stream network, based on optical flow input. The model weights are the same as in the RGB stream. Fuse the outputs of both streams by averaging the results.</li><li>Online training: Use the first frame ground truth annotation to synthesize additional, video specific training data.</li></ol><p>Note: Both of these methods rely on still-images training (as opposed to video, where the datasets are scarce and small).</p><p>To summarize, in this introductory post we’ve established the problem of Video Object Segmentation and the leading methods to solve it in 2016. Armed with this knowledge, we are ready to tackle the twice improved, state of the art algorithms proposed in 2017.</p><p><a href="https://medium.com/@eddiesmo/a-meta-analysis-of-davis-2017-video-object-segmentation-challenge-c438790b3b56">Go on to part 2: A Meta-Analysis of DAVIS-2017 Video Object Segmentation Challenge…</a></p><p>P.S. I’d like to say thank you to the wonderful <a href="http://davischallenge.org/challenge2017/index.html">team behind the DAVIS datasets and challenge</a> for all their hard work. Without you none of this would exist.</p><h3>References</h3><p>The main papers described and analyzed in this post are cited below.</p><ol><li><strong>Benchmark Dataset and Evaluation Methodology for Video Object Segmentation<br></strong>F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung, <em>Computer Vision and Pattern Recognition (CVPR) 2016</em></li><li><strong>The 2017 DAVIS Challenge on Video Object Segmentation<br></strong>J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. Van Gool, <em>arXiv:1704.00675, 2017</em></li><li><strong>Learning Video Object Segmentation from Static Images<br></strong>F. Perazzi*, A. Khoreva*, R. Benenson, B. Schiele, A. Sorkine-Hornung<br><em>CVPR 2017, Honolulu, USA</em>.</li><li><strong>One-Shot Video Object Segmentation, <br></strong>S. Caelles*, K.K. Maninis*, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool, <em>Computer Vision and Pattern Recognition (CVPR)</em>, 2017</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=758e77321914" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>