<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eli Schwartz on Medium]]></title>
        <description><![CDATA[Stories by Eli Schwartz on Medium]]></description>
        <link>https://medium.com/@elischwartz?source=rss-23022de21ddd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*2MD2VSNaSCf1XPXs27fRzQ.jpeg</url>
            <title>Stories by Eli Schwartz on Medium</title>
            <link>https://medium.com/@elischwartz?source=rss-23022de21ddd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:20:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@elischwartz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Few-Shot Learning in CVPR19]]></title>
            <link>https://medium.com/data-science/few-shot-learning-in-cvpr19-6c6892fc8c5?source=rss-23022de21ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c6892fc8c5</guid>
            <category><![CDATA[cvpr-2019]]></category>
            <category><![CDATA[few-shot-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[meta-learning]]></category>
            <category><![CDATA[cvpr]]></category>
            <dc:creator><![CDATA[Eli Schwartz]]></dc:creator>
            <pubDate>Thu, 11 Jul 2019 17:34:18 GMT</pubDate>
            <atom:updated>2019-08-01T14:17:35.215Z</atom:updated>
            <content:encoded><![CDATA[<h3>Few-Shot Learning in CVPR 2019</h3><p>TL;DR for all(?) few-shot papers from CVPR 2019. These are notes I originally wrote for myself, so take them with a grain of salt. Some of the papers I had the privilege of discussing with the authors at the poster sessions, some of them I read, and some of them I only skimmed through. Please let me know if I got something wrong.</p><h3>Background</h3><p>Few-shot object recognition became a hot topic recently (from 4 few-shot papers in CVPR18 to around 20 in CVPR19). The usual setup is that you have categories with many examples you can use at training time; then at test time, you are given novel categories (usually 5) with only a few examples per category (usually 1 or 5; called “support-set”) and query images from the same categories.</p><p>Next, I’ll try to break the few-shot methods to different families. Although these families are not well defined and many methods belong to more than one family.</p><p>“Older” suggested methods were based on <strong>metric learning</strong>, where the objective is to learn a mapping from images to embedding space in which images from the same category are closed together and from different categories are far apart. Hoping that it will hold true for unseen categories.</p><p>Following that, came <strong>meta-learning</strong> methods. These are models which are conditioned on the current task, so a different classifier is used as a function of the support-set. The idea is to find model hyper-parameters and parameters such that it will be easy to adapt to a new task without over-fitting to the few shots available.</p><p>In parallel, <strong>data augmentation</strong> methods are also very popular. The idea is to learn data augmentation so we can generate more examples out of the few examples available.</p><p>Finally, <strong>semantics-based methods</strong> are on the rise. It is inspired by zero-shot learning where classification is done based solely on the category name, textual descriptions, or attributes. Those extra semantic ques can also be of help when visual examples are scarce.</p><h3>Papers covered in this post</h3><h4>Metric learning methods</h4><p><a href="https://arxiv.org/abs/1903.12290"><strong>Revisiting Local Descriptor based Image-to-Class Measure for Few-shot Learning</strong></a>, Li et. al; <a href="https://arxiv.org/abs/1904.08502"><strong>Few-Shot Learning with Localization in Realistic Settings</strong></a>, Wertheimer et. al; <a href="https://arxiv.org/abs/1903.05050"><strong>Dense Classification and Implanting for Few-Shot Learning</strong></a>, Lifchitz et. al; <a href="https://arxiv.org/abs/1904.08482"><strong>Variational Prototyping-Encoder: One-Shot Learning with Prototypical Images</strong></a>, Kim et. al.</p><h4>Meta-learning methods</h4><p><a href="https://arxiv.org/abs/1904.03758"><strong>Meta-Learning with Differentiable Convex Optimization</strong></a><strong>, </strong>Lee et. al; <a href="https://arxiv.org/abs/1905.01436"><strong>Edge-Labeling Graph Neural Network for Few-shot Learning</strong></a>, Kim et. al; <a href="https://arxiv.org/abs/1805.07722"><strong>Task Agnostic Meta-Learning for Few-Shot Learning</strong></a>, Jamal et. al; <a href="http://arxiv.org/abs/1812.02391"><strong>Meta-Transfer Learning for Few-Shot Learning</strong></a>, Sun et. al; <a href="http://arxiv.org/abs/1905.01102"><strong>Generating Classification Weights with GNN Denoising Autoencoders for Few-Shot Learning</strong></a>, Gidaris et. al; <a href="https://arxiv.org/abs/1905.11116"><strong>Finding Task-Relevant Features for Few-Shot Learning by Category Traversal</strong></a>, Li et. al</p><h4>Data Augmentation Methods</h4><p><a href="https://arxiv.org/abs/1902.09811"><strong>LaSO: Label-Set Operations networks for multi-label few-shot learning</strong></a>, Alfassy et. al; <a href="https://arxiv.org/abs/1904.03472"><strong>Few-shot Learning via Saliency-guided Hallucination of Samples</strong></a>, Zhang et. al; <a href="http://openaccess.thecvf.com/content_CVPR_2019/papers/Chu_Spot_and_Learn_A_Maximum-Entropy_Patch_Sampler_for_Few-Shot_Image_CVPR_2019_paper.pdf"><strong>Spot and Learn: A Maximum-Entropy Patch Sampler for Few-Shot Image Classification</strong></a>, Chu et. al; <a href="https://arxiv.org/abs/1905.11641"><strong>Image Deformation Meta-Networks for One-Shot Learning</strong></a>, Chen et. al</p><h4>Semantics-based Methods</h4><p><a href="https://arxiv.org/abs/1906.01905"><strong>Baby steps towards few-shot learning with multiple semantics</strong></a>, Schwartz et. al; <a href="https://arxiv.org/abs/1812.01784"><strong>Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders</strong></a>, Schonfeld et. al; <a href="https://arxiv.org/abs/1904.05967"><strong>TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning</strong></a>, Wang et. al; <a href="http://openaccess.thecvf.com/content_CVPR_2019/papers/Li_Large-Scale_Few-Shot_Learning_Knowledge_Transfer_With_Class_Hierarchy_CVPR_2019_paper.pdf"><strong>Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy</strong></a>, Li et. al</p><h4>Beyond Recognition (Few-shot learning for other tasks)</h4><p><a href="https://arxiv.org/abs/1806.04728"><strong>RepMet: Representative-based metric learning for classification and few-shot object detection</strong></a>, Karlinsky et. al; <a href="https://arxiv.org/abs/1903.02351"><strong>CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning</strong></a>, Zhang et. al</p><h3>Performance on the mini-ImageNet benchmark</h3><p>Only papers where it’s relevant and reported; Sorted according to 1-shot performance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FGUEPggA5n0b4I49M4Qg-w.png" /></figure><h3>Metric Learning Methods</h3><p><strong>Revisiting Local Descriptor based Image-to-Class Measure for Few-shot Learning, </strong>Li et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*VK-AymZidKAR1oUcYJ2zPA.png" /></figure><p>In this work, the authors felt a bit nostalgic and went back to local descriptors from the bag-of-words days, but features are extracted with a CNN and everything is learned end-to-end. Results are on the lower side of the benchmarks.</p><p><strong>Few-Shot Learning with Localization in Realistic Settings</strong>, Wertheimer et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*Iip2Ydfig9_EcM9Tifk6VQ.png" /></figure><p>Rightfully claiming that standard few-shot benchmarks are not realistic due to their artificially balanced classes and testing on 5-way tasks, they suggest a new dataset/benchmark. The model learns simultaneous localization and classification; the obvious drawback is that a dataset with bounding box annotations is required. The classifier is based on prototypical networks but the feature vector used is a concatenation of aggregated foreground and background representations.</p><p><strong>Dense Classification and Implanting for Few-Shot Learning</strong>, Lifchitz et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pfPTdKk6-uPqlUvyuoyOkA.png" /></figure><p>Classification is performed densely, i.e. instead of global average pooling at the end all spatial locations are required to be correctly classified. In addition, instead of fine-tuning the last layer at test time, they widen each layer by adding neurons and finetune them (only the additional weights are trained, the old ones are frozen).</p><p><strong>Variational Prototyping-Encoder: One-Shot Learning with Prototypical Images</strong>, Kim et. al.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Sb1jz8Tui1wyGuYP.png" /></figure><p>This is more of a specific application of one-shot classification dealing with logos or road signs classification. In this case, they treat the graphics image (in contrast logos/signs in-the-wild) as the prototypes. And they learn a good representation by learning a meta-task of mapping the in-the-wild logo/sign images to the prototype images.</p><h3>Meta-Learning Methods</h3><p><strong>Meta-Learning with Differentiable Convex Optimization, </strong>Lee et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w0aPoVflQ-ef-esY8GNiAA.png" /></figure><p>This work is one of the top on the benchmarks (for methods that don’t use semantic information). The method is quite simple, a strong backbone coupled with an SVM classifier trained end-to-end. They show that even just using the backbone with prototypical networks achieves high accuracies. The use of SVM boost performance too. The feature dimensions (classifier input) is kept quite high, I wonder if the SVM can better handle this high dimension and this is the reason for the boost.</p><p><strong>Edge-Labeling Graph Neural Network for Few-shot Learning</strong>, Kim et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kb-B9r_D2CMxO_Hb3djxaw.png" /></figure><p>Graph Neural Network has been used before for few-shot learning. The basic idea is that each image can be presented as a node in a graph and the information (node representation) can be propagated between them according to how similar they are. While usually the classification is done implicitly according to the distance between nodes representation, here the authors suggest having explicit features per edge describing the similarity between its nodes.</p><p><strong>Task Agnostic Meta-Learning for Few-Shot Learning</strong>, Jamal et. al</p><p>In this work, to avoid over-fitting of a meta-learning model to the training tasks a regularization term is added to the output predictions. The regularization either force the predictions to have a higher entropy (i.e. predicted probabilities don’t behave like a one-hot vector) or force the model to have lower inequality between tasks (i.e. performing equally bad or good on different tasks). Obviously, having a strong regularization is important for few-shot, but I don’t have a good intuition why these particular regularization terms are desired. They test this method on top of MAML and show it performs better, would be interesting to see if it helps when using it on top of other methods as well.</p><p><strong>Meta-Transfer Learning for Few-Shot Learning</strong>, Sun et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZAo0cswQxkSNOlZva_B9yw.png" /></figure><p>The method has two main elements: 1. finetuning a pre-trained model where weights are frozen and only a scale and shift per layer are learned 2. hard task mining. If I’m not mistaken in MAML a finetune is applied to the batch norm layers, isn’t it the same effect as learning scale and shift? The hard batch mining (composing a task from classes with the lower accuracy in previous tasks) seems to be beneficial also when applied on top of MAML.</p><p><strong>Generating Classification Weights with GNN Denoising Autoencoders for Few-Shot Learning</strong>, Gidaris et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*BknpNjEXxiQf_xrgQkBVXQ.png" /></figure><p>Here, again, the model is predicting the classifier weights for the unseen classes. In addition, all classifier weights are going through a denoising auto-encoder (both base and the task novel classes) which is implemented as a graph neural network, thus 1. allowing adaptation of the classifiers to the current task classes and 2. propagating knowledge from the base classes classifiers to the novel ones. It makes sense to use denoising auto-encoder since it can help to fix the predicted classifiers which are predicted based on only a few examples and are obviously noisy.</p><p><strong>Finding Task-Relevant Features for Few-Shot Learning by Category Traversal</strong>, Li et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9lKFvjcQq9tHPnx2HNMT5Q.png" /></figure><p>This work is one of the top on the benchmarks (for methods that don’t use semantic information). Given a feature extractor, the model basically predicts an attention map over the feature vector. The “Concentrator” looks at each class (or image? I’m not sure) separately and the “Projector” combines information from all classes in the task to produce the attention map. Both “Concentrator” and “Projector” are implemented as a small CNN. I love that it is a simple module that they show to consistently improve performance when used on top of several known metric-based methods.</p><h3>Data Augmentation Methods</h3><p><strong>LaSO: Label-Set Operations networks for multi-label few-shot learning</strong>, Alfassy et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*uxjPlhZp1nBwkavMoiVuSQ.png" /></figure><p>Shout-out to my colleagues for this cool paper dealing with multi-label few-shot classification. Here, models are trained to perform set-operations (Union, Subtraction, and Intersection) on label sets for multi-label samples, in the embedding space. For instance, by applying a union operation on images of a dog and a cat we get the representation of an image with both a dog and a cat. These operations are then used to augment the data and improve classification performance.</p><p><strong>Few-shot Learning via Saliency-guided Hallucination of Samples</strong>, Zhang et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VX1bSb2Uc9T3t1VhBNlGMw.png" /></figure><p>A pre-trained (on disjoint classes) saliency model is used to segment foreground vs. background. Two feature extractors are trained, one for foreground and for background and a third model is used to combine them. The obvious drawback is that you need a pre-trained saliency model. I think it is a cool way to augment the data, but I’m not sure how important is the augmentation of background for correct classification (unlike detection) and if you are already able to segment the foreground wouldn’t it be better to use just the foreground for classification?</p><p><strong>Spot and Learn: A Maximum-Entropy Patch Sampler for Few-Shot Image Classification, </strong>Chu et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GagdQeBIaNSAQ6LI4qwhdw.png" /></figure><p>Here, instead of learning representation of the whole image, the representation is calculated per patch and aggregated using RNN with the optimal trajectory over patches (i.e. the decision which patch should be used next) predicted with a reinforcement learning model. I guess that compared to a simple attention model it’s better because of the data augmentation (using different trajectories on the same image). However, the results on the benchmarks are just okay. I wonder if augmentation by just using a random trajectory would not work as well.</p><p><strong>Image Deformation Meta-Networks for One-Shot Learning</strong>, Chen et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*IbIx7oW929cuNgeaQ8u1mA.png" /></figure><p>A very cool approach to data augmentation. Similar to “mixup”, but the image is divided according to a grid and for each cell different (predicted) mixup coefficients are used. Training is end-to-end, simultaneously learning the classifier and the optimal mixup (optimal in the sense of better classification). They compared to mixup, but I wonder how good can it work with mixup for the whole image when the coefficient is predicted and not random, i.e. what is the effect of different mix per cell.</p><h3>Semantics-based Methods</h3><p><strong>Baby steps towards few-shot learning with multiple semantics</strong>, Schwartz et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*W46svUig07ICMvy57xG53A.png" /></figure><p>Shameless plug to my own work presented at the Language and Vision Workshop at CVPR. We built on the AM3 model [Xing et. al, 2019] and generalize it to use multiple semantics. We use a short textual description of the categories (these descriptions are part of ImageNet but haven’t been used so far for few-shot) to improve performance. We begin with the visual prototypes and iteratively update them with a sequence of semantic embeddings. By doing so we observe SOTA results on miniImageNet.</p><p><strong>Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders</strong>, Schonfeld et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kZet7BpS6wWE9W6Z3JQe0g.png" /></figure><p>Two VAEs are trained, one for visual features and one for semantic features. The objective is to be able to reconstruct semantic features from the latent of visual features and vice versa. I’d expect it to be enough but they show that forcing the two latent spaces to have the same distribution helps too.</p><p><strong>TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning, </strong>Wang et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5CLwbc1-EmEoEA2sNL_7SA.png" /></figure><p>In this work, the label embeddings (GloVe) are used to predict the weights of the visual feature extractor model. They suggest a nice trick of factorizing the weights so only a lower dimension weight vector needs to be predicted. In addition, alignment between the semantic embedding and visual embedding is forced through the “embedding loss”. This paper is interesting since it combines two approaches — meta-learning (predicting the model based on task) and using semantic information (labels). However, for the few-shot task, it seems the results are lower than SOTA.</p><p><strong>Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy</strong>, Li et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mFRXfh9gHjOngSzthP6I1Q.png" /></figure><p>It’s claimed that current methods fail on large scale few-shot, i.e. pre-training on imagenet and not on as little as the 64 base classes in mini-imagenet, and are not better than naive baselines. Here, again, the semantic labels are used. They used the label embeddings to unsupervisedly build a class hierarchy, which is an interesting approach, and supposedly learning to classify in this hierarchical manner helps the model capture features that perform better on unseen classes. One caveat is that the unseen classes labels are used for building the class hierarchy, isn’t it a little bit cheating?</p><h3>Beyond Recognition (Few-shot learning for other tasks)</h3><p><strong>RepMet: Representative-based metric learning for classification and few-shot object detection</strong>, Karlinsky et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*SDagTCz-keG7Rq4A1h40Wg.png" /></figure><p>Shameless plug, again, to a work I co-authored. We are among the first to investigate few-shot object detection. Our solution is an extension of metric-based methods, such as prototypical networks, to detection. We use an off-the-shelf detector architecture (FPN-DCN) and replace the (linear) classifier head with a metric-based classifier where classification for each proposed region is done based on the distance to learned class representatives. We suggest a new benchmark for few-shot detection and show improvement over competing methods.</p><p><strong>CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning</strong>, Zhang et. al</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*ukkDzSOKzoDwdHMco9JtxQ.png" /></figure><p>Extending metric learning to the dense case for few-shot segmentation. Comparing all local features in a query image to all local features on the objects in the support set is very costly. So they chose to compare the local features in the query to a global representation of the support set images.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c6892fc8c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/few-shot-learning-in-cvpr19-6c6892fc8c5">Few-Shot Learning in CVPR19</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>