In the competitive world of e-commerce, high-quality product images are not just a luxury — they’re a necessity. TryOn Studio emerges as a game-changing solution, offering a comprehensive suite of AI-powered tools designed specifically for fashion and accessories businesses. This article explores how TryOn Studio’s innovative features transform product photography workflows, enabling brands to create professional-quality images at scale while significantly reducing time and costs.

Challenge

In today’s digital-first retail landscape, the quality of your product images directly impacts your conversion rates and brand perception. D2C brands, sellers, and E-commerce companies face significant challenges in maintaining consistent, high-quality product photography across their catalogs. Traditional solutions often involve:

• Expensive photo studio sessions
• Time-consuming manual editing
• Inconsistent image quality
• High operational costs
• Limited scalability

Introducing TryOn Studio

An AI-powered image enhancement suite designed specifically for fashion and accessories businesses, offering a comprehensive solution to these challenges.

Features and Use Cases

Remove Background

• Perfect for creating clean, professional product listings
• Eliminates the need for expensive photo studios and green screens
• Ideal for creating consistent product catalogs across multiple platforms
• Saves hours of manual editing time
• Creates transparent backgrounds for flexible product placement
• Enables quick adaptation of images for different marketing channels

Upscale Image by 4x

• Transform low-resolution product photos into high-quality images
• Perfect for repurposing existing product photos
• Maintains image quality when zooming in on product details
• Ideal for creating large format displays and banners
• Enhances product detail visibility for better customer decision-making
• Preserves image quality across different device resolutions

Crop Image

• Standardize product image dimensions across your catalog
• Focus on specific product details
• Create consistent product grids for your website
• Optimize images for different platform requirements
• Remove unwanted elements from product photos
• Ensure perfect composition for each product shot

Chat with TryOn AI

• Get instant product information and insights
• Ask questions about specific product details
• Generate product descriptions automatically
• Identify product features and specifications
• Create SEO-optimized product listings
• Extract key selling points for marketing materials
• Streamline content creation workflow

Compare Image

• Track product changes and updates
• Compare before and after image enhancements
• Evaluate different product variations
• Maintain consistency across product updates
• Document product improvements and changes
• Ensure quality control across product updates

Caption Image

• Generate detailed product descriptions automatically
• Extract key product attributes

* Product type and category

* Size and measurements

* Color and material information

* Design details (neckline, hemline, waistline)

* Style and fashion attributes

• Create structured data for e-commerce platforms
• Generate JSON format for easy integration with existing systems
• Maintain consistent product information across channels
• Automate metadata generation for better SEO

Conclusion

TryOn Studio is designed to streamline your product photography workflow, saving time and resources while maintaining professional quality. Whether you’re a small D2C brand or a large e-commerce platform, these AI-powered tools can help you create consistent, high-quality product images that drive sales and enhance your brand presence.

Next Steps

Ready to transform your product photography workflow? Getting started with TryOn Studio is easy:

1. Create your account at https://playground.tryonlabs.ai
2. Go to https://playground.tryonlabs.ai/tryonstudio
3. Upload your product image
4. Explore the features in the toolbar:

* Remove background

* Upscale image

* Crop and adjust

* Generate captions

* And more!

Need more information? We’re here to help:

• Schedule a personalized demo to see TryOn Studio in action — contact@tryonlabs.ai
• Join our community of e-commerce professionals for tips and best practices — https://discord.gg/SuM5Xr9nn9

Let’s work together to elevate your product photography. Book a call with our team to explore how TryOn Studio can help your business grow.

Stay tuned for our upcoming series of detailed tutorials and case studies showcasing how businesses leverage TryOn Studio to revolutionize their product photography workflows.

TryonLabs Playground

Presenting TryOn Studio: Instantly Upgrade Product Images In Minutes was originally published in TryOn Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Experience the next generation of AI-powered fashion creation. Discover our groundbreaking Generate & Reimagine features, engineered with state-of-the-art AI models.

At TryOn Labs, we’re thrilled to enhance your fashion creation journey. Our latest AI update is set to transform your workflow, offering innovative solutions to visualize and create fashion content like never before. Today, we unveil two pioneering features powered by advanced AI, designed to redefine how you approach fashion design.

Reimagine

Transform any image with our powerful AI-powered Reimagine feature. Visualize endless styling possibilities with just a few clicks!

• Complete Outfit Transformation: Instantly visualize an AI generated outfit on your images by simply describing what you’d like to see
• Fashion Model Customization: Change the model’s appearance, and attributes to match your brand’s aesthetic
• Background Enhancement: Transport your products to any setting — from studio environments to outdoor locations
• Accessories & Details: Reimagine shoes, jewelry, hair styles, facial features, and other elements to perfect your visuals
• One-Click Styling: Save time and resources by digitally trying different looks without multiple photoshoots

Generate

Harness the power of the advanced FLUX series of AI models to create stunning visuals from simple text descriptions. From concept to production-ready image in seconds!

• Fashion-Focused Generation: Create complete outfits, individual garments, accessories, and styling elements with industry-leading quality
• Model Diversity: Generate fashion models with diverse appearances, poses, and expressions that represent your target audience
• Product Visualization: Bring your product concepts to life with photorealistic renderings in various settings and lighting conditions
• Background Customization: Place your fashion creations in any environment — from studio settings to urban scenes or natural landscapes
• Production-Ready Output: Download high-resolution images ready for your website, lookbooks, social media, or marketing campaigns

How TryOn AI Benefits Your Business

For E-Commerce Companies

• Reduce Return Rates: Let customers visualize products accurately before purchase
• Diversify Model Representation: Show your products on various body types without multiple shoots
• Seasonal Updates: Quickly reimagine existing product photos in new seasonal settings
• Cost Efficiency: Create unlimited product variations at a fraction of traditional photography costs

For Content Creators

• Unlimited Creative Assets: Generate fresh content consistently without expensive equipment
• Style Experimentation: Test different aesthetics before committing to a specific style
• Quick Turnaround: Meet tight deadlines with rapidly generated custom visuals
• Location Freedom: Create “on location” shots from anywhere in the world

For Marketers

• A/B Testing Visuals: Create multiple campaign versions to test without production delays
• Consistent Brand Identity: Maintain visual consistency across all marketing channels
• Campaign Scalability: Effortlessly create diverse visuals for different audience segments
• Rapid Campaign Creation: Launch time-sensitive promotions without waiting for photoshoots

These powerful features are available right now in your TryOn AI Playground account! Elevate your visual content and stand out from competitors with the cutting-edge FLUX series of AI models by Black Forest Labs.

We can’t wait to see what you create! Share your reimagined and generated images on social media with #TryOnLabsCreations for a chance to be featured on our official channels and inspire others in the TryOn Labs community.

Thank you for being part of our community.

TryonLabs Playground

Follow Us:

Twitter: https://x.com/tryonlabs

LinkedIn: https://www.linkedin.com/company/tryon-labs/

GitHub: https://github.com/tryonlabs

Join Discord: https://discord.com/invite/SuM5Xr9nn9

Revolutionize Your Fashion Workflow: Explore the New Generate & Reimagine Features in TryOn AI… was originally published in TryOn Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you are following our previous articles, you must already be familiar with what Virtual Try-on is and how it is making life easier for customers of D2C fashion Brands. If not, let me quickly explain you;

Let’s say your potential customer is window shopping, they don’t have any intent to buy anything, you are not able to retain them, your bounce rate is high, your conversion rates are very low. But now you have installed and enabled our Shopify extension, which is a new activity for them, they can have fun, while trying out garments. They are taking different pictures in different poses, pairing one tshirt with another trouser, some jacket with some other belt. They like the way they look. They keep adding garment they liked on themselves, to the cart. They have exceeded their budget, so they’ll have to dump some of them, but they’ll still buy some. You have converted a customer whose only intent was to window shop, to actually shop.

The story sounds like a fairy tale for any D2C Fashion brand, but the tale is true and it is backed by numbers. In current market there’s an average conversion rate of 1.8%. Only 1% of D2C brands are currently using such try-on experiences such as, make-up try-on, accessories try-on, and garment try-on. Numbers prove that you can beat the competition if you add such experiences to your online store.

A Shopify report shows how using experiences similar to virutal try-on experience can result in 44% increase in the likeliness that customer will add object to the cart and 27% increase in the likeliness of buying that product.

In this article I’ll explain you how you as a D2C Fashion brand, can install and enable our shopify app extension for your customers.

We recently released the first stable version of our shopify app extension. And you can find it here: https://apps.shopify.com/tryonlabs-tryon-app

Click on “Open app” and install it.

Once you install it, you’ll be redirected to Admin Panel

In the left, you’ll be able to see our app; “Tryon AI” and just below it, Upload Garments Section. Click on “Upload Garments”.

You’ll be able to see the listed products in your store;

If you can’t see the listed products, try reloading and wait for a few seconds for the products to get fetched from your website.

You can select the listed products and a “Upload” button will appear in the bottom right. You can click on it, and the garments will be uploaded for virtual try-on.

Note: If you still can not see the images, you have the alternative option to upload garments using button to top right in the same page; “Upload New Garments”. Fetching store products might cause issue in a few older shopify themes.

As of now we support all types of upper garments. Flatlay images perform best for virtual tryon, refrain from uploading human fashion models, or images on hangers;

You can simply spread the garment on table and take a picture from top.

Once you are done uploading the garments, you will be able to see them in the admin panel, click on “Tryon AI” app button.

Now navigate to your “sales channel”, and click on “Online Store” in the left panel.

Click on “Customize”

Click on the third option; “App Embeds” on the left most side. And enable the “Tryon AI” app embed.

You’ll start seeing the widget, on bottom right once the app is enabled. We give you the option to customize the widget position. Eg. the default values defines the postion of app embed from bottom is 1.25 and from right is 1.25.

Now if you want to move it to a little left, increase the distance from right;

Increase the value of “Right Position ” changed the widget placement. Similarly, if you want to place the widget in bottom left, use these settings;

Click on widget and you’ll be able to see how pop-up will look like;

Once you are satisfied with the positioning of widget, click on “Save” on the top right and “Exit” from top left.

Voila!!! Tryon AI feature is live on your website.

There are several features in the app, that you and your customers can explore. So, head over to your live website to check how it looks. Try a few garments yourself.

The setup process is quite straight forward. But if you get any doubt or query, just drop an email at contact@tryonlabs.ai and we’ll there to help you.

You’ll get 20 credits for free initially.

1 Credit = 1 Tryon

You can contact us for more credits. For the first 10 D2C brands we will be giving away 500 free credits each. We also have in store amazing discounts and suitable different subscription based on your traffic. Contact us for more details.

Try out our demo store: https://test-store.tryonlabs.ai/

Announcement: Coming soon 2nd version of Tryon AI App, with many more functionalities, better try on accuracy.

About Tryon Labs:

We are building Tryon AI that empowers online apparel stores with Generative AI for Virtual Try-On and Cataloging. Online fashion stores spend a lot on photographers, models, and studios to create catalogs. Online shoppers sometimes struggle to pick clothes that will look nice on them.

Tryon AI cuts costs on cataloging for online fashion stores and improves the shopping experience for customers. Online fashion stores can offer a seamless and immersive way to try on clothes online from the comfort of their homes.

Check out our open-source implementation of TryOnDiffusion: https://github.com/tryonlabs/tryondiffusion

Visit our website: https://www.tryonlabs.ai or contact us at contact@tryonlabs.ai

Join our discord server: https://discord.gg/HYbupTzV7E

Shopify App Extension by TranonLabs.ai was originally published in TryOn Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Virtual Try-On is revolutionizing the entire online shopping experience. It empowers customers to visualize how different clothing items would appear on them, all from the comfort and convenience of their homes, enhancing customer satisfaction by providing a more personalized shopping experience. The overall experience reduces the rate of returns, a common issue faced by online retailers.

In this article, I explain the research paper “Virtual Try-On with Pose-Garment Keypoints Guided Inpainting”, in which the authors proposed a three-step keypoints-guided inpainting approach for virtual try-on. Let’s take a closer look!

Problem

There are countless approaches for Virtual Try-On. However, existing models generate distortions in garment shape and lose pattern details. In most approaches, warping of the garment image for the target pose happens first. Later, blending the person image with the warped garment image and target segmentation map generates the final image. However, inappropriate warping of the garment image or inaccurate estimation of the target segmentation map usually results in the distortion of the garment shape. The blending process might make the clothes picture fuzzy or cause it to lose some of its pattern details.

Existing approaches like Thin-plate spline transformation (TPS) suffer from severe distortion at the overlapping part, cuff, or neckline as well.

There are two approaches to performing virtual try-on:

1. 3D model-based— the high computational cost of 3D modeling and additional sensory devices makes it difficult to perform.
2. Image-based— preferred over 3D model-based approaches.

Why now?

Due to the explosion in online shopping, virtual try-on has become convenient and highly cost-effective. It allows one to visualize the fitting results without physically wearing the garments.

Proposed approach

A pose-garment keypoints guided inpainting method for the image-based virtual try-on task, which produces high-fidelity try-on images and well preserves the shapes and patterns of the garments.

Key steps:

1. Extract keypoints from the given garment and person images, respectively, and then construct graphs from the two sets of extracted keypoints.
2. Feed the graphs into a two-stream graph convolutional network to predict pose-oriented garment keypoints.
3. Use the predicted keypoints to perform garment warping and generate a target segmentation map. For garment warping, separate the garment into five sub-segments, namely left low, left up, center, right up, and right low, and then use the paired original/pose-oriented kerpoints to warp each sub-segment individually. The final warped garment is generated from the five warped sub-ones. This is to handle the overlapping deformation.
4. Generate the target segmentation map using pose keypoints, pose-oriented garment keypoints, and the source segmentation map extracted from the given person image.
5. Inputs for the try-on image generation are the warped garment image, the target segmentation map, the given person image, the source segmentation map, and the pose keypoints. To avoid blurring and loss of pattern details, recompose a person's image with incomplete fitting areas and pinpoint the missing regions according to the semantic segmentation maps.

Here are the main contributions of the paper:

They proposed:

1. A pose-garment keypoints guided inpainting method for the image-based virtual try-on task.
2. A graph-based model to extract the pose-oriented garment keypoints for garment warping and target segmentation map estimation.
3. A semantic-conditioned inpainting scheme to generate the final try-on image.
4. Conducted extensive experiments to verify the effectiveness of KGI and show quantitative and qualitative improvements compared with prior methods.

Methods

The authors proposed a three-step process:

1. Pose-Oriented Garment Keypoints Detection — Human pose and garment keypoints are extracted from source images and constructed as graphs to predict the garment keypoints at the target pose.
2. Segmentation Map Generation, Cloth Warping, and Person Image Recomposition — The predicted keypoints are used as guide information to predict the target segmentation map and warp the garment image.
3. Semantic-conditioned Inpainting — The tryon image is finally generated with a semantic-conditioned inpainting scheme using the segmentation map and recomposed person image as conditions to avoid the issue of blurring and loss of pattern details.

Let’s dive deep into the three-step process.

1. Pose-Oriented Garment Keypoints Detection

In this step, they extracted the keypoints with off-the-shelf models for human pose estimation and fashion landmark detection. The keypoints are represented with the graph structure composed of nodes and edges to better model the relationships of different keypoints.

The pose graph consists of 10 nodes and 18 edges. Nodes are the joints of the upper human body and edges are the corresponding human skeleton.

In Figure 3, 3(a) shows 10 nodes and 18 edges, 3(b) shows 32 keypoint nodes, and 3(b) and 3(c) shows 64 edges representing the contour and 28 edges representing the symmetry structure of the garment.

They devised a two-stream graph neural network with graph convolution blocks depicted in Figure 4.

In Figure 4, the mainstream takes the garment graph as the input for nodes feature regression. The pose graph is embedded as the side stream to condition the regression task by hierarchically providing pose information to the mainstream at different feature levels.

The network is trained in a supervised manner. Lₖₚ is the loss function, and it’s a combination of nodes loss L_{N} and edges loss L_{E}.

M_{N} and M_{E} denote the number of nodes and the number of edges in graphs. xᵢ, xⱼ, x ′ᵢ and x ′ⱼ are the features of the iₜₕ and jₜₕ nodes in graph g and g ′, respectively, and aᵢⱼ is 1 if an edge exists between nodes i and j and aᵢⱼ is 0 otherwise.

2. Segmentation Map Generation, Cloth Warping, and Person Image Recomposition

In this step, they perform the following steps:

1. With the predicted pose-oriented garment keypoints, they generate the target segmentation map and perform cloth warping. Afterward, they recompose a person image used to be inpainted.
2. The garment region and part of the skin regions are removed from the source segmentation map (generated from the given person image) to produce a garment-agnostic segmentation map.
3. They draw sketches of the human skeleton and garment contour using pose keypoints and pose-oriented garment keypoints.
4. The sketches are stacked with the garment-agnostic segmentation map and fed into an autoencoder to generate the target segmentation map.

This model is trained with a cross-entropy loss in a supervised manner. The pose-oriented garment keypoints are also used for fine-grained garment warping.

Problem: TPS for garment warping is ineffective when encountering folding and occlusions.

Solution: Divide the garment into 5 sub-segments left low, left up, center, right up, and right low. Then use the paired original and pose-oriented kerpoints to warp each sub-segment individually.

Lastly, combine the five warped sub-garments and recompose the final incomplete try-on image for final inpainting.

3. Semantic-conditioned Inpainting

This is the final step. In this step, they inpaint the missing regions in the recomposed person image according to the target segmentation map and the existing pixels. A binary mask is produced indicating whether the region is kept or not during inpainting. A semantic conditioned inpainting model based on the denoising diffusion is used here.

DDPM — an image x₀ can be transformed into a white Gaussian noise by progressively adding noise in T time steps and in reverse, a noise sampled from the standard Gaussian distribution can be reconstructed to an image x₀ by predicting and removing the noise step by step.

Timesteps — [1, T]

Inpainting starts from time T. Here, x₀ is the image to be inpainted, xₜ is the noise sampled from the Gaussian distribution, m is the content-keeping mask, and s is the segmentation mask.

At each time step t > 1, the image xₜ₋₁ is the composition of x^{keep}_{t−1} and x^{inpaint}_{t−1}.

diffusion model — ϵ_{θ}

The network architecture consists of spatially-adaptive normalization to embed the segmentation map into the diffusion model.

The overall loss function of the diffusion model looks like this. Where t is a given time step sampled from [0, T], s is the semantic segmentation map, x₀, x_{t−1}, xₜ are the images at corresponding time steps, ϵ and ϵₜₕₑₜₐ are the noise and the denoising diffusion model, respectively, q and p_{θ} are diffusion process posterior and the distribution of estimations.

Experiments

1. They conducted experiments on the VITON-HD dataset consisting of 13679 pairs of garment and person images.
2. They conducted experiments under both paired and unpaired experimental settings for fair comparison with prior methods.
3. They used Frechet Inception Distance and Kernel Inception Distance (KID) to evaluate the quality of models by comparing the distributions of generated images and ground truths. For paired images, in addition to FID and KID, they also computed the Structural Similarity (SSIM)and Learned Perceptual Image Patch Similarity (LPIPS).

Paired Settings Results

Task: The garment region of the person image is replaced with its paired garment and the generated try-on images are expected to be similar to the original person images.

1. Experiments are conducted at three image resolutions: 1024x768, 512x384, and 256x192.
2. KGI consistently performs best in terms of SSIM, FID, and KID evaluation metrics at different image resolutions. KGI has higher SSIM scores than prior methods.
3. For LPIPS, their method performs better than CP-VTON, ACPGN, and VITON-HD methods at all image resolutions.
4. KGI well preserves the color and detailed pattern information and after the image recomposition, a content-keeping mask is used during the final inpainting.

Unpaired Setting Results

Task: fit an arbitrary garment on the given person image.

1. Experiments are conducted at 1024x768 image resolution
2. The try-on images generated by KGI well preserve the shape, color, and textures of the garment image without obvious artifacts and semantic errors.
3. SSIM and LPIPS are not applicable as there is no ground truth under the unpaired setting.

Conclusion

They proposed a pose-garment keypoints guided inpainting (KGI) method for image-based virtual try-on tasks, which produces high-fidelity try-on images and well preserves the patterns and shapes of the garments.

Let’s recap:

1. Pose keypoints and garment keypoints are extracted from the source images and constructed as graphs to predict pose-oriented garment keypoints.
2. The predicted keypoints are used as guide information for garment warping and the target segmentation map generation. The given person image is recomposed with the warped garment image based on the semantic information of the target segmentation map.
3. The missing region of the recomposed person image is finally filled with a semantic conditioned inpainting scheme

They conducted extensive experiments on the VITON-HD dataset under both paired and unpaired settings. Their qualitative and quantitative results outperform prior methods at different image resolutions.

About Tryon Labs:

We are building Tryon AI that empowers online apparel stores with Generative AI for Virtual Try-On and Cataloging. Online fashion stores spend a lot on photographers, models, and studios to create catalogs. Online shoppers sometimes struggle to pick clothes that will look nice on them.

Tryon AI cuts costs on cataloging for online fashion stores and improves the shopping experience for customers. Online fashion stores can offer a seamless and immersive way to try on clothes online from the comfort of their homes.

Check out our open-source implementation of TryOnDiffusion: https://github.com/tryonlabs/tryondiffusion

Visit our website: https://www.tryonlabs.ai or contact us at contact@tryonlabs.ai

Join our discord server: https://discord.gg/HYbupTzV7E

References:

1. HR-VTON — https://arxiv.org/abs/2206.14180
2. VITON-HD — https://arxiv.org/abs/2103.16874
3. DDPM — https://arxiv.org/abs/2006.11239
4. Clothflow — https://openaccess.thecvf.com/content_ICCV_2019/papers/Han_ClothFlow_A_Flow-Based_Model_for_Clothed_Person_Generation_ICCV_2019_paper.pdf
5. TryonDiffusion — https://arxiv.org/pdf/2306.08276.pdf
6. OOTDiffusion — https://arxiv.org/pdf/2403.01779.pdf
7. DCI-VTON — https://arxiv.org/pdf/2308.06101.pdf
8. TryOnGAN — https://tryongan.github.io/tryongan/static_files/resources/tryongan_paper.pdf
9. Openpose — https://github.com/CMU-Perceptual-Computing-Lab/openpose, https://arxiv.org/pdf/1812.08008.pdf
10. KGI Implementation— https://github.com/lizhi-ntu/KGI

[P-1] Demystifying KGI: Virtual Try-On with Pose-Garment Keypoints Guided Inpainting was originally published in TryOn Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction:

The problem of Virtual Try-On has recently come up in the limelight especially with the rise of Diffusion models, many researches have been conducted with different approaches, some with Generative Adversarial Networks (GANs) and others with Diffusion models, both claiming to solve the problem of Virtual Try-On. Most of these models are being trained upon a popular dataset called VTON-HD as a base dataset. The objective of this write up is to provide technical explanation, comparison between different approaches and their merits and demerits, lastly to give an oversight in the paradigm of Virtual Try-On and researches being conducted.

Overview:

There are multiple research papers which have shown promising results on paper as well as in practice, one such research paper is Taming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow(11th August 2023),which uses the Diffusion model to tackle this problem of Virtual Try-On. A model called DCI has been proposed which is trained on VTON-HD dataset and the results have surpassed previously proposed methods. The comparison below illustrates the claims

From above we can see that there are mainly two problems which arise while executing Virtual Try-On i.e warping the garment properly and persevering the texture of the garment. The above image is taken from Taming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow itself.

GAN based model though preserve the exact texture and text in the garments, but struggles to generate occluded areas, like unseen segments of garments and human body parts While on the other hand Inpainting by Diffusion based Methods can preserve texture of the garment to some extent but again are in perplexity while handling the colour accuracy and warping. Henceforth, DCI comes as winner in such a case as it is able to warp the garment without distorting it’s texture

Warping Modules

1. DCI Warping Module

The DCI (Dress Code Inference) model utilizes a specialized architecture to handle the complexities of virtual try-on tasks by dividing the process into distinct modules. The first module is the Warping Module, which is primarily focused on manipulating the garment image to fit the target pose of the person. This module takes the original garment and performs warping to align it with the person’s body shape and pose. After the garment has been successfully warped, it is then passed onto the second module.

The second module is the Refinement Branch, where the warped garment undergoes further processing to ensure a natural and seamless integration onto the person. This involves inpainting and refinement using a Diffusion-based Model to address any imperfections or mismatches between the garment and the person’s body. The combined use of these two modules enables DCI to effectively address the challenges of garment warping and blending, resulting in high-quality virtual try-on outcomes.

2. KGI Warping Module

The KGI (Keypoint Guided Inpainting) model presents an innovative approach to the virtual try-on problem by introducing a Warping Module that addresses the limitations of traditional thin-plate spline (TPS) transformation techniques. Unlike TPS, which often leads to severe distortion, especially in areas of overlap, KGI’s warping module strategically divides the garment into five sections. These sections are then individually adjusted and reassembled to create a warped garment image that preserves the integrity of the original garment, avoiding the common issues of distortion, blurring, and loss of detail.

This approach is particularly effective in handling complex areas such as cuffs and necklines, where TPS transformations typically falter. The Warping Module operates in tandem with a graph convolution network that includes two streams: a main stream that processes the garment graph and a side stream that embeds pose information. This structure enables the KGI model to achieve a high level of precision and quality in the virtual try-on process.

Therefore, we can conclude that such an approach does in fact yield better results. Moreover, KGI claims to fill a gap that exists in thin-plate spline transformation (TPS) as it suffers from severe distortion at the overlapping part; other distortions may happen at the cuff or neckline. Moreover, the blending procedure may also blur the garment image or lead to the loss of pattern details. We can see the difference in the figure below

As we can understand from the figure that TPS distorts the area with occlusion and whilst the warping module of KGI splits the garment into 5 parts and recombines them to form the warped garment, preventing the distortion. To understand how the warping module works in KGI

The figure will help us see how it works in depth, The network consists of two graph convolution streams. The mainstream takes the garment graph of a given garment image as the input and outputs the garment graph at the target pose. The side stream takes the pose graph as input and hierarchically embeds the pose information and provides conditions to the mainstream. However, there is one more research paper namely

3. TryOnDiffusion Warping Module

TryOnDiffusion introduces a novel concept in the realm of virtual try-on technology with its parallel UNet architecture. This architecture eschews the conventional two-step process of warping the garment to the target body followed by blending. Instead, it integrates both steps into a single pass through the use of two UNets: one for the garment and another for the person.

The garment-UNet focuses on processing the segmented garment image, optimizing its operations to reduce parameter count by stopping after a specific upsampling block. Meanwhile, the person-UNet is tasked with handling the more complex integration of the garment with the target person. It receives a combination of a clothing-agnostic RGB image and a noisy image, merging them at the channel dimension for precise alignment.

Both UNets leverage separate pose embeddings, which are then fused into the person-UNet through an attention mechanism. These embeddings, along with feature modulation across various scales via FiLM and the incorporation of positional encoding and noise augmentation, enable TryOnDiffusion to perform implicit warping and blending. This unified approach allows for a more streamlined and efficient process, delivering high-quality virtual try-on results without the need for explicit flow computation.

• The system utilises two UNets, one for handling the garment and the other for the person.
• The person-UNet receives inputs of clothing-agnostic RGB Ia and noisy image zt, concatenated at the channel dimension due to pixel-wise alignment.
• The garment-UNet takes the segmented garment image Ic as input and stops after the 32×32 upsampling block to save parameters.
• Pose embeddings for both person and garment are computed separately and fused into the person-UNet using an attention mechanism.
• Pose embeddings are also used to modulate features across all scales using FiLM, along with positional encoding of diffusion time step and noise augmentation levels.

Comparison:

The difference between the approaches lies majorly in the garment warping except in the Tryon Diffusion paper which combines both the processes in a single pass.

In DCI and KGI both, use different models to handle warping and putting it on the person which seems to be a logical appealing approach. The DCI and both KGI both maintain the garment texture and warping by use of such approach but DCI does suffer from issues generating neck area and bottom end of garments as it is not trained on such a data by default and it also faces issues when facing occluded garments, this is where KGI comes in the picture it uses a better warping module and technique to handle occluded garments as shown in Fig. 4 above.

Now, if we take a look at Tryon Diffusion it blends the two processes as shown in Fig. 6. During the preprocessing step, the target person is segmented out of the person image creating a “clothing agnostic RGB” image, the target garment is segmented out of the garment image, and pose is computed for both person and garment images. These inputs are taken into 128×128 Parallel-UNet (key contribution) to create the 128 × 128 try-on image which is further sent as input to the 256×256 Parallel-UNet together with the try-on conditional inputs. Output from 256×256 Parallel-UNet is sent to standard super resolution diffusion to create the 1024×1024 image. The architecture of 128×128 Parallel-UNet is visualised at the bottom, see text for details. The 256×256 Parallel-UNet is similar to the 128 one, and provided in supplementary for completeness

Reason Behind Parallel UNet:

Even though the latest clothing try-on models are successful, they still use a typical UNet design with a method called channel-wise concatenation for preparing images. This method works great for tasks like making images clearer, filling in missing parts, and adding colour to images, where the input and output match perfectly. However, it doesn’t work as well for our task because trying on clothes involves complex changes like stretching the clothing. To tackle this problem, we suggest a new design called Parallel-UNet specifically made for trying on clothes. In this design, the clothes are stretched automatically using special attention mechanisms.

Muneeb Mushtaq

Deep Learning Engineer, TryOn Labs

About Tryon Labs:

We are building Tryon AI that empowers online apparel stores with Generative AI for Virtual Try-On and Cataloging. Online fashion stores spend a lot on photographers, models, and studios to create catalogs. Online shoppers sometimes struggle to pick clothes that will look nice on them.

Tryon AI cuts costs on cataloging for online fashion stores and improves the shopping experience for customers. Online fashion stores can offer a seamless and immersive way to try on clothes online from the comfort of their homes.

Check out our open-source implementation of TryOnDiffusion: https://github.com/tryonlabs/tryondiffusion

Visit our website: https://www.tryonlabs.ai or contact us at contact@tryonlabs.ai

Join our discord server: https://discord.gg/FuBXDUr3

Warping Modules in Virtual Try-On was originally published in TryOn Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Virtual Try-On is revolutionizing the way customers are shopping online. It allows customers to experience products virtually before making a purchase, enhancing their confidence and satisfaction with their online shopping choices.

In this article, I’ll share the top 6 research papers to help you with your Virtual Try-On AI research and development.

Outfit Anyone: Ultra-high quality virtual try-on for Any Clothing and Any Person

Abstract

Outfit Anyone is a Virtual Try-On AI model from Humanaigc, designed for virtual clothing outfitting. The system enables ultra-high quality virtual try-on for any clothing and any person It has also been mentioned in the context of integrating with other AI technologies to dress and animate models…

• Outfit Anyone
• GitHub - HumanAIGC/OutfitAnyone: Outfit Anyone: Ultra-high quality virtual try-on for Any Clothing and Any Person
• OutfitAnyone - a Hugging Face Space by HumanAIGC
• 【Outfit Anyone AI】 - Effortless Virtual Try-On for Every Style

Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All

Abstract

As online shopping is growing, the ability for buyers to virtually visualize products in their settings-a phenomenon we define as “Virtual Try-All”-has become crucial. Recent diffusion models inherently contain a world model, rendering them suitable for this task within an inpainting context. However, traditional image-conditioned diffusion models often fail to capture the fine-grained details of products. In contrast, personalization-driven models such as DreamPaint are good at preserving the item’s details but they are not optimized for real-time applications. We present “Diffuse to Choose,” a novel diffusion-based image-conditioned inpainting model that efficiently balances fast inference with the retention of high-fidelity details in a given reference item while ensuring accurate semantic manipulations in the given scene content. Our approach is based on incorporating fine-grained features from the reference image directly into the latent feature maps of the main diffusion model, alongside with a perceptual loss to further preserve the reference item’s details. We conduct extensive testing on both in-house and publicly available datasets, and show that Diffuse to Choose is superior to existing zero-shot diffusion inpainting methods as well as few-shot diffusion personalization algorithms like DreamPaint…

• Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All
• Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All

https://diffuse2choose.github.io/static/videos/Diffuse_to_Choose_DemoReel.mp4

Amazon's New AI Tool "Diffuse to Choose" Lets You Virtually Try Products In Any Setting

TryOnDiffusion: A Tale of Two UNets

Abstract

Given two images depicting a person and a garment worn by another person, our goal is to generate a visualization of how the garment might look on the input person. A key challenge is to synthesize a photorealistic detail-preserving visualization of the garment, while warping the garment to accommodate a significant body pose and shape change across the subjects. Previous methods either focus on garment detail preservation without effective pose and shape variation, or allow try-on with the desired shape and pose but lack garment details. In this paper, we propose a diffusion-based architecture that unifies two UNets (referred to as Parallel-UNet), which allows us to preserve garment details and warp the garment for significant pose and body change in a single network. The key ideas behind Parallel-UNet include: 1) garment is warped implicitly via a cross attention mechanism, 2) garment warp and person blend happen as part of a unified process as opposed to a sequence of two separate tasks. Experimental results indicate that TryOnDiffusion achieves state-of-the-art performance both qualitatively and quantitatively…

TryOnDiffusion: A Tale of Two UNets

GitHub - tryonlabs/tryondiffusion: TryOnDiffusion: A Tale of Two UNets Implementation

StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-On

Abstract

Given a clothing image and a person image, an image-based virtual try-on aims to generate a customized image that appears natural and accurately reflects the characteristics of the clothing image. In this work, we aim to expand the applicability of the pre-trained diffusion model so that it can be utilized independently for the virtual try-on task. The main challenge is to preserve the clothing details while effectively utilizing the robust generative capability of the pre-trained model. In order to tackle these issues, we propose StableVITON, learning the semantic correspondence between the clothing and the human body within the latent space of the pre-trained diffusion model in an end-to-end manner. Our proposed zero cross-attention blocks not only preserve the clothing details by learning the semantic correspondence but also generate high-fidelity images by utilizing the inherent knowledge of the pre-trained model in the warping process. Through our proposed novel attention total variation loss and applying augmentation, we achieve the sharp attention map, resulting in a more precise representation of clothing details. StableVITON outperforms the baselines in qualitative and quantitative evaluation, showing promising quality in arbitrary person images…

• StableVITON
• StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-On
• GitHub - rlawjdghek/StableVITON: [CVPR2024] StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-On

High-Resolution Virtual Try-On with Misalignment
and Occlusion-Handled Conditions

Abstract

Image-based virtual try-on aims to synthesize an image of a person wearing a given clothing item. To solve the task, the existing methods warp the clothing item to fit the person’s body and generate the segmentation map of the person wearing the item, before fusing the item with the person. However, when the warping and the segmentation generation stages operate individually without information exchange, the misalignment between the warped clothes and the segmentation map occurs, which leads to the artifacts in the final image. The information disconnection also causes excessive warping near the clothing regions occluded by the body parts, so called pixel-squeezing artifacts. To settle the issues, we propose a novel try-on condition generator as a unified module of the two stages (i.e., warping and segmentation generation stages). A newly proposed feature fusion block in the condition generator implements the information exchange, and the condition generator does not create any misalignment or pixel-squeezing artifacts. We also introduce discriminator rejection that filters out the incorrect segmentation map predictions and assures the performance of virtual try-on frameworks. Experiments on a high-resolution dataset demonstrate that our model successfully handles the misalignment and the occlusion, and significantly outperforms the baselines.

• High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions
• High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions
• GitHub - sangyun884/HR-VITON: Official PyTorch implementation for the paper High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions (ECCV 2022).

Taming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow

Abstract

Virtual try-on is a critical image synthesis task that aims to transfer clothes from one image to another while preserving the details of both humans and clothes. While many existing methods rely on Generative Adversarial Networks (GANs) to achieve this, flaws can still occur, particularly at high resolutions. Recently, the diffusion model has emerged as a promising alternative for generating high-quality images in various applications. However, simply using clothes as a condition for guiding the diffusion model to inpaint is insufficient to maintain the details of the clothes. To overcome this challenge, we propose an exemplar-based inpainting approach that leverages a warping module to guide the diffusion model’s generation effectively. The warping module performs initial processing on the clothes, which helps to preserve the local details of the clothes. We then combine the warped clothes with clothes-agnostic person image and add noise as the input of diffusion model. Additionally, the warped clothes is used as local conditions for each denoising process to ensure that the resulting output retains as much detail as possible. Our approach, namely Diffusion-based Conditional Inpainting for Virtual Try-ON (DCI-VTON), effectively utilizes the power of the diffusion model, and the incorporation of the warping module helps to produce high-quality and realistic virtual try-on results. Experimental results on VITON-HD demonstrate the effectiveness and superiority of our method. Source code and trained models will be publicly released at: https://github.com/bcmi/DCIVTON-Virtual-Try-On.

• Taming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow
• GitHub - bcmi/DCI-VTON-Virtual-Try-On: [ACM Multimedia 2023] Taming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow.

If you’re working on Virtual Try-On and want to chat about AI research and development in this area, please reach out to me on Twitter. https://twitter.com/Ahkailash1

Generative Adversarial Networks Projects

About Tryon Labs:

We are building Tryon AI that empowers online apparel stores with Generative AI for Virtual Try-On and Cataloging. Online fashion stores spend a lot on photographers, models, and studios to create catalogs. Online shoppers sometimes struggle to pick clothes that will look nice on them.

Tryon AI cuts costs on cataloging for online fashion stores and improves the shopping experience for customers. Online fashion stores can offer a seamless and immersive way to try on clothes online from the comfort of their homes.

Check out our open-source implementation of TryOnDiffusion: https://github.com/tryonlabs/tryondiffusion

Visit our website: https://www.tryonlabs.ai or contact us at contact@tryonlabs.ai

Join our discord server: https://discord.gg/FuBXDUr3

Essential Virtual Try-On Research Papers For Machine Learning Engineers was originally published in TryOn Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

With the rising popularity of Diffusion models in this era of Generative AI, Virtual Try-On is an application that has attracted many researchers. It synthesizes new composite images by combining the characteristics of different images thus providing a more comprehensive virtual experience for consumers. This is a complex task with a variety of challenges such as generating the required garment in the correct orientation, preserving its finer details, and retaining the characteristics of the target human to generate a realistic output.

The diffusion model has recently surfaced as a promising alternative for producing top-tier images across a range of applications as compared to Generative Adversarial Networks (GANs) which suffer from issues like mode collapse. This article aims to concisely explain the Diffusion-based Conditional Inpainting for Virtual Try-ON (DCI-VTON) mechanism proposed in the paper “Taming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow” (paper link = https://arxiv.org/pdf/2308.06101.pdf) which seeks to employ diffusion models to achieve virtual try-on by framing it as an inpainting task.

The official GitHub implementation link for this paper: https://github.com/bcmi/DCI-VTON-Virtual-Try-On/tree/main?tab=readme-ov-file

DCI-VITON utilizes 2 main models to produce virtual try-on outputs:

Warping Network

This model is responsible for warping the cloth image so that its orientation aligns with the pose and physical characteristics of the human. This is done using a “Parser Based Appearance Flow Network (PB-AFN)” which generates a set of coordinate vectors, each indicating which pixel in the cloth image should be used to fill in a given pixel in the human image. [PB-AFN paper link: https://openaccess.thecvf.com/content/CVPR2021/papers/Ge_Parser-Free_Virtual_Try-On_via_Distilling_Appearance_Flows_CVPR_2021_paper.pdf]

In DCI-VITON, the inputs for the warping module are the cloth image I𝒸 along with the Human Segmentation Map Sₚ (this segmentation map is cloth agnostic to prevent the cloth in the input human image from creating interference) and the Densepose map P concatenated along the channel dimension.

The underlying module of PB-AFN is an Appearance Flow Warping Module which utilizes two modules, namely a dual branch Pyramid Feature Extract Network (PEFN) and an Appearance Flow Estimation Network (AFEN). PEFN extracts two branches of pyramid features, one for I𝒸 and another for Sₚ & P, across N levels. AFEN contains N Flow Networks, where the iᵗʰ network takes in the extracted pyramid features at the iᵗʰ level as well as the flow output of the (i-1)ᵗʰ Flow Network to produce a more refined flow. The cloth image is warped according to the Nᵗʰ flow output.

DCI-VITON utilizes the Total Variation Loss given by:

Where Fᵢ is the iᵗʰ flow output. The total variation loss helps smoothen the warped result despite the high degrees of freedom in the appearance flow mechanism.

To preserve the finer details of the cloth image such as repeating patterns or text, a Second-Order Smoothing Constraint is added to the loss function:

Where Fᵢᵗ is the tᵗʰ point in Fᵢ, Nₜ is the set of horizontal, vertical, and both diagonal neighbors around the tᵗʰ point. 𝒫 denotes the generalized charbonnier loss function.

Additionally, the Perceptual loss and L₁ loss are also utilized in the final loss function given by:

λ_VGG, λ_TV, and λ_sec are hyperparameters that dictate the contributions of each of the individual loss functions. DCI-VITON sets these values to 0.2, 0.01, and 6 during experiments.

Where M𝒸 indicates the mask of I𝒸(the cloth image) and S𝒸 is the clothes mask of Iₚ (person image) respectively, W is the warping function, D is the downsampling function and Φₘ indicates the mᵗʰ feature map of a pre-trained VGG-19 network.

Diffusion Model

Given a person image Iₚ and cloth image I𝒸, the aim of the diffusion model is to combine them into a realistic image that has the same person attributes as Iₚ while keeping the cloth element from I𝒸.

The diffusion model training pipeline described in the paper is divided into 2 branches namely the Reconstruction branch and Refinement branch, both of which are optimized simultaneously during training.

Reconstruction Branch

The aim of the reconstruction branch is to learn how to generate the complete person wearing the desired cloth from a Gaussian Distribution. It operates akin to the standard diffusion model, employing a reverse diffusion process to generate realistic images. The authors use a Latent Diffusion Model (LDM) which operates in the latent space rather than pixel space in order to significantly reduce the computational complexity during training as well as allow for a provision to add conditioning using mechanisms like Cross-Attention.

The ground truth image (I₀) is converted to a latent representation (z₀) using an Encoder, z₀ = E(I₀). The corresponding Decoder of the same Autoencoder can be used to reconstruct the pixel-space images from the latent representation. The main diffusion steps of noising and denoising are performed on this latent space representation.

The forward noising process is then performed on z₀ as per

𝛽 is a pre-defined variance schedule in 𝑇 steps.

In order to maintain the characteristics of the cloth image as well as information regarding the desired orientation of the cloth on the person, a local condition is provided as a guide at each denoising step. These local conditions are created by adding the warped image to the inpainting image to obtain I_lc, which is transformed to the latent space as z_lc = E(I_lc). The denoising is done by an enhanced Diffusion UNet which takes {zₜ, z_lc, m} concatenated along the channel dimension as input. Here m is the inpainting mask spanning the torso and arms region and it is used to inform the model about the regions it needs to focus the denoising on. This allows the model to address poor warping results and irregularities in areas where the cloth and body meet. Moreover, a global condition c, obtained from I𝒸 via the frozen pre-trained CLIP Image Encoder, is incorporated by the UNet via Cross-Attention to emphasize the retention of cloth image features. Therefore, the reconstruction branch uses the following loss function:

where ϵ_θ is the noise predicted by the model.

While the reconstruction branch can eventually generate a synthetic image of the person and the general characteristics of the cloth, the refinement branch is required to capture the finer details of the cloth.

Refinement Branch

The Refinement Branch helps capture details about the cloth such as thickness, orientation, and color of patterns as well as their spatial information. In order to do this, while ensuring that the final results conform to the results generated by the reconstruction branch, the refinement branch utilizes the coarse result (combination of warped cloth Ĩ𝒸 and cloth agnostic Iₐ ) as initial condition I₀’ along with the local and global conditions used in the Reconstruction branch.

The latent space representation of I₀’ is obtained as z₀’= E(I₀’) after which the forward noising process is conducted to obtain zₜ’. The UNet diffusion model then takes {zₜ’, z_lc, m} concatenated along the channel dimension as input and outputs the predicted noise ϵ̂. The refined denoised latent variable ẑ can be obtained by denoising using the reverse equation of the noising process. The pixel space representation of the final predicted image can be obtained as Î = Ɗ(ẑ) where Ɗ is the Decoder function.

The loss function for the Refinement Branch is defined as:

where Igt denotes the ground truth image

On combining the losses, the main objective function used for training the diffusion model becomes:

where λ_perceptual is a hyperparameter used for balancing the 2 loss components.

Experiments

Dataset

The authors have performed extensive benchmarking tests of their proposed model on the VITON-HD dataset which contains 13,679 frontal-view woman and top clothes image pairs at the resolution of 1024×768.

Training the warping module

1. 100 epochs using Adam optimizer at a learning rate of 5x10–5.
2. λ_VGG = 0.2, λ_TV =0.01, λ_sec = 6
3. Trained under 256x192 resolution.

Training the diffusion model

1. Latent space with spatial dimension of c(H/f)(W/f), where channel dimension c=4.
2. AdamW optimizer at a learning rate of 1x10–5.
3. λ_perceptual= 1x10–4
4. Trained on 2 NVIDIA Tesla A100 GPUs for 40 epochs.

Here are some of the impressive results that the authors of the paper have managed to achieve:

The above image depicts how well the model can fit the cloth image onto the human’s original image while preserving the pose and the details of the garment.

The following image further highlights the model’s ability to preserve the finer details of the garment like striped patterns while warping it as per the human’s pose.

Special credits to Unnikrishnan Menon and Anirudh Rajiv Menon for compiling this article.

About Tryon Labs:

We are building Tryon AI that empowers online apparel stores with Generative AI for Virtual Try-On and Cataloging. Online fashion stores spend a lot on photographers, models, and studios to create catalogs. Online shoppers sometimes struggle to pick clothes that will look nice on them.

Tryon AI cuts costs on cataloging for online fashion stores and improves the shopping experience for customers. Online fashion stores can offer a seamless and immersive way to try on clothes online from the comfort of their homes.

Check out our open-source implementation of TryOnDiffusion: https://github.com/tryonlabs/tryondiffusion

Visit our website: https://www.tryonlabs.ai or contact us at contact@tryonlabs.ai

Join our discord server: https://discord.gg/FuBXDUr3

Demystifying DCI-VTON: Diffusion-Based Conditional Inpainting for Virtual Try-On was originally published in TryOn Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.