RICK REA: Helping You Grow Through Online Marketing
  • Home
  • Blog
    • Social Media News
    • SEO Marketing News
    • Digital Trends News
    • Photography News
    • Mobile Marketing
    • Business News
    • Gadget News
    • Printing News
  • Contact
  • About
  • Subscribe


Automating Shop The Look on Pinterest Pinterest Engineering Medium

2/11/2019

0 Comments

 
Automating Shop The Look on Pinterest – Pinterest Engineering – Medium

http://bit.ly/2SjyC4g


Go to the profile of Pinterest Engineering
Pinterest Engineering
Feb 6

Kunlong Gu | Engineer, Discovery

Every day, people come to Pinterest to discover new ideas — and when they find a product they want to buy, it should be easy to purchase it. In home decor specifically, millions of people come to Pinterest to find inspiration. That’s why in 2017, we launched Shop The Look as a way to help Pinners get to the products they love faster, and for brands to put customers on a path to purchase. Shop the Look identifies specific objects in a larger scene with white dots that connect the Pinner to the checkout page.

We previously experimented with a human-in-the-loop approach to match product links with Pins, but needed a better way to scale across the billions of images we show Pinners. As a solution, we used computer vision to fully automate the process of matching products to scenes. Today, we’re announcing a fully automated experience on home decor scenes on iOS, with Android coming soon. This update increases our Shop The Look Pin coverage by 22.5x across billions of Pins and products, and in early testing has already lifted engagement by 7%.

Here, we’ll share how we leveraged computer vision and the dataset unique to Pinterest (175B+ Pins saved with user-added context) to automate this experience.

From Pin to purchase

On Pinterest, 97% of the 1,000 most popular searches are non-branded (meaning people generally start with basic searches for products like “shoes” or “couch”), and so brands of all sizes have an opportunity to reach people who are in a shopping mindset. As we scale, automated Shop the Look will tag organic Pins that have not been linked to a business account. For those brands who’d rather not have their Pins tagged, they can claim their domains.

Behind the scenes

We built this technology in three stages: data collection, machine learning (ML) modeling, and serving.

The data collection stage prepares the data that our machine learning models use for training, while the ML modeling stage prepares our models to identify and localize home decor objects in a scene. This process classifies objects into product categories and represents them with embeddings, which is a digital representation of images so that visually similar images have a shorter distance between their embeddings compared to dissimilar images. (You can find more in our recently submitted paper.) Finally, the serving stage uses our trained model to identify the product category from the query Pin (‘the look’), and then uses visual embeddings to find the closest product candidates to ‘the look’.

Pinterest’s unique dataset

Every computer vision task starts with image data. One of the special things about Pinterest is that most of the Pins people have saved are larger images (from blogs, retail sites, etc.) with many products (a lamp, couch, rug) in one scene. Through an earlier curation effort, we’ve accumulated a high quality dataset of 270,000 scene-product (around 1 million object-product) matching pairs. In addition, we annotated in-house 80,000 scenes images (250,000+ objects) with bounding boxes and categories based on Google Product Taxonomy (GPT). We went through multiple iterations of GPT before we had an ideal annotation guideline. Initially we strictly followed GPT, but the model didn’t perform well in some coarse-grained categories (i.e. beddings, tables). We found that products in these coarse-grained categories are of vastly different shapes and functions. For example, the category of “bedding” include bed canopies, bed sheets, pillows. We then manually cleaned up the existing datasets (fine-grained labeling) and improved the model significantly.

ML Modeling

In this stage, we trained a Feature Pyramid Network Faster R-CNN detection model that parses scenes into objects and annotates them with product categories.

We also trained an Embedding Model to represent images as mentioned above (which we will talk about in an upcoming blog post — get excited!). The embedding learns from visual similarity across Pin images. In particular, we added this high-quality dataset of 270,000 scene-product matching pairs so the embedding can handle the domain shift from scenes to product images. The domain shift means the objects in the scenes are of various lighting conditions, rotation, and noisy backgrounds while product images usually contain a high-quality front photo with clear white background. Our Embedding Model captures that information well while providing product similarity.

The third model is a re-ranking model that mainly trains on user engagement data and re-ranks the visually similar candidates so the product is optimized for engagement. The re-ranking model also takes in semantic and contextual information, such as the category of the scene image, the boards that frequently include this Pin, and other objects in the scene.

Serving

There are three steps in Serving Stage.

When a user engages on a Pin, we first decompose the scene using the detection model. We restrict the search space by matching the annotation from detection and the category annotations in the shopping corpus. Then we use visual similarity scoring (the distance between embeddings) in the shopping index to generate result candidates. Finally, we apply the reranking model on those candidates to fine-tune the results.

Impact

This update brings more computer vision-powered results across Pinterest, showing visually similar ideas to more people. With more Shop the Look Pins in the system, Pinners can expect to see a much more consistent user experience across all home decor scenes.

In terms of internal operation, the automation frees people from doing repetitive work so they can spend more time being creative.

In the long term, the scene images are great resources to learn the relationship between objects, i.e. what objects complement each other or go well together in a certain style. We hope to leverage this rich data of object occurrence and build a sophisticated object graph for every object in the world, making Pinterest a personalized stylist for home, fashion and more.

You can expect to see more from us in shopping and visual search in the coming months!

Acknowledgements: Automating Shop the Look is a collaborative effort at Pinterest. Special thanks to Chuck Rosenberg, Andrew Zhai, Dmitry Kislyuk, Raymond Shiau, Eric Kim, Francis Chen, Jeffrey Harris, Angela Guo, Tim Weingarten, Jen Chan, Joyce Zha, Amanda Strickler, and Stephen Carr for the collaboration on this product.



Social Media



via Medium

February 11, 2019 at 11:20AM

0 Comments



Leave a Reply.


    Amazing WeightLoss

    Click Here!

    Categories

    All
    Analyze Top Competitors
    Anti-Abuse
    Apple
    Apple Watch
    Blog Posts
    Brainstorm
    Brand Awareness
    Communications
    Content Marketing
    Conversion Rates
    Editorial Calendar Tips
    Engagement
    Facebook
    Google Analytics
    How To Marketing Tips
    Influencer
    Instagram
    Instagram Live
    Keyword Search
    Marketing
    Marketing Automation
    Picture Quotes
    Podcasts
    Recording Videos
    Repurpose Blogs
    Research Trends
    Sales Funnel
    SEO Marketing
    Sharing Posts
    Slide Shows
    Smartwatch
    Social Media Marketing
    Social Media News
    Social Media Tools
    Social Selling
    Target Marketing
    Twitter
    Twitter Notifications
    User Interaction
    Video
    Video Marketing

    Archives

    November 2020
    October 2020
    September 2020
    August 2020
    July 2020
    June 2020
    May 2020
    April 2020
    March 2020
    February 2020
    January 2020
    December 2019
    November 2019
    September 2019
    August 2019
    July 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    January 2019
    December 2018
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    March 2018
    February 2018
    January 2018
    December 2017
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    May 2017
    April 2017
    March 2017
    February 2017

    RSS Feed

Powered by Create your own unique website with customizable templates.
  • Home
  • Blog
    • Social Media News
    • SEO Marketing News
    • Digital Trends News
    • Photography News
    • Mobile Marketing
    • Business News
    • Gadget News
    • Printing News
  • Contact
  • About
  • Subscribe