Bosch Annotate

Summary

This project was sponsored by Bosch RTC. Bosch RTC is working on a new internal project that will attempt to replace rote tasks currently being completed on Amazon’s Mechanical Turk by a platform of contractors or annotators, paid more fairly and with more enjoyable work than Mechanical Turk workers.

The current plan is for Bosch internal teams to migrate to the new platform (Bosch currently uses Amazon Mechanical Turk for a lot of work in autonomous vehicle and machine learning labeling tasks), then open it up to Bosch partners, and eventually open the platform to external developers.

Being a 2-sided marketplace we designed a new gamified, mobile annotation platform that annotators around the world could enjoying working on, delivering a great user experience to incentivize them to create high quality annotations. We also designed a new streamlined platform that developers could leverage to create and manage annotation tasks.

The Problem

Crowd workers today perform annotation tasks on platforms like Mechanical Turk for low wages. Apart from the low monetary incentive, annotation tasks are repetitive and boring and often times the results are of low quality.

These annotation tasks feed Bosch's autonomous (self) driving vehicle algorithms where even slightly low quality annotations could be fatal!

The Solution

To investigate breakdowns in the current task flow on both sides of the marketplace, with a goal of delivering a delightful user-experience to users incentivizing them to provide high quality annotations.

Crowd workers today perform annotation tasks on platforms like Mechanical Turk for low wages. Apart from the low monetary incentive, annotation tasks are repetitive and boring and often times the results are of low quality.

These annotation tasks feed Bosch's autonomous (self) driving vehicle algorithms where even slightly low quality annotations could be fatal!

Opportunity

Current Situation at Bosch

How Bosch operates today

It is evident there are several breakdowns present in the way Bosch RTC operates!

• Internal annotators are paid employees and there is a fixed limit on the size of the team. However, demand for annotation work exceeds supply per year.
• 33% of annotation work are outsourced on platforms like Mechanical Turk and the quality of these tasks are low.
• Different developers create customized annotation requests, and the current format is in the shape of a 30+ page document, that could potentially be a reason for the low quality results

Market Size - Is this problem worth solving?

Market size of autonomous annotations

Current Total Addressable Market size is currently at $137M. This figure is based on our estimates for a wide variety of annotation applications, from health care, autonomous driving, and manufacturing applications.

Service Attainable Market is solely based on Automotive Annotations. The current size is $53M.

Estimated Target Market projection, by 2021, is $12M, based on a 30% year over year growth of annotation request, internally, within Bosch. Now, this is a conservative estimate based on our current info on Bosch’s current annotation capabilities with the Bangalore team and the use of 3rd party platforms.

Personas

The Developer, Susan Sanders

Developer Persona: Susan Sanders

Susan is a developer at BMW in Germany. BMW plans to roll out their own lines of autonomous vehicles and requires almost 600,000 images of real time traffic data to be annotated with 100% accuracy.

Currently, BMW submits this data to Amazon MTurk but there lies two main problems:

• There are a lot of scam annotators on MTurk. The quality of the rest of the results are not high enough to eventually train autonomous driving cars
• Creating customized annotation results take a minimum of 2 days and the results are very poor.

Developer Pain Points

• 3rd party platforms lack effective annotation customization options.
• Lengthy and inconsistent internal custom RFQs, lead to inefficiencies.
• There aren't any platforms enabling communication between annotators and developers.

The Annotator, Ravi Kumar

Annotator Persona: Ravi Kumar

Ravi works as a sales assistant in Bangalore, India during the day and works on average 3 hours a day on Amazon Mechanical Turk. He has completed almost 100,000 tasks and has been rejected 24 times. Out of these 24 times, one request was in a language foreign to him and he understands that this was his fault. The rest were a result of requesters scamming the system to get work done for free.

This frustrates Ravi, who puts in the effort to ensure the quality of his work. He wishes a platform existed where annotators could communicate to developers so as to minimize misunderstandings and navigate potential language barriers.

Annotator Pain Points

• Lack of consistent jobs and consequently, low retention rates.
• There is no platform for annotators to provide feedback on the tasks they completed, that could potentially foster task improvement.
• Tasks are monotonous and tedious, and engagement is low.
• Annotators do not feel valued and needed.

Research Phase

Competitive Analysis

We used a radar diagram to visualize our competitive analysis, to be consistent with Bosch RTC. We performed our analysis based on several decision criteria:

• Platform features: The variety of developer side and worker related features that the platform has centered around annotation
• Support Complexity: Data source complexity, the range of data sets that can be classified through the platform
• Data Quality: Quality related features that reflect the presence of QA process
• Workforce Diversity: Addresses the location/presence of the workforce, does the worker presence allow the platform to operate 24/7?
• Workforce Allocation: Does the platform allow for tasks to be allocated efficiently throughout the workforce
• Workforce Quality: Does the platform enhance the quality & performance of the workforce through training, analytics etc?

Competitive Analysis on the autonomous annotation industry

User-Centered Research

Our final deliverable was and all of our decisions throughout the project was informed by users. We interviewed industry and academic experts (developers) and potential users (annotators).

Developer Interview Feedback

"Annotation quality on MTurk is really bad because the instructions are not comprehensive enough. Annotators can not provide high quality results even if they wanted to!" Eshed Ohn-Bar, Postdoctoral Fellow, Robotics Institute, CMU

"The GUI on existing platforms like Crowd Flower are horrible." Eshed Ohn-Bar, Postdoctoral Fellow, Robotics Institute, CMU

"I currently use MTurk. One reason that would convince me to immediately switch services is the number of scammers on MTurk." Gunnar Sigurdsson, PhD Student, advised by Abhinav Gupta, Robotics Institute, CMU

Developer Interview Insights

• Human annotation is still much MORE effective and accurate than current machine learning algorithms.
• When machine learning is used for bounding objects for example, and humans are asked to adjust the results, humans are more likely to think the given results are “good enough” and not bound further objects that the algorithm may have missed. This is why these algorithms haven’t been integrated into many existing platforms yet.
• The amount of time people spend on the platform could predict churn rate, and differentiate serious annotators from the rest.
• The GUI on existing platforms like Crowd Flower are horrible.
• Annotation quality on MTurk is really bad because the instructions are not comprehensive enough.

Annotator Interview Feedback

"There should be a system to remove poor requesters. For example, after so many negative reports they are suspended. If it is found that they are fraudulent, then they are removed or never allowed back on the platform. Requesters are starting to think they can get free work." Annotator 92

"I am part of community to see things I didn't see on the platform yet." Annotator 64

"I have 24 rejections in nearly 100000 submitted tasks. Only one was because of something I did wrong. The rest were mistakes from the requester that were never rectified or straight-up scams." Annotator 57

Annotator Interview/Survey Insights

• 49% of annotators say that performing crowd sourced work is their primary source of income
• 67% of annotators have worked for different crowd sourcing platforms simultaneously. 31% of the surveyed annotators said that AMT was the only platform they served as an annotator.
• 44% of the surveyed annotators have experience in bounding boxes and 30% of the surveyed annotators have experience in semantic segmentation.
• 64% of surveyed annotators say they have received little to no training for their annotations. Most annotators are self-taught and seek out resources on various community boards and forums unsupported by Amazon Mechanical Turk.
• 54% of survey annotators report that they do not feel valued as workers for Amazon Mechanical Turk. Many annotators reported their perceived lack of value was rooted in poor pay and little support on behalf of the platform.

Minimum Viable Product

Developer Dashboard

A seamless data submission dashboard including a preset list of annotation tasks with an option for task customization as necessary. Real-time review and task progress update for quality assurance.

Developer Dashboard to streamline task creation and management

Annotator Platform - Needs Validation

Storyboard for Bosch Annotate

User Feedback

"I would use this. I have always wanted a platform like this on mobile that I could use when I'm bored, like when I'm on the bus coming to school." Master's Student, Robotics Institute, CMU

"I never knew such platforms even existed. I currently work as a bartender after school. I would be interested in this app. While making the world a better place sounds great, I would still use the app for the money!

Bosch Annotate

An annotation platform for both the web and smart phones, using gamification to incentivize users.

MVP - UI Click Through

As part of our final deliverable to Bosch RTC, we consolidated all our findings and insights into a Design Guidelines document consisting of the design process and design thinking that went into all aspects of the MVP. These categories range from onboarding, to the Gatcha mechanic, to the feedback platform.

Design Guidelines for Bosch Annotate. Download the document here

Pricing Plan

The new product will automatically assign workers tasks based on their performance and efficiency with certain tasks. These per image cost estimates include semantic segmentation and 20 bounding boxes. Our new total costs are now in the competitive range of some of the more leading platforms, such as Scale API.

Pricing Plan for Bosch's new Internal Platform

Final Thoughts

Crowd sourcing jobs have always been looked at with a negative sentiment. People have generated mental models comparing crowd workers to robots working in factories. By moving away from Desktops on which crowd sourcing platforms have been always built, by moving towards mobile and by designing a meaningful, interesting user experience, not only can we expect high quality results from existing crowd workers but we can also hopefully create a new diverse, segment of crowd workers and by doing so redefine the industry!