IBM Cloud Databases - Structured Ideas

This Ideas portal is being closed.  Please enter new idea at

Annotating data using Watson Studio using Watson Knowledge Studio with Figure Eight or DefinedCrowd

Crowd Sourcing data annotation for Text-based documents using the combined power fo Watson Knowledge Studio and Crowd sourced annotation platforms like figure figure eight or definedcrowd. Watson Knowledge Studio offers the capability to assign annotation tasks and provides capabilities to measure inter-annotator agreement for these tasks. It would be ideal if a project administrator could leverage crowd source annotation platforms like figure eight, defined crowd, cloud flower or others as part of a native integration. Crowd sourcing annotation tasks in WKS using these platforms would be a very powerful way to develop and refine annotation models rapidly. 

  • Guest
  • Jan 18 2019
  • Needs review
Why is it useful?
Who would benefit from this IDEA?
How should it work?
Idea Priority High
Priority Justification
Customer Name
Submitting Organization
Submitter Tags
  • Attach files
  • GRAHAM WALKER commented
    January 20, 2019 10:11

    How do you ensure that you get quality annotations from crowd sourcing?  Is there a way to vet some of the annotations to then flag someone as doing quality work?

  • MICHAEL HELLER commented
    January 20, 2019 10:18

    WKS already has an inter-annotator agreement capability to compare performance across annotators. By enabling the crowd sourcing of annotations, it helps to scale this capability where useful thereby cutting down on the time necessary to improve performance through training.