Self-Service
Crowdsourcing at your fingertips
Judge/Relevance
Judge/Relevance projects typically require contributors to score how well an object matches the presented criteria (e.g., search relevance, image relevance, video relevance, etc.).
Best practices:
- Use our Ratings tag or Radios tag to capture input.
- Instead of using linking to products, try and display your content directly on the Contributor form with an image tag or iFrame (CML editor required).
- Use Gold Standard Data to ensure quality - Gold tests contributors as they progress through the job, removing anyone who can't perform up to your standards.
- Make sure your Gold units account for all potentially correct responses. Otherwise, you may punish quality contributors.
- Use gold to train contributors around edge cases as well as your more common cases. Try and make your Gold as evenly distributed as possible across options as possible.
- When considering which geographic locations to run your jobs, think about cultural barriers that could cause conflicts.
- Choose your form language carefully - Clarity is important, even if your job is simple. Examples are highly recommended.
- Make sure your job is yielding adequate results before running a large job. Tests are highly recommended.
- Be sure to skim through the options available to you in the Advanced Options on the Edit page before running.
More to come.
