FAQ
Is my project a good fit for CrowdFlower?
A variety of projects run on the platform. Jobs work best when they can utilize our quality control technology, Gold Standard Data (explained below). We do not recommend running a job with a simple set of questions and no plan for quality assurance. Gold data, validation, qualifications, and confirmation codes are highly recommended.
Spam and traffic building projects (jobs requiring contributors to vote, "like", back- link, etc.) are not permitted on the platform due to their cheating nature and the negative response we see from our contributors.
Can you provide some tips on creating a job?
CrowdFlower is compatible with a wide array of job types, each with a different set of goals and processes. Before you get started, you should answer the following questions:
- What problem are you trying to solve?
- What is an appropriate solution to this problem (what data do you want)?
- What questions do you need to ask to obtain the data?
- How do you want to ask these questions (i.e., what language will you use), and what type of form element (e.g., pull-down, text field, etc.) will you ask it with?
- In what order do you want to ask these questions?
- How might these questions behave with each other dynamically (does this form require logic or contingencies)?
- Have you provided enough background information?
- Do your instructions make sense?
- Do your questions beg the specific answers that you want?
How can I test my job? Can I access my job as a contributor?
You have access to the CrowdFlower Internal Interface, which allows you and your team to submit responses in your job as contributors. You will find the Internal Interface in the list of available contributor channels when ordering judgments. After running the job on the channel, a link will appear on the Overview page that will provide access to the contributor interface.
How is contributor trust calculated?
Contributor trust is calculated based on a contributor's performance on Gold questions within a specific job.
In the aggregate results, how is the confidence level calculated?
Confidence is determined by agreement between contributors and accuracy on Gold (trust).
What is Gold Standard Data?
By saving the correct answers to a small set of units prior to running a job, we can calculate the quality of a contributor's performance and reject them if their accuracy drops below 70%. Gold Standard Data acts as hidden tests that are randomly shown to contributors as they complete the job. You can create Gold units individually with our Gold-digging tool (shown below), or in bulk via a spreadsheet.
Things to know:
Good Gold has a definite answer and should account for all correct answers. You can save multiple answers per questions. Your Gold should be free of ALL ambiguity. To help train contributors, you can include reasons with your Gold that will be shown to contributors when they miss a question.
How much Gold should I create?
As a rule of thumb, we suggest 5-10% of your total dataset be Gold units. This is a loose rule and can vary on job type. Basically, contributors shouldn't see a Gold Unit enough times to be able to recognize it. If you have 10,000 units, don't feel compelled to dig 1,000 Gold Units. As long as you are confident that contributors will not be able to pick up on patterns (ratio of responses, remembering specific units, etc.) your job should run smoothly.
You can also utilize the Max Judgments Per Contributor feature (found in the Advanced Options under the Edit tab) to restrict the amount of judgments contributors can submit in your job. Note that this can hinder throughput if it is low.
How do I create Gold with a spreadsheet?
It's relatively easy to create Gold with a spreadsheet.
1. Format your data
As seen above, your data should contain two columns for every field in your form that will contain Gold values. The two Gold columns should be formatted as followed:
"question_name" is the field's label with spaces converted to underscores and uppercase letters converted to lowercase — e.g., if your field's label is "Enter some information.", it should be converted to "enter_some_information" in your data. All punctuation should be excluded from the header. If using the CML Editor, the "name" attribute can be used as well.
"question_name_Gold" should contain the correct answers for the field. If there is more than one correct answer for the field, the answers should be delimited with newline characters.
"question_name_Gold_reason" should contain Gold reasons for the field - optional (but highly recommended) explanations that will be shown to contributors who wrongly answer the field.
A column with the header "_Golden" is also required, which will be used to change a unit's state to Golden. A unit should contain the value "TRUE" in this column if it is Gold. All non-Gold units should be left blank.
2. Upload the file and flag your Gold data
After the file has been uploaded, select "Convert Uploaded Gold" to set the state of all units with "TRUE" specified in the "_Golden" column to Gold. Use this to flag your Gold units instantly when uploading your data. In the example above, three Gold units would be created.
3. Link your Gold Data to your form
Using the Graphical Editor:
Select "Link Uploaded Gold Data" to link the fields in your form to your uploaded data. Remember that your label must match the header in your data (excluding case and underscores) for them to link properly. E.g., "Enter some information" must be "enter_some_information_Gold" in your data. Fields that have been linked properly will be highlighted Gold in the Graphical Editor. An error message will be displayed if nothing is linked.
Using the CML editor:
You are permitted to set the "name" OR "label" attributes when linking your Gold data, but note that "name" takes precedence if both attributes contain values. The "Gold" attribute will need to be set to "true" (Gold="true") as well for each field containing Gold information.
E.g.,
Gold reasons will be automatically linked to the field
How is Gold distributed throughout the job?
A Gold unit will be present on every page a contributor completes
What is a unit?
A unit is a single row from the data you upload. For example, let's say your project is to categorize dresses and you uploaded 100 images of dresses into your job. A unit would be considered a single dress, regardless of how many questions you ask about the single dress.
What are judgments?
A judgment is a single contributor's answer to a unit.
What is a page?
To avoid contributors having to constantly click "Submit" in your job, we allow you to display multiple units on a single page of work. We highly recommend including enough units on a page to equal 3-5 minutes of work. There will be one Gold unit hidden on each page.
Can I set restrictions on contributors?
You can do so in the Advanced Options button under the Edit tab. Jobs can be set to admit only those contributors from a specific country. On the other side, you can choose to exclude specific countries. You may also select how much of your job each contributor can see by setting the Max Judgments Per Contributor feature.
My job collected more judgments than I ordered. Why is this?
Contributors will sometimes move back and forth from trusted to untrusted based on their accuracy. When a contributor is considered untrusted, the system won't include their judgments in the results and collects another judgment to meet the number you requested. When said contributor moves back into trusted, our platform then adds an extra judgment into the spreadsheet. Note that you are not responsible for paying for these extra judgments.
How do I read my results?
Aggregated Results
The Aggregated Results CSV shows the dominant response for each form element (question) in your job, as well as all source data presented to contributors or included in the original unit of data. Note that the Column Headers for contributor responses are pulled from the name (or label) attribute of the form element in question. Each aggregated response will have an associated confidence value, which measures the agreement among contributors on a scale from zero to 1. For example, if all contributors answer “Yes” to a given question, that corresponds with a confidence value of 1. Also note that confidence values are weighted by each contributor’s individual Trust Score, which reflects that contributor’s accuracy on Gold units.
The following table shows common headers in an Aggregated Results CSV and how to interpret them.
_unit_id: Unique identifier in the CrowdFlower system for each unit of data
_Golden: TRUE If the unit in question is a Gold Standard unit.
_trusted_judgments: Number of trusted judgments collected for this unit. Note that because we send out units to many contributors at once, you may notice that some units have more trusted judgments than you ordered.
{{field_name}} Aggregate contributor response fields, where each column header is taken from the name attribute of the relevant form element. This will show the dominant response (result) for each form element from trusted contributors.
{{field_name}}:confidence: Measures agreement among contributors for the dominant response. Note that this score is weighted by the individual Trust Score of each contributor.
{{field_name}}:confidence_summary: Shows each unique trusted contributor
response and the associated confidence value for form elements that are not aggregated.
{{field_name}}_Gold: Gold Standard (correct) response for this unit, if the unit has been defined as Gold (i.e, if "_Golden" equals TRUE).
{{field_name}}_Gold_reason:Message a contributor sees after answering a specific Gold unit incorrectly.
The Full Results CSV
Shows each individual trusted judgment collected for your job, as well as all source and Gold Standard Data included in the original unit.Each judgment will also contain the Trust Rating for the contributor that provided this judgment. The Trust Rating reflects a contributor’s overall accuracy on Gold Standard Data in this job.
The following table shows common headers and how to interpret them.