Requester Interface
This page is intended to provide a basic overview of the CrowdFlower user interface. Here you can learn to navigate the interface and access the many features and tools available to you.
Dashboard
http://crowdflower.com/jobs/{job_number}The dashboard is the best location to view your job’s progress and overall status.
- Unit Count
- The total number of units in your job. If you were to upload 18 rows of data, your unit count would display 18 units. If you do not upload data, your job will show 1 unit by default.
- Gold Count
- The total number of active Gold units in the job. Gold units are included in the total unit count.
- Judgments Needed
- The total number of judgments needed in the job. By default, the job is set to collect 3 judgments per unit. This can be altered when calibrating prior to running the job.
- Pending
- The number of judgments needed for the job to complete. This number will fluctuate due to contributors occasionally transferring from trusted to untrusted. This is only relevant when a job is running.
- Trusted and Untrusted
- The total number of trusted and untrusted judgments collected in the job. Please visit our Gold documentation to learn more about trusted and untrusted judgments.
- Actions
- This drop-down menu (displayed as “Not Running” prior to running the job) allows you to order judgments, copy, delete, and cancel your job. After the job begins to acquire judgments, you will be able to download the trusted judgments acquired in aggregated and full CSV formats. Note, reports can only be generated once while the job is running. Your reports will automatically be regenerated once the job completes.
Data Page
http://crowdflower.com/jobs/{job_number}/unitsThis page allows you to view the dataset you have uploaded. It is recommended that you verify that your data was uploaded correctly. Note that each of your units have been assigned a unique unit_id. This id will be used to keep track of the unit throughout the completion process.
- Add Data
- Allows you to add new data to a job that already contains a dataset. Your new data must be in the same format as the previous data, including headers.
- Delete Data
- You cannot currently delete columns or rows in a pre-existing dataset. Selecting this button will automatically copy your job to a new job that will allow you to upload a new/altered dataset.
- Split Column
- Allows you to define a delimiter that will be used to parse out cells within a specified column into separate units. E.g. you wish to delimit 10 options found within a column’s cells in your spreadsheet. If the the 10 options were on separate lines within the cell, you could delimit the cells with a newline character. This will result in 10 separate units per row when regularly the cell would be considered one unit.
Edit Page
http://crowdflower.com/jobs/{job_number}/editThe Edit page allows you to create and edit the form that your contributors will interact with. There are currently two options available to you: the graphical editor and the CML editor.
- Graphical Interface
- The simplest way to create your form. This editing interface is most appropriate for simple jobs that will not contain complex features such as logic, custom formatting, JavaScript, etc.
- Title and Description
- Contributors will first see your title. If compelling, they may choose to view your description/instructions and accept the job. It is recommended that you be as concise and clear as possible in your instructions. Examples are highly recommended.
- Apply Your Data
- Found directly under the description, select this box to apply the data you have uploaded. This information will be dynamic in your job (e.g. a unique URL displayed with every unit). Text is permitted in this area.
- Add Questions
- Select this box displaying “build form” to add your form’s questions from a list of question types. No coding required!
- CML Editor
- CML (CrowdFlower Markup Language) allows you to use code to implement special contingencies (logic, HTML, Javascript, CSS, etc.) in your form. You can learn more about CML by reading the CML documentation. If you choose to edit with CML, we insist that you read this documentation.
- Advanced Options
- Allows you to set specific requirements for your job such as location restrictions, contributor restrictions, keywords, etc.
Preview Page
http://crowdflower.com/jobs/{job_number}/previewThe preview page displays your form from a contributor’s perspective. Test all form logic, JavaScript, CSS, etc. here prior to running your job. If the form is not functioning correctly in the preview interface, it will not function correctly for the contributors. Use “Test Validation” to test any validators that have been added to your questions.
Gold Page
http://crowdflower.com/jobs/{job_number}/goldsThe Gold page allows you to view and manage your Gold Standard Data. Take a look at our Gold documentation for a detailed explanation of Gold.
- Create Gold
- This button directs you to the Gold creating interface.
- Gold Report
- The best way to obtain a broad perspective of the health of your Gold Standard Data. Here you may sort the gold listings by the number of contentions that contributors have written when missing a unit, as well as by the percentage of judgments missed. Gold units with low agreement may indicate that the unit is faulty.
- Upload Gold CSV
- This button allows you to upload edits made to your Gold Data in CSV format. The safest and easiest practice is to modify units one by one from the Gold page, but if you choose to upload via a Gold CSV, be sure that your columns do not change. If you leave a field blank in the CSV, it will delete any existing data.
You can download a spreadsheet version of your Gold units in CSV format with the following URL:
http://crowdflower.com/jobs/{job_number}/golds/report.csv No source data will be included in this file.
Contributors Page
http://crowdflower.com/jobs/{job_number}/contributorsThis page contains information on all contributors that have participated in your job. Trusted contributors will be displayed in green. Untrusted contributors will be displayed in red.
- Show Details
- Displays information on specific contributors including answer distribution and missed Gold units. You can forgive a contributor’s missed Gold units as well. This forgiveness only applies to the individual contributor.
- Flag Contributor
- Removes contributors that are not performing to your standards. This will not remove their judgments from your results. Typically, you should let your Gold remove poor contributors (and their judgments) from the job.
Calibration
The calibration page allows you to set all of the order parameters for your job; including the number of units displayed per page, the contributor pay per page, and the labor channels that will work on your job. Please make sure to thoroughly check your form, Gold units, and data prior to commencing this step.
- Judgments Per Unit
- The number of judgments that each unit will obtain. Even trusted contributors can make mistakes, but if multiple trusted contributors submit responses on the same unit, it is more likely that the correct response will be submitted. Aggregation will display the most agreed upon answer.
- Units Per Page
- Contributors see one to several units at a time. A page is a collection of units that a contributor must complete before submitting their answers. When you have Gold, a gold unit will be randomly inserted in the page. This allows you to control how much gold the contributors will see. If you would like contributors to see 1 Gold Unit for every 3 Source Units, you would set Units Per Page to 4. In this case, contributors will complete four units, submit their responses, and receive notification when/if they miss the gold unit hidden within the page.
- Pay Per Page
- The amount a contributor is compensated for the equivalent of one page of units.
