API

The CrowdFlower API gives developers the ability to build applications that interact with and use all the features of CrowdFlower in an automated fashion. Tasks can be generated, work can be ordered, and your application can be notified as data is processed and judged by the CrowdFlower platform. The methods and practices described in this documentation are subject to change as the CrowdFlower API matures.

CrowdFlower uses a RESTful API. It accepts data as URL-encoded key value pairs. Authentication is currently key-based. Responses are currently restricted to the JSON format.

Overview

URLs

All URLs begin with the following pattern:

https://api.crowdflower.com/v1/...

Authentication

CrowdFlower uses key-based authentication. You can obtain or change your key in the account pane. To maintain security keep the key secret and use it only over SSL. Each request to the API must contain a key parameter with your account key.

https://api.crowdflower.com/v1/jobs?key=1a2b3c4d5e6f7g8h9i0j

Formats

The API currently supports JSON only. You must set the HTTP header Accept: application/json or append .json to the URL of the request. For example:

https://api.crowdflower.com/v1/jobs.json?key=1a2b3c4d5e6f7g8h9i0j

Messaging

If the status code of a response is something other than 200, you will find a JSON message in the body of the response. Messages are used to give you the status of a given operation, to report an error. CrowdFlower can return three types of messages:

{success: {message: "Job created successfully."}}
{notice: {message: "Your job is being completed."}}
{error: {message: "Job could not be canceled."}}

Webhooks

Webhooks allow your system to receive notifications when interesting events happen to the jobs you create on CrowdFlower. Select the Webhooks link at right to learn more.

REST

The API supports the four standard HTTP methods, each altering the behavior of a given URL. These are referred to as VERBs in the rest of the documentation.

Status Codes

Reference Implementations

We currently maintain a Ruby gem that implements the functionality described in this documentation. You can view and contribute to the code at https://github.com/dolores/ruby-crowdflower.

Questions?

We are actively working on our API and would love to hear from you! Drop us a line at support@crowdflower.com.

Uploading Data

CrowdFlower currently provides three import options:

• Bulk Upload (JSON)
• Data Feeds (RSS, Atom, XML, JSON)
• Spreadsheets (.csv, .tsv, .xls, .xlsx, .ods)

Methods

Bulk Upload (JSON)

The JSON bulk upload action is the preferred method for importing data into a CrowdFlower job. You can upload a collection of JSON objects in the same way that you would upload a spreadsheet, and the JSON file can contain any number of JSON objects. Each of these JSON objects will be imported into your CrowdFlower job as one unit of work (this is the equivalent of one row of data in a spreadsheet import).

Content Type

When uploading JSON data in bulk, set the HTTP header Content-type: application/json.

Parameters

If you specify a {job_id}, the data submitted will be appended to the existing job. If you don’t specify a {job_id} in the request, a new job will be created.

Response

A successful request will return a newly created job structure.

Sample JSON File

In this example, the first JSON object has three columns and the second only object has two. When a CrowdFlower job is created, the first object will be used to set the column headers. If any object after the first has a key-value pair whose key does not match one of the first object’s keys, that pair will not be included. If an object after the first is missing a key that was in the first object, you will see a blank cell.

{
  "column_1":"You say goodbye",
  "column_2":"And I say hello",
  "column_3":"Hello, hello"
}
{
  "column_1":"I don't know why you say goodbye",
  "column_3":"I say hello"
}

cURL Examples

curl -T 'sampledata.json' -H 'Content-Type: application/json' https://api.crowdflower.com/v1/jobs/upload.json?key={api_key}

curl -T 'sampledata.json' -H 'Content-Type: application/json' https://api.crowdflower.com/v1/jobs/{job_id}/upload.json?key={api_key}

Result

This job was created by uploading the sample JSON file.

Data Feeds

If your data is accessible by a public URL, CrowdFlower can pull the data and load it into a job for you. This is a great way to analyze public RSS feeds or data in Google Data Protocol’s JSON format. Currently supported formats are RSS 1.0, RSS 2.0, Atom, generic XML, and JSON.

Parameters

You can submit a feed for seeding your job with data by setting the following post parameter:

If you specify a {job_id}, the data submitted will be appended to the existing job. If you don’t specify a {job_id} in the request, a new job will be created.

Response

A successful request will return the job that the data was loaded into.

cURL Examples

curl -d 'job[uri]=https://search.twitter.com/search.atom?q=helloworld' https://api.crowdflower.com/v1/jobs/upload.json?key={api_key}

curl -d 'job[uri]=https://search.twitter.com/search.atom?q=helloworld' https://api.crowdflower.com/v1/jobs/{job_id}/upload.json?key={api_key}

Spreadsheets

Content Type

You can upload a spreadsheet by setting the body of the request to the spreadsheet content and setting the Content-Type header to one of the following supported formats:

• application/vnd.oasis.opendocument.spreadsheet
• application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
• application/vnd.ms-excel
• text/csv
• text/plain

Parameters

If you specify a {job_id}, the data submitted will be appended to the existing job. If you don’t specify a {job_id} in the request, a new job will be created.

Response

A successful request will return the job that the data was loaded into.

cURL Examples

curl -T 'sampledata.csv' -H 'Content-Type: text/csv' https://api.crowdflower.com/v1/jobs/upload.json?key={api_key}

curl -T 'sampledata.csv' -H 'Content-Type: text/csv' https://api.crowdflower.com/v1/jobs/{job_id}/upload.json?key={api_key}

Jobs

A Job is the basis for all other resources. It contains all of the information needed to display the task in one of our labor pools and to aggregate the results once they have been collected. With CML it defines output judgement structure as well as the way job input data is represented.

Attributes

Methods

Common

Parameters

job[title]=Some+title&job[instructions]=Instructions

Copy

Copying jobs is great way to group your work into “batches.”

Parameters

Response

A successful request will return the job that the data was loaded into.

Pause

You can pause a job at any time if you want to temporarily stop judgments from coming in.

Response

A successful request returns the job that was paused.

Resume

You can resume a paused job at any time.

Response

A successful request returns the job that was resumed.

Cancel

You can cancel a job at any time if you want to permanently stop judgments from coming in and refund your account for any judgments not yet received.

Response

A successful request returns the job that was paused.

Status

You can check the status/progress of your job at any time with this method.

Response

{
  "all_judgments": 50,
  "golden_judgments": 5,
  "tainted_judgments": 15,
  "needed_judgments": 200,
  "all_units": 100,
  "golden_units": 10
}

Legend

The legend will show you the generated keys that will end up being submitted with your form.

Response

{
  "phone_number": "Phone number",
  "website_url": "Website URL",
  "email_address": "Email address"
}

Adding / Removing Gold

You can add or remove gold-quality checking units via this API call.

Parameters

Response

Upon success you will be informed of the number of units flagged as gold, and the unique set of values in your gold set:

{
  "affected": 5, 
  "populated": ["(123) 456-7890", "(800) 111-1111"]
}

Creation of the gold can fail for a few reasons. If a gold set could not be created for the specified column, the response status will be 400, and the body will be one of the following:

{
  "error": {
    "message": "We couldn't find any prepopulated gold standard in your dataset."
  }
}

{
  "error": {
    "message": "We didn't mark any rows as gold because marking 100 rows would have affected over 1/3rd of them."
  }
}

These errors are related to the number of units with non-empty values in your data set. We generally recommend having approximately 10% of your entire data set flagged as gold. More than 33% is currently not allowed.

Channels

You can view and update the job’s channels via this API call.

Parameters

Response

The GET request will return a JSON response that lists the available and enabled channels for this job:

{
  "enabled_channels": [
      "amt"],
  "available_channels": [
    "amt",
    "sama",
    "gambit",
    "mob",
    "iphone"]
}

A PUT request allows you to set the channels for this job:

curl -d 'channels[]=amt&channels[]=mob' https://api.crowdflower.com/v1/jobs/{job_id}/channels?key={api_key}

The response will contain a success or error message, the list of available channels, and the list of enabled channels.

{
  "success":"The job has been updated.",
  "enabled_channels":["amt","gambit","mob","sama"],
  "available_channels":["amt","gambit","iphone","mob","sama"]
}

{
  "error":"The channel amtt could not be found.",
  "enabled_channels":[],
  "available_channels":["amt","gambit","iphone","mob","sama"]
}

Units

A Unit represents information you want annotated. If you were to upload a spreadsheet of data, each row would be turned into a Unit. Each Unit for a particular Job should contain the same data. A Unit must belong to a Job.

When Units are created, they are not automatically ordered by default. If you want all Units added to a Job to be automatically ordered, set auto_order to true on your Job.

Attributes

Results

Any unit that has gathered trusted judgments will also have a results attribute.

results has a top-level attribute judgments, which contains a JSON array of all the Judgments that have been gathered so far during your job.

{
  "results": {
    "judgments": []
  },
  "created_at": "2010-05-03T14:04:31-07:00",
  "data": {
    "name": "pirate radio"
  },
  "updated_at": "2010-05-03T14:04:31-07:00",
  "judgments_count": 0,
  "id": 735275,
  "difficulty": 0,
  "job_id": 124056,
   "message": {
      "error": "Unit was successfully created, but it could not be ordered because channels have not been set for this job."
  },
  "state": "new"
}

Any aggregate calculations that are done on your data will also be included in results. For example, if your job asks workers to visit a website and answer a series of questions about the website, your results structure might look like this:

{
  "results": {
    "what_is_the_url": { 
      "agg": "https://blog.crowdflower.com",
      "confidence": "0.9" 
    }
  }
}

See aggregation for more details on the types of aggregations that are possible.

Methods

Common

Parameters

unit[golden]=true&unit[data][somekey]=Some+value

If auto_order is enabled, the unit will be ordered as soon as it is created. If you have not set channels for this job, however, the unit will be saved successfully but will return the following error:

{
  "results": {
    "judgments": []
  },
  "created_at": "2010-05-03T14:04:31-07:00",
  "data": {
    "name": "pirate radio"
  },
  "updated_at": "2010-05-03T14:04:31-07:00",
  "judgments_count": 0,
  "id": 735275,
  "difficulty": 0,
  "job_id": 124056,
  "message": {
    "error": "Unit was successfully created, but it could not be ordered because channels have not been set for this job."
  },
  "state": "new"
}

Status

If you are adding units from a spreadsheet or a feed, you can use ping to determine the status of the upload:

Response

Pending / processing:

{
  "done": false,
  "count": 100
}

Finished:

{
  "done": true,
  "count": 200
}

Failed:

{
  "done": false,
  "count": 55,
  "error": "Unable to parse row 57. Please be sure your file is UTF-8 encoded."
}

Although multiple queued uploads are allowed, ping only returns the status of the most recent upload.

Cancel

If a unit has been ordered, you can use this method to cancel an individual unit. This action does not refund your account.

Response

{"success":"Unit 1038109 has been cancelled."}

In case of an error processing the request, the error will appear in a JSON key called error:

{"error":"Job 6245 does not contain Unit 1038109"}

cURL Example

curl -X PUT https://api.crowdflower.com/v1/jobs/6245/units/1038109/cancel.json?key={api_key}

Bulk Split

Parameters

Response

A successful request will return a 200 “OK” response.

Explanation and Example

Note: You are strongly advised to post complex data formats to CrowdFlower using JSON rather than CSV. JSON is better suited for complex data — the split operation is provided as a convenience operation for existing datasets, and it is not recommended for new users.

If you upload a spreadsheet that has more complex data than a CSV can naturally represent, the split operation can be useful. Any of the columns of your uploaded CSV may be internally delimited by a special character (by default, a blank space). Calling split on this column will let CrowdFlower know that the contents of this column should be treated by CrowdFlower internally as a collection of discrete items rather than a block.

Suppose your existing dataset is an arbitrary collection of major authors.

author,major_works,countries_active
Homer,The Iliad|The Odyssey,Greece
Dickens,David Copperfield|Bleak House,England
Nabokov,Camera Obscura|Lolita,Russia|United States
Rabelais,Gargantua and Pantagruel,France
Cervantes,Don Quixote,Spain

When this data is posted as a CSV to CrowdFlower, one unit is created for each of the five rows of data. The units each have data associated with the three CSV columns provided. When initially posted, CrowdFlower treats all of the values transferred as free text values with no depth or structure. After the initial data post, Dickens’ major works field is set to David Copperfield|Bleak House

To let CrowdFlower know that the major_works and countries_active columns are each actually collections of delimited values, you can use the split operation.

PUT https://api.crowdflower.com/v1/jobs/{job_id}/units/split?on=major_works,countries_active&with=|

(Be careful to URL-encode the parameters — this is not done above for readability.)

After the PUT, CrowdFlower will consider Dickens’ major_works field to be set to the collection [ "David Copperfield", "Bleak House" ]. Similarly, Nabokov’s countries_active field will be set to [ "Russia", "United States" ]. The square brackets here indicate a data structure that is analogous to a List or Vector in Java, a list in Python, an Array in Ruby, etc. If you were to request Homer’s major_works from CrowdFlower, it would be returned as a JSON array:

{major_works: [ "The Iliad","The Odyssey" ]}

Because the author field was not split, it will not be treated as a collection:

{author: "Homer"}

Judgments

A Judgment represents data collected from our labor pool. Each Unit can have multiple Judgments. An aggregate structure can be queried with meta information, and the best answer is chosen from multiple Judgments.

Attributes

Methods

Common

Parameters

judgment[data][some_key]=Some+value

Bulk Download

A CSV of all judgments collected for a given job is available.

Parameters

Response

It can take up to a minute to generate a bulk download, depending on the point in the job cycle that you make the request. If we need to generate or re-generate the CSV for you, the status of the response will be 202 and the body will be as follows.

{ 
  "notice": {
    "message": "Your CSV is being generated.  Try again in 10 seconds."
  }
}

Orders

An Order must be placed for a Job to collect Judgments. When you create a new debit order, you can specify how many units you would like fulfilled for the Job. You must have enough money in your account to create an Order.

Attributes

Methods

Common

Parameters

Response

If you do not have enough credits to create an order, the response status will be 402 and the body will be:

{
  "error": 
    {"message": "You have $20.00 available but this order will cost $50.00.  You can purchase more at https://crowdflower.com/orders/new"}
}

If the order creation is successful, you will be redirected to the new order and the body of the redirect will be:

{
  "success": 
    {"message": "Your job is now collecting judgments!"}
}

Examples

A simple example of creating an order for Amazon Mechanical Turk and Samasource using the common command-line tool “wget:”

wget --post-data="key={api_key}&debit[units_count]=18&channels[]=amt&channels[]=sama" https://api.crowdflower.com/v1/jobs/{job_id}/orders

Webhooks

Webhooks provide callbacks between your web application and CrowdFlower. This integration point allows developers working with CrowdFlower to create robust applications that interact with CrowdFlower and take actions based on the judgments that the CrowdFlower platform provides.

Basics

Webhooks are enabled by setting the webhook_uri property for a Job.

When a webhook is enabled for a Job, CrowdFlower will POST to that URL whenever an interesting event happens to your Job. See the Events section below for details on exactly which events trigger webhooks.

Each POST that your app will receive will have two parameters: signal and payload.

The signal parameter describes the type of event that has occurred. See below for the types of events and signals that will be sent.

The payload parameter will contain a JSON representation of the associated object. For example, when your job completes, you will receive the job_complete signal, and the associated payload will be a JSON representation of that Job.

unit_complete is the most frequently used webhook. The Unit payload includes all of the Judgments for the work completed, as well as additional useful aggregate statistics. See the Units documentation for details on its payload.

Note that the unit_complete event does not apply to gold units. To manually send webhooks for gold units, make a GET request to

https://api.crowdflower.com/v1/jobs/{job_id}/golds/fire_webhooks

Error Handling

If CrowdFlower encounters difficulties reaching your server for more than a few hours, it will automatically disable the webhook and send you an email informing you of the problems it has encountered with the URL you have provided. To keep the Webhook up and running correctly, please ensure that your server responds to each POST from CrowdFlower with an HTTP 200 (OK) header. The body of your response will be ignored.

You can re-enable the webhook by saving it again in the job’s Advanced Options. As soon as the webhook is re-enabled, all completed but unsent units will be posted to your server.

Events

Guidelines

As our API matures, we will add support for more events. When you handle the POSTs from CrowdFlower, be sure your system responds gracefully with a 200 (OK) to new signal types. A future version of the API will allow subscribing and unsubscribing from different event types independently.

cURL Examples

You can find your API key on the account page.

Create a new job without data:

curl -d 'job[without_data]=true' https://api.crowdflower.com/v1/jobs.json?key={api_key}

Create a new job by uploading a JSON file:

curl -T 'sampledata.json' -H 'Content-Type: application/json' https://api.crowdflower.com/v1/jobs/upload.json?key={api_key}

Create a new job and populate it with data from a csv file:

curl -T 'sampledata.csv' -H 'Content-Type: text/csv' https://api.crowdflower.com/v1/jobs/upload.json?key={api_key}

Add data to an existing job by uploading a JSON file:

curl -T 'sampledata.json' -H 'Content-Type: application/json' https://api.crowdflower.com/v1/jobs/{job_id}/upload.json?key={api_key}

Add data to an existing job by uploading a csv file:

curl -T 'sampledata.csv' -H 'Content-Type: text/csv' https://api.crowdflower.com/v1/jobs/{job_id}/upload.json?key={api_key}

Pull data from a Twitter feed and add it to a new job:

curl -d 'job[uri]=https://search.twitter.com/search.atom?q=helloworld' https://api.crowdflower.com/v1/jobs/upload.json?key={api_key}

Pull data from the Twitter feed and add it to an existing job:

curl -d 'job[uri]=https://search.twitter.com/search.atom?q=helloworld' https://api.crowdflower.com/v1/jobs/{job_id}/upload.json?key={api_key}

Create a new job with title “Some title” and “Instructions”:

curl -d 'job[title]=Some+title&job[instructions]=Instructions' https://api.crowdflower.com/v1/jobs.json?key={api_key}

Change the title of a job (must be 5 to 255 characters):

curl -X PUT --data-urlencode "job[title]=some new title" "https://api.crowdflower.com/v1/jobs/{job_id}.json?key={api_key}"

Get all jobs in JSON format:

curl https://api.crowdflower.com/v1/jobs.json?key={api_key}

Get your account information:

curl https://api.crowdflower.com/v1/account.json?key={api_key}

Order a job:

curl -d 'key={api_key}&channels[0]=amt&channels[0]=sama&debit[units_count]=20' https://api.crowdflower.com/v1/jobs/{job_id}/orders.json

Pause a job:

curl https://api.crowdflower.com/v1/jobs/{job_id}/pause.json?key={api_key}

Resume a job:

curl https://api.crowdflower.com/v1/jobs/{job_id}/resume.json?key={api_key}

Cancel a job:

curl https://api.crowdflower.com/v1/jobs/{job_id}/cancel?key={api_key}

Check the status of a job:

curl https://api.crowdflower.com/v1/jobs/{job_id}/ping.json?key={api_key}

Delete a job:

curl -X DELETE https://api.crowdflower.com/v1/jobs/{job_id}.json?key={api_key}

Copy a job with no units:

curl "https://api.crowdflower.com/v1/jobs/{job_id}/copy.json?key={api_key}"

Copy a job with only its gold units:

curl "https://api.crowdflower.com/v1/jobs/{job_id}/copy.json?key={api_key}&gold=true"

Copy a job with all of its units:

curl "https://api.crowdflower.com/v1/jobs/{job_id}/copy.json?key={api_key}&all_units=true"

Get the legend of a job:

curl https://api.crowdflower.com/v1/jobs/{job_id}/legend.json?key={api_key}

Set payment per assignment (set-only, cannot view the from the API):

curl -X PUT --data-urlencode 'job[payment_cents]=1000' --data-urlencode 'key={api_key}' https://api.crowdflower.com/v1/jobs/{job_id}.json

Exclude workers from certain countries:

curl -X PUT -d 'job[excluded_countries][]=US&job[excluded_countries][]=AD' https://api.crowdflower.com/v1/jobs/{job_id}.json?key={api_key}

Set auto_order to true:

curl -X PUT --data-urlencode 'job[auto_order]=true' --data-urlencode 'key={api_key}' https://api.crowdflower.com/v1/jobs/{job_id}.json

Delete a unit:

curl -X DELETE https://api.crowdflower.com/v1/jobs/{job_id}/units/{unit_id}.json?key={api_key}

Update a unit:

curl -X PUT --data-urlencode 'unit[data][choose_one_gold][]=Yes' --data-urlencode 'unit[data][choose_one_gold][]=No' "https://api.crowdflower.com/v1/jobs/{job_id}/units/{unit_id}.json?key={api_key}"

Change the state of a unit:

curl -X PUT --data-urlencode 'unit[state]=new' --data-urlencode 'key={api_key}' https://api.crowdflower.com/v1/jobs/{job_id}/units/{unit_id}.json

Set units per assignment:

curl -X PUT --data-urlencode 'job[units_per_assignment]={n}' --data-urlencode 'key={api_key}' https://api.crowdflower.com/v1/jobs/{job_id}.json