Crowd Flower

Email Address Collection

Find Financial Aid Offices for Public Colleges and Universities

Task Setup

Our goal is to find the email addresses of financial aid departments for a sample list of colleges.

We start with a spreadsheet of 80 colleges and we create a small task for every row in the spreadsheet. We call each of these small tasks a "unit" that one or more workers will perform.

In this case, each unit requires the worker to visit the website and look for the email address of the financial aid department. You can see a sample task below:

The result is a spreadsheet containing the colleges and the email addresses of their financial aid departments.

How CrowdControl™ Helps

Gold Standard

We take a set of known items, a "gold standard", which we use to train workers. In each task, we mix in the known items with unknown items to test how well workers are doing. If a worker makes a mistake on one of our known results, we can notify and show the worker where they erred, thereby training the worker to do the task better.

This training mechanism, with a gold standard, gives us better results as shown in the following graph:

The error rates without the gold standard is more than twice as high as when we do use a gold standard. It helps in two ways: 1) it improves worker accuracy by showing them exactly what results are expected, and 2) it allows our system to quickly determine who is giving the highest quality results.

Confidence Rating

Our system computes a confidence score on every result. You can use the confidence score to determine which results have lower chances of errors or ambiguity.

We compute the confidence score from: 1) the workers' historical accuracy and 2) the level of agreement among workers on the task.

You can see from the above graph, that answers with higher confidence tend to have fewer errors. For example, on this task, results with confidence values above 0.5 had no errors. Of course, if we collected more data, we might find a small number of errors in the higher confidence answers.

This graph also shows that training workers with a gold standard improves our accuracy over a larger range of confidence values. The solid gold line shows the accuracy of our system with three workers including gold data, while the dashed green line shows the accuracy of our system with three workers and no gold data. Another great side effect of gold data is an overall increase in the confidence scores of all the data.