Modeling Crowdsourcing for Cultural Heritage

Julia Noordegraaf, University of Amsterdam, The Netherlands, Angela Bartholomew, University of Amsterdam, The Netherlands, Alexandra Eveleigh, University College London, UK


Despite the widespread prevalence of crowdsourcing projects in the cultural heritage domain, not all initiatives to date have been universally successful. This study has revealed that the conditions in which projects are realized, and the design properties of those projects, have a significant impact on success or failure rates. Through a literature analysis and close study of two cases—Red een Portret (Save a Portrait) at the Amsterdam City Archives and a photo-tagging project of the Maria Austria Institute on the Vele Handen (Many Hands) crowdsourcing platform—this study identifies those conditions and implements them in the design of a model to assist in the design and evaluation of effective cultural heritage crowdsourcing projects.

Keywords: crowdsourcing, cultural heritage, photo-tagging, conditions, design properties, modeling

1. Introduction

Recent years have seen numerous practical experiments by museums and other cultural heritage institutions in exploring the potential of crowdsourcing as a means of promoting increased public participation in core tasks such as collecting, describing, categorizing, or curating heritage collections. Despite some uneasiness expressed by commentators with both the terms ‘crowd’ and ‘outsourcing’ (Brabham, 2008; Owens, 2013; Eveleigh, 2014), the idea that the public might help create or improve information on collections has clear organizational appeal in an age of austerity. Crowdsourcing is also claimed as a valuable platform for increasing audience engagement with cultural heritage (Ridge, 2013), enabling new access routes to cultural collections that can be shared, recommended, remixed, embedded, and cited as an integral part of the Web. This viewpoint draws strongly upon broader societal and cultural trends in which collaborative creativity is flourishing (Shirky, 2010), and is underpinned by postmodernist theories that endorse the democratization of professional, expert practice by fostering multiple public interpretations around the medium of cultural heritage content (Evans, 2007).

Driven by such practical pressures and societal trends, crowdsourcing seems likely to become a permanent feature of the workflow of heritage institutions. However, not all crowdsourcing initiatives in the cultural heritage domain to date have been universally successful. For instance, some have failed to recruit adequate numbers of participants or those with the requisite skills to complete the desired task, as has been the case with the transcription of a collection of officer’s logs (1811–1940) from the Dutch National Military Museum on the Vele Handen platform. Others have faced challenges of incompatible organizational and volunteer cultures, such as the Maria Austria Institute photo-tagging project described below. Yet other projects have struggled to justify the ongoing organizational resources required to sustain participants’ endeavors, as was the fate of the National Archives’ (UK) wiki site, Your Archives, which quietly went offline in 2012, transferring the gathered content into a read-only site.

Given the variable success rate of crowdsourcing projects, the question that presents itself is: What decides the success or failure of a crowdsourcing project? Which factors, or conditions, determine whether a project is successful in terms of reaching its goals within a set time frame, and others less so? Finally, is it possible to organize those conditions in a model that can be instructive in designing more effective projects? These questions have been investigated in the context of the research project Modeling Crowdsourcing for Cultural Heritage (MOCCA), a collaboration between the University of Amsterdam’s Centre for Cultural Heritage and Identity, the Amsterdam City Archives, and Picturae, a creative industry company that specializes in digitizing cultural heritage collections. Besides a review of extant literature on crowdsourcing for cultural heritage, it comprised a detailed study of two cases: Red een Portret (Save a Portrait) at the Amsterdam City Archives and a photo-tagging project of the Maria Austria Institute on the Vele Handen (Many Hands) crowdsourcing platform (, developed by the Amsterdam City Archives and Picturae. The results have led to the development and testing of a model for designing and evaluating crowdsourcing projects for cultural heritage.

2. Conditions for success or failure

One of the principal difficulties for both the practical design and academic study of crowdsourcing ventures is that the term ‘crowdsourcing’ is itself ill-defined and has been applied inconsistently, such that almost any Web-based activity that involves the public may be labeled as crowdsourcing (Estelles-Arolas & Gonzalez-Ladron-de-Guevara, 2012). It is a common misconception that crowdsourcing in cultural heritage is commensurate with Web 2.0 user-generated content projects, which lack a pre-defined organizational goal or target participant range (Ridge, 2013).

Several early studies of crowdsourcing in the cultural heritage field tried to define and classify projects according to the task performed, or the purposes for which such projects are being used by cultural heritage organizations (Oomen & Aroyo, 2011; Dunn & Hedges, 2012; Van Vliet et al., 2013). This typological literature offers a broad overview of the field, providing generalizations about the factors that shape crowdsourcing projects, but rarely discusses the impact of project design (the conditions, attributes, or characteristics of crowdsourcing projects) on project success or failure. Oomen and Aroyo (2011), for example, argue that crowdsourcing has the potential to help build more accessible, connected, and engaged cultural heritage organizations, and suggest how it can be inserted into the workflow of institutions. Yet they do not go further to offer a practical model for how such a project may be developed and implemented to ensure it is viable, successful, and sustainable. Additionally, existing typologies focus on participation to the neglect of the “organizational and macrostructural properties that are important to designing and managing effective projects and technologies” (Wiggins & Crowston, 2011). An exception is Holley (2010), who provides a step in this direction by proffering a checklist of ‘tips’ for libraries to follow in the design of successful crowdsourcing projects.

While typologies are helpful to gain an understanding of the broad factors that shape the field of cultural heritage crowdsourcing, case study-based analysis is more applicable to answering questions related to effective project design. Through an applied discussion of conditions, these case studies (which feature an in-depth analysis of one study or a comparison of several related cases) provide valuable insight into the factors that determine the relative success of crowdsourcing projects. Case studies help to answer questions concerning: the type of crowd and the ways in which they participate (Holley, 2009); what motivates, incentivizes, or discourages participation (Arends et al., 2012; Eveleigh et al., 2013, 2014); the impact of relating clear research goals to the public (Romeo & Blaser, 2011); how crowdsourcing fulfills aspects of the institutional mission statement (Noordegraaf, 2011; Ridge, 2012; Owens, 2013; Eveleigh, 2014); or how game play impacts participation (Flanagan & Carini, 2012). But those case studies that reach publication are more likely to report upon successful examples than relate project failures. Consequently, it can be difficult to ascertain why certain attributes appear to have impacted project success or failure, or to generalize beyond any specific context. Also, to date there is no analysis that looks holistically at the different attributes and conditions of these studies as dimensions that may simultaneously have both positive and negative effects upon a project’s success. For example, uniformity in the collection may simplify the design of the crowdsourcing interface but also limit the appeal of the task for the public; the existence of an already engaged audience might guarantee the recruitment of ready and willing volunteers, but their relatively expert interest may jeopardize hopes of opening up a collection to a wider public that may be too intimidated to participate on a more casual basis.

A comprehensive view of the impact of these project attributes and conditions on the outcome of project success, and of the relationships that exist between them, is lacking and presents itself as a logical next step in research.

3. Methodology

The research had the double aim of, on one hand, identifying the conditions for success or failure of crowdsourcing projects and, on the other, organizing these conditions in a model that demonstrates their implications for the design and evaluation of crowdsourcing projects.

The identification of the project attributes and conditions that might determine the relative success or failure of crowdsourcing projects took place in two steps. First, we conducted a review of literature on crowdsourcing for cultural heritage to identify specific attributes or conditions that play a role in a project’s success, for example: the type of collection, size of the institution, complexity of the task, or choice for a platform versus a dedicated project site.

Secondly, in order to help determine the conditions’ relative weight and impact, we conducted an analysis of two current crowdsourcing projects: Red een Portret (Save a Portrait) and the Maria Austria Institute photo-tagging project. Both projects were in their initial launch phases and therefore would be ongoing through the period of our analysis, allowing for live data collection and interviews with project planners and participants. The two cases are similar enough to be compared: both are photo-tagging projects and take place in an archival context. At the same time, the projects differ enough to test the effect of a range of conditions: Red een Portret is dedicated to one specific collection and genre (the Merkelbach photo studio portrait collection), is hosted on a dedicated project site (, and combines the crowdsourcing of descriptions (names, dates, keywords, stories) with crowdfunding and offline events (exhibition, catalogue). The Maria Austria Institute project comprises a more heterogeneous collection (high-quality photography of a wide variety of subjects), focuses on one task (description), and is hosted on the Vele Handen crowdsourcing platform. The purpose and design of the two case-study projects were verified through interviews with project leaders and designers and on the basis of project-related materials. A quantitative analysis of the data generated between April 8, 2013, and January 21, 2014 (for Red een Portret), and between July 10, 2013, and January 21, 2014 (for the Maria Austria Institute project) yielded insights into the specific behavior of participants.

The outcomes of these two steps were then combined in a model that groups the conditions, design decisions, and desired outcomes in a typology that locates them at different places and stages in the workflow of heritage institutions. The model was designed in the form of a flow chart: through answering a series of questions, respondents are guided towards a set of design principles for effective crowdsourcing projects. A first version of the model was tested by archive, library, and museum professionals attending the Digital Strategies for Cultural Heritage conference (Rotterdam, The Netherlands, December 2, 2013). The results of this test were used to improve the final version of the model.

4. Developing the model prototype

We began our research by building up a list of existing projects and the explicit and implicit goals of these projects as determined from previous typologies and case-based literature. Determining a series of project ‘types,’ we then matched the projects with a separate list of goals, also determined from the literature. Matching the appropriate project type with the explicit goal, however, did not get us much closer to determining how our research could guide institutions to better implement crowdsourcing projects. We realized that there is much more at stake than selecting the right type of task for the given project. Rather, we deduced that it was the conditions in which projects were realized, and the design properties of those projects, that had a greater impact on their relative success or failure.

For example, literature suggests that the amount of scaffolding built into a project, or the limitation in variability in response (through the implementation of pull-down menus rather than open-text fields, for instance), might be vital to both attract and maintain project participation (Causer et al., 2012). The complexity of the initial task, the level of requisite specialized knowledge, imprecise direction, or a lack of feedback when the task is completed are all noted project pitfalls (Ridge, 2013). What’s on the Menu?, a project to digitize the New York Public Library’s collection of historic menus, focuses participants on the relatively simple task of transcribing menu prices and item names, thereby eliminating any potential uncertainty by providing clear direction and specified entry fields. Participants are able to find the task more enjoyable and barriers to participation are minimized, leading to a higher project yield (Ridge, 2013; Holley, 2010).

While scaffolding should be considered in line with the level of complexity of the given task, it should also be in agreement with the experience of the participant with the project. The literature further suggests that while a high degree of scaffolding in the early stages of a project encourages initial participation, such scaffolding can be removed as the user becomes more adept at the task to make it more challenging and to sustain participation (Ridge, 2013; Eveleigh, 2014). Seeking to both attract and maintain participation, some cultural heritage crowdsourcing projects allow for gradation in the level of difficulty over time. Participants begin with highly scaffolded, simple tasks, and move on to more challenging ones.

The Old Weather Project is a good example of a project that has maintained a high level of participation because of the variety of different tasks available, and an ability to adapt and expand over time (Eveleigh, 2013). While Old Weather initially began as a citizen science project to transcribe weather data for the purposes of climate change research, it has evolved into a project where history is as much a focus and a motivation for the participants as the science that their participation enables. The entry-level task is to partake in a relatively simple transcription task of extracting meteorological observational data from ships’ logbook pages, but a deeper commitment to the project is encouraged via the interactions that take place on the dedicated project forum (Romeo & Blaser, 2011). Many of the more committed participants additionally undertake the editing of entire logbooks, enjoying the historical detail recounted in these day-to-day journals of life on board the ship. This is an effective approach that attracts participants by gradually introducing them to the project through guided tasks, and seeks to maintain their involvement by encouraging participants to take on larger and more challenging roles over time.

To look closer at how conditions identified from the literature play a role in shaping project success, we identified two current ongoing projects to which we had close access, both of which involved the Amsterdam City Archives and Picturae.

5. Analysis of cases

The two case studies closely followed were Red een Portret (Save a Portrait) and the Maria Austria Institute photo-tagging project, each endeavored through collaboration between the Amsterdam City Archives and Picturae. Red een Portret involves users in identifying, describing, and telling stories about the portraits of the Merkelbach Photography Studio, in celebration of the one-hundredth anniversary of its opening in 2013. The Merkelbach collection includes more than forty thousand delicate glass negatives that reside in the collection of the Amsterdam City Archives and have been digitized on the occasion of this project. The Maria Austria Institute collection comprises over seven million photographic negatives by fifty-two photographers, mostly from Amsterdam. To assist with the mass digitization and online accessibility of this archival collection, the Institute, together with the Amsterdam City Archives, designed a crowdsourcing task, Tags en Uitleg (Tags and Explanation, a pun on the Dutch phrase ‘tekst en uitleg,’ meaning ‘to explain oneself fully’), to add metadata to the collection in order to facilitate its accessibility (and searchability).

Red een Portret (Save a Portrait)

The Red een Portret project site was launched to the public on April 8, 2013. At the time provisional results were collected, in mid-January 2014, forty weeks had passed and 8,281 stored contributions by 376 different users were logged by the project (22 percent of the project goal). With the project launch in April, Red een Portret was the topic of various national televised news, radio, and newspaper features. As might be expected, this period of debut marked the largest spike in crowd participation, with 19 percent of the total contributions in the first month. This level of activity dropped off the following month and remained steady until the summer, when activity saw another drop, likely while participants were away on holiday. Contributions picked up slowly again in September, the period that marked the opening of an exhibition of Merkelbach’s photographs at the premises of the Amsterdam City Archives, and shot up near the end of the year just before and after Christmas. Several smaller spikes can also be seen on weekends, suggesting that participation in this crowdsourcing project is largely done during participants’ leisure time.

Figure 1: Red een Portret Project contribution by month 595w, 500w" sizes="(max-width: 300px) 100vw, 300px" /> Figure 1: Red een Portret Project contribution by month







As is typically seen with crowdsourcing projects, in the case of Red een Portret only a few, very engaged participants have done the majority of the work. The ten most productive participants were responsible for 75 percent of the total project entries (stories and keywords combined), with the remaining 25 percent completed by the remaining 366 participants combined. The vast majority of the participants (259 out of 376) made five or fewer separate entries (again, keywords and stories combined); by contrast, the top-ranked contributor alone made 1,806 entries. This participation thus follows a power law distribution that is common among crowdsourcing projects (Barabasi, 2002).

Figure 2: Number of users ranked by contribution (x) by number of contributions (y) 500w, 578w" sizes="(max-width: 300px) 100vw, 300px" /> Figure 2: Number of users ranked by contribution (x) by number of contributions (y)







Complimentary task types: Tags and stories

Of the 376 project participants, most (75 percent) entered keywords (tags, names, and dates) and stories, 16 percent added only keywords, and 9 percent entered only stories. Thus, users tended to add both keywords and stories, but keywords were added more often, possibly because it takes less effort and time, and previous knowledge about the photograph (or its context) is not required for adding tags. The data shows that participants who contribute many keywords tend to also contribute many stories; however, it is not the case that the most active keyword contributors are necessarily the same as the most active story contributors (the Kendall’s Tau rank computation shows a correlation of 0.48, which is significant but not very strong). Red een Portret has succeeded in attracting participants who are active in both tasks, and participants that are active in one or the other, suggesting that tagging and storytelling are complimentary task types that can coexist within a project’s design.

Looking more closely at the types of keywords being entered, we find data that shows users mainly add the names of individuals featured in the photographs, rather than tags that identify items of clothing, or contextual information. It might be expected that there would be much overlap between keywords and stories, and that the descriptive stories entered (since they relate to the individuals featured in the photographs) would also include many of the same keywords entered as tags. However, we were surprised to find that there was little overlap between the words entered in the stories and the tags linked with the same photographs. These findings suggest that tagging and storytelling are complimentary tasks that provide different types of knowledge.

Crowdsourcing and crowdfunding

Besides generating tags and stories, Red een Portret also aims to generate funding from the crowd. The project website invites the crowd to contribute financially through the purchase of printed photographs (from the newly digitized glass negatives) or by making a monetary donation. Red een Portret has been successful to date in raising €34.150 of its goal amount of €50.000. While this is a substantial amount, the vast majority of these funds were raised during a separate fundraising evening held just before the launch of the site, and just following a televised program that promoted the project. The fundraising element of the site alone has been far less successful than these more conventional fundraising modes. These findings seem to suggest that, while both activities are in support of the institution, the mode of support is fundamentally different enough that there may be reason to separate these two aims in project design. Crowdfunding may be best performed via a separate platform.

The Maria Austria Institute Project

The results of the Maria Austria Institute project provide altogether different conclusions. Hosted on the Vele Handen platform, the Tags en Uitleg project asks participants to respond to a series of questions related to the what, who, where, and when of the photograph presented and its atmospheric elements or the theme it evokes (e.g., ‘love,’ ‘flower power,’ ‘tourism’); and to provide a descriptive title. The project debuted on the Vele Handen site on July 10, 2013; thirty weeks later, metadata had been added to 7,448 photographs (48 percent of the goal) by 207 participants.

Participation in the project, though initially slow, spiked around the Christmas months. The growth in participation has been relatively gradual, which can be explained by two factors: one, the debut of the Maria Austria Institute Project was not advertised as extensively as Red een Portret, and therefore did not generate as much response in the early days of its launch; and two, the project is hosted on a platform that comes with its own dedicated crowd of participants who regularly interact with different crowdsourcing projects on the same platform. The number of participants in the Tags en Uitleg project that are also active with other projects on Vele Handen (86 percent) far outnumber those that solely participate in Tags en Uitleg (14 percent). These statistics are particularly notable when compared to overall participation behavior on the platform. When viewed in sum, more than half of the participants on the Vele Handen platform contribute to only a single project (54 percent). This means that the Tags en Uitleg project has been particularly appealing to those already frequenting the platform. Further, while it is clear that the majority of participants to Tags en Uitleg are also contributing to other projects, there is not a significant statistical link between Tags en Uitleg participants and another specific type of project. Tags en Uitleg participants are not necessarily the same as those that otherwise only do transcription projects, or only classification projects, for instance. Therefore, it can be concluded that the Tags en Uitleg project has a wide appeal to a general Vele Handen participating audience.

While Tags en Uitleg has successfully made progress towards its goal of adding metadata to a selection of photographs from the collection, the organizational status of the Maria Austria Institute has resulted in some ethical concerns about the potential exploitation of crowdsourced labor for capital gain. The Maria Austria Institute, in collaboration with the Amsterdam City Archives, decided to open up its archives digitally to target a broader audience and embrace the possibilities for the presentation of its collection through a fully online, searchable database. Yet, despite providing access to preview quality versions of the images online, the Maria Austria Institute collection is not public domain. The paradoxical position of the Institute, moving away from a closed, for-profit photo agency into a not-for-profit, but still charging, image bank has proven confusing for prospective participants who have expressed concerns for the potentially commercial goal of their pooled labor.

Furthermore, Tags en Uitleg has a high degree of scaffolding that allows for minimal variation in response, and does not allow the crowd (through single- or multi-track methods) to monitor the accuracy of the tags (Brumfield, 2013). The notion that the crowd, rather than a specialist, would have full responsibility for the tagging of images was not acceptable to those involved in the decision making for the Institute. The solution was that professionals at the City Archives and the Maria Austria Institute would control all entries added by the crowd. This decision reveals a conflict between the self-perception of the Institute as a photography collection with a curatorial role, and the most efficient design for the project, which would rely entirely on the crowd to both contribute and monitor the addition of metadata to the images (Fleurbaay & Eveleigh, 2012).

6. Case-study conclusions

When viewed together, the two projects reveal a number of insights about the conditions that impact crowdsourcing projects. While tagging and storytelling appear to be complimentary project types, crowdsourcing and crowdfunding may not be as successful in combination. Adding fundraising to the project goals, while potentially appealing to the altruistic motivations of participants, is not likely to appeal to other motivational factors cited by crowdsourcing project participants, among them genuine interest in the material, intellectual challenge, and enjoyment of the task (Crowston & Fagnot, 2008; Causer et al., 2010).

The studies also show that it is important to know the limitations and possibilities of crowdsourcing, and what makes crowdsourcing well suited for the particular goals of a project. In the case of the Maria Austria Institute, there is some unwillingness to give up control over the description of the collection. Viewed from the alternative prospective, however, the decision not to allow the crowd to have control over the descriptive process sets a limit and goal with which the institution is comfortable.

Despite some challenges with advertising the Maria Austria Institute project, hosting it on the Vele Handen platform has proven to be a successful strategy to garner participation. Given that there are no fundraising projects on Vele Handen, the association of the project with commercial exploitation is avoided. Since there are currently eleven other active projects hosted by cultural heritage organizations, the Tags en Uitleg project, with its emphasis on the tagging of a great variety of high-quality photographs, offers a potentially refreshing alternative to the other projects, which are more transcription or classification based. Whilst hosting Tags en Uitleg on the platform seems to have bolstered its success, Red een Portret seems to be better served by its own project site. While Red een Portret also features a photographic archive and the addition of metadata, the collection itself and its relation to the larger crowdsourcing initiative of the institution (that includes also an exhibition, a catalogue, and a fundraising campaign) is more emphasized. Therefore, what can be concluded is that the decision of whether to host a crowdsourcing project on a platform should be based on a number of specificities of the project, among them the nature of the collection, number of facets that the project will take, and amount of publicity that will be generated for the project by the institution.

Additional conditions of project design, such as the level of scaffolding for the task, can also be abstracted from our case studies. In the case of the Tags en Uitleg project, the high level of scaffolding is not ideal if the goal is to gain the most broadly ranging tags possible. Yet the goal of this project does not seem to be to make the collection as open and accessible to the public as possible, but instead to make it better categorized for professional use, and for this purpose the Tags en Uitleg project is successful. Red een Portret, on the other hand, has less scaffolding. Allowing for the entry of generic tags and open fields for storytelling, the data collected will be less precise but not less valuable. Crowdsourcing metadata for image collections opens up the possibility of a broader range of terms from widely varying perspectives, thus enhancing the ability of the public to find images that might not have been accessible before (Flanagan & Carini, 2012). This closing of the semantic gap is a goal of many archival tagging projects, but it will be important for the Amsterdam City Archives to find a practical application for this new form of knowledge.

7. Modeling conditions

With our findings about effective project conditions and design strategies, we sought a format through which we could apply our findings to the crowdsourcing projects of cultural heritage institutions. Such a format would require the input of the organization, which would stimulate thinking and call for decisions to be made regarding core elements of project design. The goal of the model would be to respond by providing a list of criteria, and a given direction, for how to develop and implement a project to achieve the desired goals with the given project characteristics. Given the question-answer nature of the model, a questionnaire in the form of a flow chart, which takes respondents through the questions to eliminate certain design decisions and direct to the implementation of others, came to the fore as the most clear option. This questionnaire called for an ordered grouping of conditions that should be taken into account when designing crowdsourcing projects, which were formed into six pillars.

The first pillar is institution, which covers organizational concerns about staffing, budgets, the familiarity of the institution with projects that involve digitization, and intellectual property rights. The second pillar aims to narrow thinking down from the institutional perspective to the level of the actual or proposed crowdsourcing project, and the collection or collections that will be the focus of that project. There is a correlation between the collection that forms the basis of the project and the type of crowd that the project might hope to attract. Factors that should shape project design include the medium of this collection, how large or how complex (or standardized) it is, the collection’s likely appeal to the public, and a preexisting audience for the material. The third pillar focuses on the crowdsourcing project’s goal or goals. This notion is highly complex; in the course of analysis, the goal of project emerged as the characteristic of projects that must be most deeply mined. As the goals of projects are, ultimately, related to more than just the specific project goals (such as transcribing a collection of manuscripts), many projects include various separate aims for the institution, in relation to the patrons of the institution, the crowd the project wishes to attract, or for the collection itself. Therefore, the goal and its specificity as related to the institution (and the field) is the most highly nuanced area of the model.

The next pillar relates to the crowd of volunteers the project aims to attract. How these volunteers will be recruited, how to keep them interested and motivated, how they will be trained, and the value of their experience are all notions that are included in this section. Like the project’s goal, the crowd that is involved in the project is a complex area of study, with many different causal factors shaping the amount and length of crowd involvement. Research has shown, for example, that factors that shape a change in motivation are related to personal interests as well as external factors such as attribution and acknowledgment (Rotman et al., 2012). Identifying the right crowd and attracting them to the project is more important than designing the right interface (Arends et al., 2012).

The responses to these sections feed into design properties covered in the infrastructure pillar. To ensure that projects magnify user efforts, tools must put a potential user in exactly the right position, “with the right knowledge, just at the moment he or she needs it, to accomplish a given activity” (Owens, 2013). Therefore, the infrastructure of a project must be designed with concern for the variables provided in the previous sections. The complexity of the task being asked of the crowd, whether it can be broken down further into components, whether or not the user interface should be scaffolded to encourage members of the crowd, and whether a generic platform should be used to host the project are all questions that arise in the design of the project infrastructure. The final pillar is evaluation. The measures, quantitative or qualitative, that can be identified to help monitor progress and evaluate the project’s success are factored into this section. Evaluating a project’s areas of success or failure is imperative to ensuring that knowledge is incorporated from the experience of the project, and can go so far as to inflict change upon the cultural heritage field in general.

8. Tested in practice

The first version of the model was tested at the Digital Strategies for Cultural Heritage Conference in Rotterdam on December 2, 2013. In total we had eighteen respondents complete and submit a questionnaire: representatives came from museums (7), archives (6), and libraries (5).

Respondents found the questionnaire helpful in narrowing the goals of projects and thought it could be used to align stakeholders. But the respondents were at very different stages in the design of their projects, and felt that not all the questions asked were applicable. This finding has resulted in a new direction for the second version of the model, wherein the initial question asks the phase of the project’s development before leading to three different strands of questioning: an orientation phase, an implementation phase, and an evaluation phase.

Workshop delegates raised additional development areas, such as more direction about how to best communicate projects to the public, the type of PR that should be involved, and how to handle collections that contain sensitive or copyrighted material.

Finally, the workshop revealed that the initial version of the model contained a number of general principles about crowdsourcing that would be more suitable to transfer into the format of a preamble that includes ‘10 Misconceptions’ for respondents to read before taking actual questionnaire. For example, a common misconception is that the main beneficiary of crowdsourcing projects is the institution, where it should be the crowd itself (Owens, 2013). Such a preamble could help to better inform respondents before they embark upon project-specific questions.

9. Conclusion

Through literature and case-study analysis, we have successfully identifyied a broad spectrum of conditions of a crowdsourcing project’s design that have a significant impact on the success or failure of the project to reach its intended goals. Compiling these conditions into a format that would allow them to direct the design of future successful crowdsourcing projects, we have designed a responsive questionnaire, informed by the findings of previous cultural-heritage crowdsourcing projects, that poses questions to respondents about their own specific projects and provides in return suggestions for the effective design (concerning the level of scaffolding required, type of task, formulation of clear goals, use of a platform or dedicated project site, etc.). This questionnaire is most helpful in the pre-planning stage of crowdsourcing projects, allowing for reflection on the possibilities, but also limits, of crowdsourcing. It pushes institutions in the direction of clarifying the tasks they want to accomplish with the project, setting more concise end goals and ensuring that they are more likely to eventually be implemented into the workflow and made accessible to the public. As such, institutions can begin crowdsourcing projects knowing from the start where the project is headed, what type of knowledge they wish to acquire, and whether this knowledge is in line with the identity of the institution. This does not make changes impossible nor change the flexibility of the project, but it does ensure that projects do not come to a close without any consequent implementations for the knowledge gathered over the project—which would be a clear failure of any cultural heritage crowdsourcing project.


We would like to express our gratitude to Ellen Fleurbaay, Nelleke van Zeeland, and Marc Holtman at the Amsterdam City Archives, and Pieter Woltjer, Ronald Carpentier, and Ellen van Noort at Picturae, for their helpful contributions, especially for providing generous access to the data of the two case-study projects and the Vele Handen platform. A special thank you to Marijn Koolen for his contribution to the data analysis.

This research has been supported by the Creative Industries Research Centre Amsterdam ( and the Centre for Digital Humanities ( at the University of Amsterdam.


Arends, M., J. Froschauer, D. Goldfarb, D. Merkl, & M. Weingartner. (2012). “Analyzing user motivation in an art folksonomy.” Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies (i-KNOW ‘12). New York: ACM. Consulted October 11, 2013.

Barabasi, A. L. (2002). Linked: The new science of networks. Cambridge, MA: Perseus.

Brabham, D. (2008). “Crowdsourcing as a model for problem solving: An introduction and cases.” Convergence: The International Journal of Research into New Media Technologies 14(1), 75–90.

Brumfield, B. (2013). “Choosing crowdsourced transcription platforms.” Collaborative Manuscript Trancription Blog. Consulted November 13, 2013.,

Causer, T., J. Tonra, & V. Wallace. (2012). “Building a volunteer community: Results and findings from Transcribe Bentham.” Digital Humanities Quarterly 6(2). Consulted October 21, 2013.

Crowston, K., & I. Fagnot. (2008). “The motivational arc of massive virtual collaboration.” Proceedings of the IFIP WG 9.5 Working Conference on Virtuality and Society: Massive Virtual Communities. Consulted November 12, 2013.

Dunn, S., & M. Hedges. (2012). Crowd-sourcing scoping study: Engaging with crowd with humanities research. King’s College London: Department of Digital Humanities. Cconsulted November 12, 2013.

Estellés-Arolas, E., & F. González-Ladrón-de-Guevara. (2012). “Towards an integrated crowdsourcing definition.” Journal of Information Science, 38(2), 189–200. Consulted October 12, 2013.

Evans, M. J. (2007). “Archives of the people, by the people, for the people.” The American Archivist 70, 387–400.

Eveleigh, A. (2014). “Crowding out the archivist? Locating crowdsourcing within the broader landscape of participatory archives.” In M. Ridge (ed.), Crowdsourcing our Cultural Heritage, Ashgate (forthcoming).

Eveleigh, A., C. Jennett, S. Lynn, & A. L. Cox. (2013). “‘I want to be a captain! I want to be a captain!’: Gamification in the Old Weather Citizen Science Project.” Proceedings of the Gamification 2013 conference. New York: ACM.

Eveleigh, A, C. Jennett, A. Blandford, P. Brohan, & A. L. Cox. (2014). “Designing for dabblers and deterring drop-outs in Citizen Science.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’2014) (forthcoming).

Flanagan, M. & P. Carini. (2012). “How games can help us access and understand archival images.” American Archivist, 75(2), 514–537. Consulted October 21, 2013.

Fleurbaay, E., & A. Eveleigh. (2012). “Crowdsourcing: Prone to error?” International Council on Archives Conference. Brisbane, August 20–24. Consulted September 15, 2013.

Holley, R. (2009). Many hands make light work: Public collaborative OCR text correction in Australian historic newspapers. National Library of Australia. Consulted November 22, 2013.

Holley, R. (2010). “Crowdsourcing: How and why should libraries do it?” D-Lib Magazine March/April, 16(3/4). Consulted November 28, 2013.

Noordegraaf, J. (2011). “Crowdsourcing television’s past: The state of knowledge in digital archives.” Tijdschrift voor Mediageschiedenis, 14(2), 108–120.

Oomen, J., & L. Aroyo. (2011). “Crowdsourcing in the cultural heritage domain: Opportunities and challenges.” Proceedings of the 5th International Conference on Communities and Technologies. New York: ACM. 138–149.

Owens, T. (2013). “Digital cultural heritage and the crowd.” Curator: The Museum Journal, 56(1), 121–130.

Ridge, M. (2012). “Frequently asked questions about crowdsourcing in cultural heritage.” Open Objects Blog. Consulted October 15, 2013.

Ridge, M. (2013). “From tagging to theorizing: Deepening engagement with cultural heritage through crowdsourcing.” Curator: The Museum Journal, 56(4), 435–450.

Romeo, F., & L. Blaser. (2011). “Bringing citizen scientists and historians together.” In J. Trant and D. Bearman (eds.), Museums and the Web 2011: Proceedings. Toronto: Archives & Museum Informatics. Consulted October 30, 2013.

Rotman, D., J, Preece, J. Hammock, K. Procita, D. Hansen, C. Parr, D. Lewis, & D. Jacobs. (2012). “Dynamic changes in motivation in collaborative citizen-science projects.” In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, 217–226. Consulted October 31, 2013.

Shirky, C. (2010). Cognitive surplus: Creativity and generosity in a connected age. New York: Penguin Press.

Van Vliet, H., R. Brussee, D. Swart, E. Hekman, & M. Rovers. (2013). Crowdsourcing. Utrecht: Hogeschool Utrecht – Centre of Expertise, Creative Industry Utrecht/Crossmedialab. Consulted November 12, 2013.

Wiggins, A., & K. Crowston. (2011). “From conservation to crowdsourcing: A typology of citizen science.” System Sciences (HICSS), 44th Hawaii International Conference. Consulted November 14, 2013.

Cite as:
. "Modeling Crowdsourcing for Cultural Heritage." MW2014: Museums and the Web 2014. Published February 17, 2014. Consulted .

Leave a Reply