Seeing the Forest and the Trees: How Engagement Analytics Can Help Museums Connect to Audiences at Scale
Robert Stein, American Alliance of Musuems, USA, Bruce Wyman, USD Design | MACH Consulting, USA
Despite recent advances, museums still have a problem. They know relatively little about their visitors, and their understanding of how best to influence visitor behavior significantly lags common practice in other sectors. Behavioral analysis of visitors to museums typically starts with attendance statistics and frequently involves staff-administered surveys, but is seldom broad or frequent enough to aid in the continuous improvement of museum practice. As a result, the knowledge we have of our visitors is minimal and episodic, and our tools are crude and inefficient.
In 2012, the Dallas Museum of Art (DMA) launched an effort to transparently and continuously monitor the long-term engagement of visitors with the museum. Dubbed DMA Friends, the approach emphasizes the repeat participation of visitors with the museum’s collections and programs, offering customized rewards in return for frequent engagement. This scalable and flexible digital platform gives the museum the ability to track individual activity, with nearly fifty thousand unique visitors joining in the first year.
In this paper, the authors provide a detailed analysis of the data gathered to date concerning the adoption and visitor engagement generated by the DMA Friends program. The paper also proposes new methods for the real-time collection and analysis of in-gallery engagement data, and investigates statistical trends and correlations that can be used by museums to enhance the impact of their engagement programming. And while the program has exceeded our initial hopes and expectations, the results from several qualitative surveys and in-depth interviews with members of the DMA Friends program suggest a path for several important improvements.
Keywords: gallery engagement, analytics, metrics, digital learning, badge systems, big data
Museums have a problem. Facing financial challenges, many are being asked to justify and quantify their impact to the communities they serve while knowing relatively little about their visitors. The understanding of visitor behavior in museums significantly lags common practice in the commercial sector and fails to provide adequate insight into how best to achieve the field’s mission imperatives. With a focus on overly simplistic attendance statistics, most museums invest little in the detailed understanding of the actions, experiences, and ongoing participation of visitors once they enter the building. Despite occasional excellent probe studies that provide some insight into the relative success of programs, didactics, and interactives in the galleries, these efforts are minimal and episodic, with tools that are crude and insufficient for knowing how to achieve long-term relevance.
Seeking to address these conditions, and needing a method to measure engagement with its audiences, the Dallas Museum of Art (DMA) created an engagement platform called DMA Friends. Built using a digital badging system called BadgeOS (http://www.badgeos.org), DMA Friends launched on January 21, 2013, as both a technology platform and a series of visitor experience changes at the museum. Bearing many similarities to loyalty and affiliate programs in other industries, the DMA Friends platform encourages repeat visitation and long-term relationship-building. By suggesting a menu of participation experiences for visitors and awarding credit and recognition for their involvement, the DMA has created an experience economy expressly aimed at promoting repeat visitor engagement across a broad audience. The overall design and theory behind the program was documented previously in the literature (Stein & Wyman, 2013) and at several public presentations to the field (Stein, 2013).
In the first year, nearly 50,000 individuals have joined the program, with a running average of 900 new friends per week, primarily from a local audience. Becoming a DMA Friend requires direct on-site enrollment and connection to Museum staff, making the enrollment statistics an important measure of local adoption. The Museum has awarded over 343,000 badges for participation during the launch year and has given away nearly 12,000 rewards connected to points earned in the program.
In addition to providing a means for the staff of the Museum to structure and measure its performance for generating engagement, the DMA Friends program also generates copious amounts of data about the behavior of individuals as they connect with the Museum’s collections and programs. Truly one of the few big data problems in the museum-space, the Friends program generates more than 21.8 million discrete fields of data annually.
2. Design choices
A number of critical decisions were made during the initial implementation that have shaped the public response and perception of the project. The basic premise of using a points-for-participation economy was chosen early on to give Museum staff an ability to influence behavior of visitors and enable promotion of programs and activities. Discrete activities are assigned point values representing relative effort and value. Simple visits to Museum galleries are worth 100 points, while more in-depth interactions are worth up to five times as many points. The ability to control this economy gives the Museum a subtle yet powerful tool to influence behavior and provides a hook to market and incentivize priority initiatives. During the rollout phase of the Friends program, careful design balanced the ease of earning points with the “cost” of rewards for both new Friends and already dedicated participants. In the end, the team attempted to design point values making it easy to earn enough points in a single visit that could be used for free parking or free exhibition rewards on the next visit. By providing advance value for the next visit, we incentivize future repeat visits among Friends.
During early focus group studies, audiences expressed an awareness and concern about how their information was being collected and used. There was sensitivity around any kind of hidden “tracking” or “surveillance” by the Museum of its visitors. In response, the DMA made a handful of additional important design decisions. First, rather than collecting complete demographic and personal data at enrollment, the initial collection is minimal, with subsequent opportunities to collect additional information as visitors learn to trust the program and build their relationship with the museum. Second, the program defaults to an “opt-in” option for data sharing each and every time. The DMA Friends program collects no data from Friends without a Friend expressly submitting information to the Museum. Voluntary submission of data to the project is an effective proxy of trust and engagement with the program and Museum. Ultimately, if the DMA treats its Friends well and respects the use of their data, then the Friends will continue to participate with the program. Lack of participation is an early signal of trust issues and a possible perception around a low value for continued participation. The rate of data collected is itself a useful metric to guide the programs development.
3. Staffing and the human component
Believing that the ultimate success of an engagement platform in museums rests on the positive interactions visitors have with museum staff, the DMA chose to reorient its staff in anticipation of the program. This internal realignment is one of the most critical factors in the program’s success to date.
To start, an existing force of gallery attendants were relocated from the Security department to the Visitor Services department and retrained to focus on hospitality. A warm and sincere welcome is the first interaction a visitor should have with the Museum. The newly minted Visitor Services Attendants embraced the change warmly and grew to see it as their primary role to ensure that the entire city of Dallas would feel a part of the Museum when they arrived. Additionally, small teams of Visitor Services Attendants were redeployed to act as ambassadors for the DMA Friends program. Their goals were to inform new visitors—asking them to join if interested—and welcome return visitors with a smile. Team members began to challenge themselves to hone their “pitch” for DMA Friends and used simple graphs and charts to track their progress each day and week. The positive competition proved infectious and resulted in significant gains seen in new DMA Friends recruitment that continues to this day.
Perhaps the most daunting challenge for launching the new program lie with the DMA’s Education team. Having created and hosted more than six thousand educational programs the year before, the addition of a new slew of “activities” was an overwhelming proposition. The team conducted an inventory of existing educational activities, aligning them to a number of well-known audience-segmentation schemes from the education literature. The result provided a framework for prioritizing the most effective programs while limiting duplicative ones. Additionally, the team realized many current educational programs could be recast and enlivened in DMA Friends as new activities or badges. In effect, the DMA Friends platform became similar to the activities the team was already accustomed to hosting. The program inventory further revealed an unrealized focus on time-specific activities rather than “anytime” activities, which led to additional changes in programming.
The design, launch, and execution of DMA Friends occupied each and every part of the Museum in some way and continues to require ongoing planning and attention for a cross-departmental team. The DMA Friends team is led by the Museum’s chair of Education and Learning Initiatives and meets weekly to review key goals, asses ongoing performance to the project’s key metrics, and initiate new activities or outreach efforts. Staff from the Museum’s education, marketing, PR, curatorial, IT, and creative services departments all work together to ensure that DMA Friends is cohesive and integrated with the overall initiative and directions of the Museum.
4. Data infrastructure
The technical underpinnings of the DMA Friends platform start with BadgeOS, which is a sophisticated plugin for the WordPress blogging platform. In adapting WordPress, all content for the platform, including the system’s Activities, Badges, Rewards, and other basic Web content, was stored as various Post content types, allowing WordPress to easily render content into the system’s current and future templates. While efficient in the early stages, as the Friends program gained popularity, it quickly became apparent that this approach would not scale.
The issue is readily revealed—each Friend in the system might participate in several dozen activities during a single visit, and the number of Activities in the Posts table starts growing exponentially, especially with enthusiastic visitors. Within a few weeks of launch, over 99 percent of the content in the Posts table were Activity entries, which slowed all other operations requiring the other much smaller subset of equally important data stored in the same table.
To resolve the problem, the team chose to segment the Activity Posts from the default WordPress implementation and instead treat them as a standard log file format. By doing so, the problem can be treated with already-known methods for logfile analysis and can be denormalized with data from other tables. Additionally, this approach makes the problem a good match for several off-the-shelf and open-source data analysis tools adopted by the team. In the future, this logfile strategy can be further enhanced by denormalizing additional fields from the system at INSERT time, thereby saving query-time during report generation.
5. Useful reporting tools and frameworks
An additional goal of the project is to generate actionable data for Museum staff to use in decision making. A few reporting tools stood out as being useful and easily replicated for many different museum applications.
Chart.io is a simple, Web-based business intelligence tool that focuses on providing a drag-and-drop interface to design charts and graphs from a variety of different data sources. A nominal monthly licensing fee per user is required, offset by a non-profit discount.
Chart.io has been used by the DMA as the primary reporting tool for front-of-house staff to track their performance engaging with DMA Friends, and by the Museum’s senior leadership in generating and reviewing weekly and monthly reports. The support for custom SQL queries for each report provides a great degree of flexibility and a real-time interface to the projects data.
Tableau is a visualization and business intelligence desktop application with advanced capabilities to process and represent large datasets both offline and online. Also a proprietary system, Tableau’s per user licensing fees are significantly more expensive than those from Chart.io, but arguably the tool provides out-of-the box reporting abilities that Chart.io does not.
For this project, the DMA staff use Tableau to connect to a variety of disparate data sources including attendance, Web, and participation data to explore patterns and relationships that might exist in the data. By using Tableau’s public visualization repository (http://public.tableausoftware.com), the DMA can easily embed and share data about the DMA Friends program (see http://www.dma.org/friends/by-the-numbers). This is a feature that Chart.io—while being inherently Web-based—does not provide.
For a more integrated, heavy-duty approach to data analysis, a number of open-source Python projects and modules have proven helpful. By using the following collection of tools, developers from the DMA can conduct more sophisticated and customized analysis of the data in Friends. As the datasets in Friends continue to grow, the datasets will likely become the primary analysis and reporting tools used for the project due to performance issues of dealing with data of this size.
IPython Notebook (http://ipython.org/notebook.html) is an in-browser shell for IPython that allows in-place scripting and inline visualization of data
pandas (http://pandas.pydata.org) is a Python module to support data analysis, statistics, and data modeling
NumPy (http://www.numpy.org) is the standard Python module for scientific computing and is particularly well suited to mathematical array operations and statistics
matplotlib (http://www.matplotlib.org) is a two-dimensional plotting library for Python that can produce publication-quality images and can display inside of an IPython Notebook via a Web browser
6. Lessons learned
The software and data platform behind Friends has collected over one million rows of data about the participation of Friends with the Museum during the launch year. Much of the staff’s post-launch activity with Friends has centered around reorienting museum practice to consider and respond to the data being collected and what is being revealed about staff performance and how visitors use the Museum.
Even basic data analysis guided the early stages of the program after the initial launch. While the launch included public fanfare and a substantial new physical installation of kiosks in the museum, enrollments were slowly leveling off. Internal anecdotal conversations highlighted the problem, while it was also plainly evident in the data chart of weekly sign ups. It was quickly assessed that the Museum lacked a clear and concise explanation of the program. As as result, the director of Visitor Services established the previously mentioned ambassadors. These staff members greeted visitors at the door and explained the DMA Friends program. In just a few short weeks, the team had optimized their approach and were able to immediately see the results of their efforts. This improved performance has continued steadily since that point and has driven the enormous success in the growth of DMA Friends.
7. Activity behavior by DMA Friends
In designing the activities for participation, staff were keen to include those that cause Friends to engage directly with works of art in the collection. Very early in the program, it was clear that simply providing activities that encourage visitors to visit galleries in the museum, and grouping those activities together under badges with fun names like Globe-Trekker, All-American, and Classics-Lover, intrigued Friends enough that they seek out those activities in order to complete the challenge of the badge. The degree to which such a simple technique drove visitors into the permanent collection galleries was surprising to the team and has continued to be robust throughout the program.
In addition to activities that encouraged gallery visitation, the team designed activities that allow Friends to interact with works of art. For example, visitors can select their favorite works of art by texting the accession number of those artworks to the Museum. Unlike the gallery visit activities, these “Love a Work of Art” activities did not see the same immediate engagement. In response, staff members began talking to visitors about the interaction and explained that they could choose works that they love from their phone. As staff have worked more of this description into their interactions with visitors, we’ve seen a significant increase in these behaviors from visitors (Figure 7).
Response was so substantial—both pace and number of works—that the team was initially concerned that this was either erroneous or fake activity. Looking at the analytics, however, we can see patterns in the data that correspond to real visitor use and have validated the majority of check-ins with actual works of art in the collection. From earlier literature, the steve.tagger (Trant, 2009) project described a crowd-sourcing phenomenon that works well for discriminating valid artworks from fakes. If more than one user submits an identical text string to the system, then the probability of that string being a valid behavior rises dramatically. That pattern is present in DMA Friends as well, and of 7,766 artworks submitted, 3,025 have been submitted by more than one user (39 percent).
8. Using Friends to motivate action
One of the key metrics the team set out to accomplish was to build a system that could be used to motivate action on the part of its audience. The key factor here is two-fold: to ensure that visitors find DMA Friends attractive enough to enroll, and to ensure that staff find the platform useful for promoting museum activities and priorities.
In many circumstances during the first year, the Museum has seen evidence of this call-and-response behavior in the system. Case in point, one of the possible rewards of the program has been free special exhibition tickets. In December 2013, the Museum featured two special exhibitions during the holiday season. Overall attendance to the Museum was robust and growing year over year, but the relative attendance to those special exhibitions was lagging. The Friends team made the decision to drop the number of points required for a Special Exhibition Ticket from 3,000 to 1,000, making it significantly easier to earn this reward (Figure 8). Reward redemption for special exhibition tickets grew by a factor of ten, and more than half of all special exhibition rewards were redeemed during that promotion (744 tickets redeemed). To be fair, the overall numbers are still low with respect to total exhibition attendance, but the degree to which the promotion shifted baseline behavior was significant.
9. Demographic extrapolation
Like most museums, the DMA lacks a robust method for determining an accurate demographic segmentation of its audience. The Museum’s current technique uses intercept surveys of museum visitors taken periodically to extrapolate demographic properties to a much larger audience. Because of the labor-intensive nature of this task, the DMA and many other museums conduct this kind of sampling infrequently.
With that in mind, and with DMA Friends capturing so much visitor data already, DMA staff set about inferring this information without additional visitor effort—DMA Friends lets the team learn simply by watching. One assumption in the Museum’s current strategic plan suggests that if the DMA is doing a good job of representing the local Dallas community, then its audience should mirror the demographics of the Dallas-Fort Worth Metropolitan Statistical Area (MSA). By collecting zip code data from DMA Friends, the Museum can make a some good statistical assertions. Since the sample size of DMA Friends is much larger than the usual Museum demographic survey data, this method ought to converge on census demographics linked to zip code.
To explore this hypothesis more completely, DMA software developers built a real-time mapping tool that can represent the number of friends from each zip code. The map can be color-coded in several different ways, including by the number of Friends normalized by the population in that zip code. Obviously, neighborhoods with more dense population should have correspondingly more Friends. Likewise, the map also plots a bar chart of Friends population alongside zip code population and sorts that chart by the distance from each zip code center to the DMA. We assume that travel time to the Museum will be a determining factor in participation in Museum programs.
By using similar methods, we can compare the Friends population to the 2010 Census demographics survey to determine an estimate of the demographic makeup of the DMA Friends program. By multiplying these demographic percentages by the Friends in each zip code and summing across all zip codes represented in the program, we can estimate the total demographic segmentation of Friends.
An idealized DMA Friends audience should mirror the population density of the MSA. This means that a zip code with a balanced Friends contingent would have the same percentage of Friends from that zip code as the ratio of that zip code population to the overall population of DFW MSA. The following map shows areas with a more significant population of Friends in green along with a less significant population of Friends in red (Figure 10). Zip Codes with a balanced set of DMA friends appear in gray.
10. Repeat visits
Another key metric that the DMA seeks to promote is the repeat participation of DMA Friends with a variety of the Museum’s programs and activities. For the purposes of this project, we consider a repeat visit to occur when a Friend checks in to any activity, claims any reward, or visits any kiosk on each discrete day. If a Friend participates in twenty-five activities all in the same day, this counts as one visit. Likewise, if a Friend participates in twenty-five activities one day, and returns to claim one reward on a different day, that counts as two visits.
In all, the project finds that 10.2 percent of DMA Friends have repeated their visit during the first year of the program. Many of those visitors have repeated their visit more than a dozen times, with a few DMA Friends returning in excess of one hundred times in just twelve months. The DMA staff’s top priority is to increase the number of Friends who come back to the Museum tracked by the program. We are currently exploring a variety of techniques and conditions that will correspond to increased repeat visitation and are hopeful that we may discover a few that have a dramatic impact on our performance.
11. Friends Survey results
In the Fall of 2013, the DMA conducted an online survey of the DMA Friends population (n=1,449) to better understand more about the audience and their current experience and understanding of the DMA Friends program. The survey showed that the age distribution of Friends (Figure 12) is well distributed, with slightly higher percentages of 20- to 40-year-olds participating in the program. It was encouraging to find that a significant portion of the population over 50 was participating with the system and does not seem to be intimidated by the technological aspects of the program.
The DMA’s overall visitor profile skews towards more female participation than male, and the DMA Friends population skews even more female (72 percent of Friends). Most Friends reported visiting the Museum with other adults (58 percent), followed by groups that visited with adults and children together (26 percent) (Figure 13). Only 15 percent of DMA Friends report visiting the Museum alone.
The team was surprised to learn in the survey that 69 percent of DMA Friends surveyed reported having visited the Museum more than once in the past six months (Figure 14). This would seem to contradict the statistics being recording regarding repeat visits via DMA Friends, which show that only 10.2 percent of Friends have repeated their visit.
There could be a few different explanations of this discrepancy. Friends who responded to this survey may be more likely to have been among the more active participants in the program. It’s also possible that survey respondents are reporting repeat visits that occurred before they joined the Friends program. However, because checking in to activities at the Museum is completely optional, it is most likely that Friends often visit the Museum without checking in to any activity. The DMA team is brainstorming approaches that would encourage Friends to register their presence at the Museum in the easiest way possible so that they can better record the repeat visitation that is already happening.
In all, it seems that Friends are very satisfied with their experience of the Museum and of the Friends program, with over 90 percent indicating that they would recommend the DMA Friends program to their friends (Figure 15) and more than 95 percent indicating they are satisfied or very satisfied with their experiences at the DMA (Figure 16).
12. Areas for improvement
As stated above, the DMA staff sees the Friends program as a tool that facilitates an iterative process of learning about visitors and staff performance with respect to delivering engaging experiences with art and long-term relationships with the public. In that light, many areas of the program still fall short and need to be improved.
The DMA is actively exploring techniques that can drive more repeat visitation to the Museum through the Friends program. Attempts are being made to capture actual visits that are currently underreported and also to experiment with new approaches and incentives that will motivate the Friends audience to return to the Museum.
For audiences that are already passionately participating with the DMA Friends program, the staff is attempting to deepen and expand the richness and complexity of the visitor experiences with Friends to continue to provide a meaningful experience. This means that the staff is consistently refreshing activities and rewards present in DMA Friends and exploring the ability of creating tracks of experience in addition to levels of achievement that are currently not present.
While the DMA Friends program does include a Web portal, its features and utility are limited. In the initial phase of the project, the decision was made to limit interaction with Friends to the physical museum experience only. This was to ensure that Friends participation recorded by the system was reflective of actual museum visitors in Dallas. Today, the DMA staff feel that by expanding the online Web portal for Friends, we can effectively offer a visit-planning feature and progress check for DMA Friends that may prove useful. Efforts to expand those features are currently underway.
13. Expanding to a national platform for engagement
In November 2013, the DMA was awarded a National Leadership Grant from the Institute for Museum and Library Services to experiment with a pilot of a DMA Friends-style engagement platform in three major metropolitan areas around the country. Learning from experiences at the DMA, the Denver Art Museum (DAM), Los Angeles County Museum of Art (LACMA), and Minneapolis Institute of Arts (MIA) have partnered together to understand how the Friends platform can be best adapted to work successfully in a variety of other museums with different business models and varying kinds of engagement needs and membership programs. Soon after the grant was approved, the DMA also partnered with the Grace Museum in Abilene, Texas, to validate how a smaller museum can adapt to this approach.
Together, these five museums represent a contrasting set of variables requiring substantial flexibility and imagination in advancing the platform. The grant will allow for customizations for each museum, while collecting data about local engagement in a centralized repository. The desire of the project is to enable analysis of data from all the participating museums that will hopefully reveal useful patterns. Our ultimate intent is to discover which methods are the most successful at driving repeat engagement in the arts.
The project is currently in its planning and development stages, with pilot projects scheduled to launch in the fall of 2014. All software and findings from the project will be released under open-source licenses to allow other museums to benefit from the work of the team.
Stein, R. & B. Wyman. (2013). “Nurturing engagement: How technology and business model alignment can transform visitor participation in the museum.” In N. Proctor and R. Cherry (eds.), Museums and the Web 2013. Silver Spring, MD. January 31. Consulted January 24, 2014. http://mw2013.museumsandtheweb.com/paper/nurturing-engagement/
Stein, R. (2013). DMA Friends: Promoting participation and engagement with art. April 23. Consulted January 24, 2014. http://www.slideshare.net/rstein/nurturing-engagement-final
Trant, J. (2009). “Tagging, folksonomy and art museums: Early experiments and ongoing research.” Journal of Digital Information, North America, 10. January. Consulted January 29, 2014. http://journals.tdl.org/jodi/index.php/jodi/article/view/270/277
Stein, Robert and Bruce Wyman. "Seeing the Forest and the Trees: How Engagement Analytics Can Help Museums Connect to Audiences at Scale." MW2014: Museums and the Web 2014. Published February 1, 2014. Consulted .