Reflecting an ambition to work at the highest international standards, The Media Majlis developed an evaluation strategy prior to opening, so that the museum could review both its delivery and impact.  Morris Hargreaves McIntyre (MHM) has been working collaboratively with the museum to guide the development and implementation of the evaluation framework and its implementation. 

The framework follows a logic model approach, illustrated below.  Two previous articles on Majlis360 describe the first four stages of the logic model. The first (How will The Media Majlis know what difference it has made to audiences?) articulated how The Media Majlis developed the Strategy Tree for the evaluation which articulated the rationale for and ambitions of the museum, identified the strategies and activities through which these would be achieved, and determined the outputs and outcomes which would measure if these have been successfully achieved. The second article (Developing an evaluation methodology for The Media Majlis) describes how the evaluation audiences were determined and the potential methods that will capture data from these audiences were chosen, which were then presented in a Methodology Matrix which is used to guide the data collection.

___Evaluation logic model

This third article explores the final stages of the logic model approach: monitoring and evaluation and review (the green and blue circles in the diagram above). The monitoring phase includes how the data is being collected and analyzed, and the evaluation and review phase is where the resulting insight is used to assess progress against the ambitions of the museum and to confirm the strategies and activities are working or where they might need amending.

___Half of potential primary research methods piloted

In developing the evaluation methodology, a total of twenty-two potential methods were identified for nine evaluation audiences. In practice during the museum’s first exhibition (Arab Identities, images in film, 2019), data was collected from four of the nine evaluation audiences, using seven of the thirteen secondary research methods and five of the nine primary research methods as illustrated in the table below. Research was not undertaken with Northwestern faculty and students as the innovative nature of the exhibition and limited experience in the use of exhibitions as a teaching resource meant that ongoing familiarization to embed use of the resources in the academic programs was being undertaken by The Media Majlis. It was also felt that some faculty and student responses would likely be captured by tools and research methods which were in place for other audiences.

Having a small core team of 6 staff and no dedicated evaluation position internally meant capacity within The Media Majlis to manage and undertake the evaluation was inevitably limited. Research with on-site visitors was therefore prioritized as this was most needed to inform future exhibition design and would potentially capture data from the widest range of target audiences. Due to this limited capacity only secondary data was collected from program/event attendees, group bookers, potential visitors’, prospective students, and sector/peers.

The primary research methods consequently piloted were gallery observations, digital exhibit analytics, a gallery exit survey during the run of the first exhibition, and additionally, vox-pops during the second exhibition. The efficacy of each of these is discussed below.

___Rich observations data informing exhibition design

Observation of in-gallery behavior was felt to be critical as the gallery infrastructure includes highly interactive elements using the latest digital technology, alongside potential for more traditional object displays. Observations were also prioritized as there is a limited tradition of visiting museums and galleries in Qatar, or of participation in visitor research, so observations were felt to be a relatively nonintrusive method. The potential to use CCTV, so that observation did not necessarily have to be undertaken in real time also made this a cost-effective method. Ethically, the use of CCTV for observations was not felt to be an issue primarily due to its omnipresence in the Gulf region and its requirement for use in all public or semi-public spaces in Qatar (Erskine-Loftus, 2016). The museum notes on the outside of its main exhibition space, on its website and in the pick-up brochure which accompanies each exhibition that CCTV is actively used.

A detailed map of the exhibition was devised which divided the exhibition into eight zones, and identified and coded each zone according to the categories in the table below.

The map also required the observer to record nine items of information including demographic make-up of the group; dwell-time; route taken by visitors; first object engaged with; group ‘stickiness’ (did they stay together or split up); inter-group discussion; visitor behavior modes such as whether visitors browsed randomly or followed sequentially and depth of engagement with different exhibits. 

Thirty observations were undertaken and while this is a relatively small sample, the observations yielded rich data which has informed the design of subsequent exhibitions. The data revealed high engagement across all exhibit types, with an average interaction of 81% across the eight zones, significantly higher than interaction from a comparable study of another Qatar museum. Feedback suggests this was due to the relatively manageable size of the exhibition; visual appeal with a significant amount of moving image; the extent and range of interactive content, and a balance of exhibit types across the exhibition from moving image and interactives to more traditional display cases and text panels.

Visitors appeared to be confident behaving in different ways in the exhibition with balanced proportions appearing to engage in four types of exhibition behavior:  browsing; following the overall exhibition layout; searching (engaging with particular zones), and choosing (selecting perhaps just one exhibit to engage with).

The primary weakness identified from undertaking the observations solely by CCTV is the inability to ask any questions of visitors which would help interpret the findings in the context of the visitor profile, and their response to and outcomes from the visit, to understand if different types of visitors and those behaving in different ways, such as browsers or followers, have equally positive outcomes. Another weakness emerged in that some recorded CCTV footage was found to be technically unstable so live observation via CCTV was preferable.

___Limited use of exit survey

A survey was intended to be the key tool to gather a wider range of quantified data to evidence if The Media Majlis is achieving its broad aims across all seven desired areas including artistic, learning approach, growth in visitation, audience engagement, building knowledge, achieving recognition, and organizational excellence.  Comprising thirty-six questions, it was intended to capture data on visitor profiles; motivations; prior knowledge of the subject matter; experience of and response to the exhibition; learning outcomes, and intention to return and to recommend. The survey, available in both Arabic and English, was designed to be undertaken with visitors exiting the exhibition, by an interviewer on an iPad.

Only eleven interviews were undertaken during the run of the first exhibition in 2019, due to three factors which constrained the sample achieved. Firstly, the flow of visitors to the exhibition, spread across 6 days a week, meant that it was not possible to undertake a significant number of interviews in any single research shift. (Staffing size and the number of students employed by the museum meant that only certain times on certain days could be staffed for this work.) Secondly, the number of questions meant the interview was perceived as relatively long.  While the length is not dissimilar to surveys undertaken in major museums in the USA and the UK, feedback suggests it is too long for audiences in Qatar where there is significantly less experience in participating in visitor research. Thirdly, while training was provided to the university student workers undertaking the survey, interviewing is a skill at which interviewers become more adept with experience, but the limited visitor flow meant there was insufficient opportunity for this experience to be significantly gained. 

While the in-exhibition digital analytics provide quantified data on attitudes and learning outcomes through the responses to questions in the exhibition, not achieving a sufficient sample of exit interviews means that there is limited quantifiable data on the profile of visitors; motivations for visiting; prior knowledge of the subject matter; experience of and response to the exhibition, and intention to return and to recommend.

___Embedded interactive evaluation is highly effective

It was envisaged that the innovative technology built for the museum would yield rich and quantifiable data for evaluation and a sizeable sample as the systems record not only responses to posed questions, including multiple choice and sliding scale responses, but also use data such as languages accessed, and length of time spent with particular digital content. 

Visitor responses illustrated the effectiveness of evaluation being embedded into exhibition design. The six question stations asked visitors to respond to multiple choice questions relating to the content of the exhibition ultimately designed to assess their knowledge of and attitudes towards the subject matter. Visitors were clearly enthusiastic to respond in this way: the question stations had the highest attractor power (% of visitors engaging with the exhibit type) of all the interactive exhibits and the analytics yielded a sample of 1,176 responses across the questions.

Rich data was gathered through these including for example 29% of respondents saying they would watch film differently as a result of the exhibition. Interestingly a more low-tech, dot-voting question located towards the end of the second exhibition, also clearly appealed with eight-five responses.⁽¹⁾

“Main thing I learned (is) that museum exhibits can be really engaging using digital and digital platforms. Like I didn’t know you could use screens like that and get lost in an exhibit like that.”
Visitor to From Visionaries to Vloggers: media revolutions in the Middle East.

 

Significant analytics about use data from interactive screens was not available in practice as the software was more limited than envisaged. However, observations revealed that the interactive screens had the second highest attractor power of the interactive elements, after the question stations, engaged with by 71% of visitors.

 

___Vox pops

Vox-pop interviews were used for visitor research for the first-time during the second exhibition (From Visionaries to Vloggers: media revolutions in the Middle East, spring 2020).  Derived from the Latin ‘vox populi’, meaning the voice of the people, these short, audio-recorded interviews asked fourteen questions about visitor responses to and outcomes from the exhibition. Although a small sample of nine interviews was undertaken⁽²⁾, they represented a range of audience segments including alumni, students, academics, people from Qatar and professionals in media/journalism/communication, and yielded rich data as illustrated in the quote below.  The effectiveness of the vox-pops is likely to be partly due their duration—being much shorter than the exit interviews—and the conversational format with the interviewer able to prompt for deeper and wider responses than is possible in surveys.

“Eye opening, artistic— I guess it’s kind of cheesy to use the word revolutionary, but it’s pretty revolutionary in there. There’s a lot of stuff about like the future is female and how female artists are portraying themselves in a different light using female empowerment stuff from other countries, but then making it Middle Eastern, which is cool.” 
Visitor to From Visionaries to Vloggers: media revolutions in the Middle East

___Embedded evaluation, vox-pops and observations most effective in pilot

Key strengths of the research methods piloted during the first two exhibitions include the power of built-in technology which enables questions for visitors to be embedded in the exhibition content.  Observations through CCTV are providing comprehensive data on dwell-time and how visitors behave in the exhibitions including wayfinding and use of exhibit data and vox-pops are yielding rich data on audience experiences and outcomes, although with limited sample sizes to date.

Due to a combination of visitor flow, staff capacity and the impact of the COVID-19 pandemic there are some significant gaps in data. Having undertaken only a small number of exit surveys means there is limited quantitative data to enable findings to be regarded as true across the wider population of exhibition visitors. Similarly, the lack of profile data for observations undertaken through CCTV where it is obviously not possible to ask question of the visitor to ascertain their profile, their response to the experience or outcomes from their visit, means behavior cannot be interpreted in the context of visitor profiles.  Despite significant numbers of people attending the programs accompanying exhibitions (631 attendees across exhibitions 1 and 2), no research has yet been undertaken with this cohort due largely to European General Data Protection Regulations (GDPR) on using booking data to contact participants. Though the museum is outside Europe these regulations would apply due to the museum visitation and employment of European Union nationals.⁽³⁾ Similarly dedicated research with faculty and students has yet to be undertaken. As the museum’s online offer is developed there will also be an opportunity to undertake research with users of this content.

___A collaborative approach to interpretation of research data

For the first exhibition the research data was analyzed and then shared in an all-staff workshop facilitated by the evaluation consultants MHM. Over a day all nine staff explored the extent to which The Media Majlis had achieved its six key aims through answering a range of questions under the four broad categories of what was learnt, what was achieved, how this happened, and how this will inform practice going forward, as well as exploring the evaluation approach itself.

This insight workshop proved valuable in enabling staff to interrogate and add to the interpretation of the data from their own experience and perspectives, resulting in stronger insights. Sharing the analysis in this way was also useful in bringing staff, who may only be responsible for working on one particular aspect of the exhibition or audience development process, together to take a holistic overview. External facilitation of the workshop was felt to be of benefit as it provided a neutral space in which potentially sensitive findings could be explored with a level of objectivity. The workshop also ensured that the resulting insight was fully owned by museum staff.

___Evaluation actively informing strategy and design

The research data is informing ongoing strategy and exhibition design through reinforcing the effectiveness of the interactivity provided and the combination of digital and more traditional object-based exhibition components.

___Evaluation useful for advocacy and wider dissemination

The range of data being gathered, combined with highly visual, info-graphic style reporting, has provided the museum with content to use for advocacy., with key findings also being shared with existing and potential audiences through social media channels.

__________

This is the third article on the museum’s audience and exhibition evaluation process.  Prior articles in this series by Jo Hargreaves can be found on Majlis360:

How will the Media Majlis know what difference it has made to audiences?

Developing and evaluation methodology for The Media Majlis.

__________

(1) The museum’s second exhibition, From Visionaries to Vloggers: media revolutions in the Middle East opened in January 2020.  Due to the closure of museums in Qatar as a component of COVID-19 lockdown the exhibition closed in early March and therefore was only on view for 7 weeks.
(2) Vox-pop interviews were affected by the close down of the museum due to COVID restrictions.
(3) See https://gdpr.eu/what-is-gdpr

References

Erskine-Loftus, P. (2016) ‘Technology and the ‘point of experience’: aspects of CCTV as possible museum exhibition evaluation and experience tracker in Qatar’, Multaqa Gulf Museum Educators Journal 2(autumn), 20­–27.