Every museum wants to make a difference and the ambitious vison of The Media Majlis is to help audiences understand that there is always another side, to global, regional and local stories and to empower audiences to interrogate the media around them. 

But how will The Media Majlis know what its impact has been?  Evaluation helps us find this out.  A clear definition is that evaluation is “a structured process of assessing if a project has met its goals and reflecting on the lessons learned” (Swinburn University of Technology, 2011). Perhaps more simply its purpose is also described as being to “prove and improve” (Sanfilippo et al., 2005). The critical point in this is that its purpose is not just to provide evidence of what has been achieved, but importantly to illuminate how this achievement has been brought about.

Morris Hargreaves McIntyre (MHM) is working collaboratively with The Media Majlis to help the museum develop an evaluation framework and strategy, and to embed evaluation skills and practice within the organization.  In terms of evaluation practice, this is an exciting prospect as the museum is unique in many ways.  It is the first media museum in the Arab World and has a highly interactive exhibition space using the latest digital technologies, drawn from multiple sectors such as broadcast media for example. The museum also has a number of relatively discrete target audiences who will have different needs, motivations and behavior and therefore require different research approaches.  These include university students and faculty, media professionals, locals and tourists.  Located in Qatar, the museum is also in a country with a limited tradition of visiting museums, and little experience of participating in visitor research, particularly in museums. This context both offers innovative evaluation opportunities but also challenges.

___A logic model approach

MHM uses a six-stage logic model approach to evaluation as illustrated in the following model:

Increasingly evaluation is described as “telling the story of a project” (National Lottery Heritage Fund, 2012) and the logic model illustrates this process.  Adopting this approach enables us to tell the ‘story’ of The Media Majlis, and like all good stories it has a beginning, a middle and an end.

At the beginning, for the evaluation to be meaningful and useful, we need to understand four things:

Rationale: why does the museum exist?

Ambitions: what is the museum’s vision, aims and objectives?

Inputs: what is the museum putting in to achieve these ambitions in terms of staff, exhibitions, design, interpretation, technology, formal and informal learning opportunities, programs and events and publications, both on-site and online.

Desired results: what does the museum want to achieve?  These are expressed as either outputs (such as number of exhibitions, programs or visitors) and outcomes—the difference we want to make as a result of our activities—to individuals, to communities and to the sectors we work in.

In the middle, once the context is articulated, we need to work out how we can assess whether the museum has achieved its ambitions and delivered the desired results and equally importantly, how these were brought about. Gaining this understanding requires a program of monitoring which includes identifying the right measures for the outputs and outcomes, the most effective methods to capture this data, and how the data will be analyzed to provide meaningful insight.

The final stage of evaluation is to review what we have learned, reflect on this learning and to use this insight to prove the impact and to understand what worked and how it worked, and importantly, what didn’t work and why. This insight can then help the museum to capitalize on what is successful and to make changes to anything that might not be working.

In this article, the first of three, we explore the beginning: how The Media Majlis has articulated its rationale, ambitions, inputs and desired results to ensure that the subsequent monitoring program collects the right data, in the most effective way, to evaluate the first year of the museum.

___Developing a vision-led, outcome-focused evaluation framework

MHM’s approach to the development of evaluation frameworks is both rigorous and creative.  The process culminates in a two-page evaluation framework which comprises a one-page strategy tree and a one-page methodology matrix.  These are deceptively simple documents, managing to convey complex program and methodology information in a highly-visual and therefore easy-to-comprehend way.

A year prior to the opening of the museum we worked with The Media Majlis team to review the museum’s context, plans and strategies. MHM initially undertook desk research to review the museum’s existing strategies and plans. The results of this research were then explored with the museum team in an all-staff workshop to facilitate the development of a Strategy Tree. The Strategy Tree is a one-page visual which articulates the “why, what, who and how” of an organization or project (Tjan, 2011).

The strategy tree works as a logical cascade, based on three principles:

1. The hierarchy:
   The vision informs the mission. The mission is translated into aims and objectives and the objectives are served by strategies: the broad ways in which the objectives will be achieved.  These strategies deliver outputs and outcomes, which are evidenced through the use of performance indicators. 
   
   
2. Deceptively simple, but revealing:
   If the hierarchy seems simple it’s because it is. While simple in its presentation however, the clarity of the tree stems from the thorough review of plans and interrogation in the staff workshops. The ‘tree’ analogy comes from the ‘branches’ that clearly connect what the museum wants to achieve (vision, mission, aims, objectives), how it plans to do this (strategies) and what the results will be (outputs and outcomes).  This ensures that the route through which every aim and objective will be achieved is easy to follow. 
   
   
3. Readable top-down or bottom-up:
   If you start at the top of the tree, every time you read ‘down’ a level, you can just insert the word ‘by…’ in front of the next level: We achieve our vision by committing to a mission; the mission by setting objectives; the objectives bypursuing strategies.  Conversely, you can read up the tree by inserting the phrase ‘in order to…’ to explain how every strategy serves an objective that helps to achieve the mission and vision.

At the start of this process museum already had an extremely inspiring vision with clear aims and objectives.  A range of key performance indicators and measures had also been identified.  Developing the strategy tree was therefore a process of mapping these onto the tree format, clarifying which strategies were serving which aims, and identifying where additional outputs and outcomes were needed beyond the existing key performance indicators (KPIs).

For The Media Majlis the result is a strategy tree with seven broad thematic aims encompassing artistic content, learning approach, growth of visits, audience engagement, building knowledge, achieving recognition, and delivering organizational excellence.  This is designed to be a living strategy that is reviewed and adapted in the light of ongoing feedback. 

___A collaborative approach with significant benefits

Developing the strategy tree in this way has both strategic and practical benefits:

Wide ownership: developing the strategy in this way meant that it was shaped and honed by the team responsible for delivering it which results in a strategy which is both appropriate and achievable.

Rigor: the approach encouraged the team to explore, question and challenge the strategy throughout its development so that the final structure and content truly reflects the ambitions, strategies and outcomes the museum wants to achieve.

Creativity: the approach facilitates discussion and creative thinking. Importantly it also tries to encourage the use of language which reflects the vision, passion and creativity inherent in the artistic purpose of the museum rather than simply ‘business-speak’.

Skills development: the facilitation and mentoring approach means that these skills are shared with the participants.

Visual: the graphic presentation of the strategy as a tree is visually compelling and therefore easily understood across the organization. 

___Measuring achievement

The Strategy Tree is supplemented by a Methodology Matrix, which takes the outputs and outcomes from the strategy tree; turns them into indicators that can be measured and then identifies the most effective methods for gathering this data.  The Methodology Matrix will be discussed in the next evaluation article.

References

National Lottery Heritage Fund (2012) Evaluation Good Practice Guidance. Retrieved from: heritagefund.org.uk/publications/evaluation-guidance

Sanfilippo, L., Cooper, M., Kjell, P., Marks, N., Murray, R. et al. (2005) Proving and improving: a quality and impact toolkit for social enterprise, New Economics Foundation. Retrieved from: proveandimprove.org

Swinburn University of Technology (2011) A Short Guide to Monitoring and Evaluation. Retrieved from: evaluationtoolbox.net.au

Tjan, A. K. (2011, June 1) Strategy on One Page, Harvard Business Review. Retrieved from:  hbr.org/2011/06/strategy-on-one-page