by Mark Kelleher, Catherine Cameron and Katerina Cerna (Agulhas)

In the first few months of 2020 one issue has dominated worldwide: the COVID-19 pandemic, and the way that this emergency has changed patterns of living and working. Commentators have pointed to a total re-evaluation of work as we know it and business leaders have heralded the emergence of a “new normal” of remote working. [1]

However, one wonders how many of these changes are ‘revolutionary’, and how many are changes that many have wanted to make for a while. Particularly relevant for our work on climate change – the working methods we see today brought on by COVID-19 are synonymous with the sorts of changes that are routinely recommended as ways to reduce our carbon footprint.

To illustrate how organisations can undertake effective Monitoring and Evaluation work during COVID-19 (and in future) we reflect on the approach and methodologies Agulhas used for a recent evaluation of the Initiative for Climate Action Transparency (ICAT) supported by The Children’s Investment Fund Foundation (CIFF) and the practical lessons on remote working that we learned along the way.Although COVID-19 was not a pertinent issue for this particular evaluation, at the design phase we worked with the CIFF and ICAT to make our evaluation as low-impact and collaborative as possible. This meant that the way our international team of researchers – split across both hemispheres – conducted this evaluation took advantage of virtual working, involved a minimum of travel, and utilised the knowledge and cooperation of existing stakeholder partnerships. The methodology for this evaluation was designed to provide both robustness and flexibility in the cross-triangulation of data from document reviews, interviews, a workshop and two on-line country stakeholder surveys.

Let’s have a look at the five steps we followed:

Step One:

Starting from the source material. Desk reviews are an important part of any evaluation so make sure that you have all the material you need to begin.

Ensuring early, open and transparent access to project documents is important.

Step Two:

Phone calls and videoconferencing are important tools to use when conducting remote evaluations. We held monthly check-ins with the ICAT Secretariat and CIFF; and conducted remote semi-structured interviews.

Where possible use an online meeting tool that the key stakeholders and their network are already familiar with.

Step Three:

Other methodologies, for example using online surveys, support in-depth data collection from larger sample sizes. Given the high number of countries that ICAT is supporting, (over 30) it was clear that we could not seek to interview all national stakeholders in person. In order to include voices from all participating countries in the evaluation equally, we developed an online survey for key national stakeholders to complete. It is extremely important to share your survey design with key stakeholders, in order to refine it thoroughly before it is sent out.

Using the communication networks of the organisation you are evaluating greatly improves survey response rates.

Step Four:

Findings need to be triangulated and sense-checked. A survey generates a lot of data, sense-checking was done through an interactive workshop with the ICAT network. The workshop was held (pre-C19) in London although some stakeholders joined remotely from around the world. Since the lockdown we have held similar sense checking workshops for other assignments using only online participation.

Running group sessions in an online environment allows participants to join that would not otherwise be able to make it in person, and places all participants on a level playing field.

Step Five:

Case studies help to bring out in-depth examples of a programme’s impact. For the case studies we conducted of six ICAT partner countries, we were able to complete these remotely through a combination of online interviews, desk reviews and the fact checking of ICAT’s implementing partners.

Carrying out these case studies in-country would have been highly time and carbon consuming, which we were able to avoid.

The carbon emissions saved

By doing the country case studies remotely we saved almost 22 tonnes of carbon dioxide by not taking the flights shown on the map. [2]

Source: http://www.gcmap.com

So what are the drawbacks?

Some fieldwork and data collection methods are quite difficult to adapt to remote working

In environments where IT access is a challenge, video conferencing and skype calls may not be an option. For ICAT, we were conducting interviews with professional staff, largely capital city based, working in an office environment. However, in cases where one needs to gather broader public input this can be challenging. Particularly in cases where partners are located in developing countries with already low capacities, an in-country visit may be required to advance the process for the ICAT leadership team. However, adaptability and flexibility is key, and it is always useful to consider alternatives. In the case of the evaluation team there are a number of adaptive data collection methods that can be used for remote fieldwork.

Our colleagues at the Dala Institute in Indonesia have helpfully produced a blog on some of the challenges and solutions when carrying out fieldwork on the ground involving community members. [3]

Holding seminars and workshops online loses some human contact and discussion ‘in-the margins’

Although we are all rapidly adapting to new methods of working, anyone that has participated in an online meeting or workshop over the past month will already know that these can be a little slower, that the conversation doesn’t always flow quite as well, and there can be glitches in even the best wifi.

Allow the space for everyone to log-on, to respond during the meeting and build regular breaks into the agenda in order to keep the conversation fresh and engaging. Divide the meeting into key segments so that busy stakeholders can join for key sessions.

We may need to change what is being measured

In addition to adapting in terms of HOW we are conducting our evaluations by finding other ways to collect data (e.g. phone-based, online surveys, expanding desk reviews), we may also need to change WHAT we are measuring in cases where disruptions to programme implementation or delays in measurement are a threat to validity (for example, doing an endline just after/later than planned in an evaluation could lead to lower statistical power and a null result, which may be due to COVID19 and not to programme failure).

This is more pertinent to health evaluations rather than the case study here.

There may be ethical issues to moving to online data collection

For example bias from excluding those without phones or access to power to charge phones, or the ethics of calling those who consented to healthcare follow-up, but not to research where databases do exist, shared phones, risks of data breaches.

Colleagues at 3iE have produced a useful blog that reviews some of these issues. [4]

To conclude:

It is possible to conduct efficient, robust, productive evaluations remotely. This requires some planning ahead, and access to good communications technology, but can save on costs and carbon. And in fact, during lockdown conditions, there is potentially an opportunity for teams working on climate transparency to intensify some of their data gathering and analysis work, while other activities are being placed on hold.

Finally, the first quarter of 2020 has seen an unprecedented fall in CO2 emissions and other pollutants worldwide, with analysis showing that the CO2 reduction this year will be greater than any year previously. [5]

Post 2020, it will be difficult – if not impossible – to try to maintain that so many meetings need to be conducted face-to-face, that each interaction needs to be finalised with a handshake: after all, virtual elbow bumps are available.

Infographic of the Agulhas approach to conducting evaluations remotely.

[1] https://www.wired.co.uk/article/covid-19-work-new-normal-capita

[2] 2 people flying to 6 countries, calculated through the www.Myclimate.org.

[3] https://www.dala.institute/post/fieldwork-during-covid-19

[4] https://www.3ieimpact.org/blogs/phone-surveys-developing-countries-need-abundance-caution

[5] https://www.carbonbrief.org/analysis-coronavirus-set-to-cause-largest-ever-annual-fall-in-co2-emissions