By mid-March 2020, many MEDA offices in North America and around the world were working from home. As the weeks mounted, projects had to find new ways to monitor progress amidst COVID-19 pandemic. This provided MEDA with an opportunity to pilot a new lean data approach which used phone surveys to monitor project impact.
Although phone surveys are low cost and low tech, they are a vital tool that yields informative data about our project impact. The technology we used was presented by 60 Decibels, a social enterprise through a fellow organization, Acumen. 60 Decibels is an, “impact measurement company that helps organizations around the world better understand their customers, suppliers, and beneficiaries.” Although we faced many challenges implementing this new approach, we also learned many important lessons, and made recommendations for further improvements.
Two MEDA projects shifted from in-person surveys to phone surveys: the Improving Market Opportunities for Women (IMOW) project in Myanmar and the Ethiopians Motivating Enterprises to Rise in Trade and Agri-business (EMERTA) project in Ethiopia.
We chose to test this technology on these projects because they are mature projects and both are established multi-year Global Affairs Canada (GAC) funded projects.
When running international projects, organizations are held accountable by their funder for their progress. This means that every year, each project must conduct an annual survey to understand project impact and progress.
This year, we could not visit our clients in person, so we adapted our survey collection methods to utilize phones.
Challenges and solutions
Adapting established data collection tools
When adapting a workflow designed for face-to-face interactions to remote data collection, questions and challenges are bound to arise. We faced two large questions:
- Could all the requisite data be collected through phone?
- Will the data be statistically significant?
Fortunately, IMOW and EMERTA had already completed multiple rounds of annual studies. This means that data has already been collected on the project for a couple years and their tools were refined to meet project monitoring and reporting needs.
To address the first question, the projects scaled back the survey to only ask the questions needed to update their high-level impact. IMOWs annual survey remained the same and EMERTAs survey became about 60% shorter.
IMOW shifted to a purposive sampling approach favouring the collection of high-quality data from fewer representative respondents. With this approach, IMOW aimed for 30 respondents per cohort. A random sampling approach may have asked certain partners to interview over 120 respondents in a cohort.
While both had previously used random sampling approaches, only EMERTA retained it for this survey. In fact, they oversampled by 20% to ensure they met their target of 106 farmers and 20 micro, small, and medium enterprises (MSMEs).
Choosing the appropriate data collector
Given COVID-19, in-country project offices could not convene enumerators for training on remote data collection. In response, EMERTA assigned two in-country staff on the Monitoring and Impact Measurement (MIM) team to conduct the phone surveys. The in-country MIM manager handled data validation by contacting a small sample of respondents. In total, they surveyed 88 farmers and 20 MSMEs over two weeks.
IMOW opted to use enumerators who had at least two years’ worth of data collection experience with the project. MIM staff in-country trained enumerators via telephone on the informed consent form and survey approach via telephone. No other training was needed since IMOWs annual survey did not change. 16 MIM staff in Myanmar across 4 in-country partners took two weeks to survey 568 clients.
Connecting with respondents
Reaching farmers was the biggest challenge. The main reasons were timing, phone usability, and lack of trust.
Farmers tend to work on their fields during the day, same as our data collectors. This meant projects had to call during off-hours and weekends to complete the surveys. IMOW enumerators had to contend with language barriers for certain respondents. In these situations, a local translator was needed making the coordination of three schedules even more challenging. In addition to facing similar challenges, the EMERTA had the compounding problem of having to work around the country’s many holidays.
It also became clear that phones did not always belong to the farmer. This meant that enumerators were not always able to connect with the right individual. In IMOW, connection was hampered due to rain which begins around May and continues until October. In EMERTA, this was exacerbated due to phones running out of battery or kept off to conserve battery life.
Lastly, clients, notably those in the newest cohort, are skeptical about answering questions over the phone due to less familiarity with the project and staff. Newer clients have yet to complete all the business management training, so the MIM team also had concerns with their data. The mutual trust needed develops organically alongside a real relationship.
Lessons learned and recommendations
Below are some of the lessons learned and recommendations we think would apply to individuals or organizations exploring a lean data approach through phone survey data collection during periods of restricted travel or wanting to adopt this approach permanently.
Mind the survey length
We recommend capping the survey to about 15 – 30 minutes. This could be in addition to the time needed for the informed consent, especially for new clients. To make this possible, the data collection tool must be digital and designed to flow and complete easily.
IMOW continued their use of tablets since enumerators were familiar with them. EMERTA switched to webform versions of the survey tools on laptops. If you anticipate many text responses, a laptop may be your only viable option. You may also conduct further pilots to narrow down response options and create a menu of responses from which the enumerator can select.
Customize training and quality check procedures
Allocate more time to remote trainings on phone or video. It is also helpful to reduce the number of enumerators being trained at the same time given the distance and potential technical challenges. IMOW found the use of scripts was easier for phone interviews and were especially helpful for the enhanced consent forms. Finally, make sure to quality check data right away. EMERTA found it helpful to have one team member focus solely on data validation.
Schedule surveys outside working hours
Plan to conduct surveys during hours when the respondent is not usually at work. While the initial call to schedule the interview may be during work hours, the actual survey may have to take place evenings or weekends. IMOW found that successful interviews were most likely when the rapport between enumerator and respondent already existed, especially when collecting sensitive or financial information. As such, phone surveys may only make sense when these two parties have already met in person. The EMERTA team kept a log of the respondents they were calling, noting the time they called and when they called back, to get an overall sense of the best time to conduct calls with respondents.
Steve Jobs once said, “Innovation is the ability to see change as an opportunity – not a threat.” IMOW and EMERTA projects embraced their circumstances and learned a lot from their phone survey ventures. They are excited to iterate on this lean data approach by examining what added value and learn from what did not. Please reach out if you’d like to learn more about their journey!
