Home » M&E

Category: M&E

2 1024x576 Featured Image

Audit Tool: Calculate the Time that a Data Collector takes per Question using ODK

One question that many M&E officers or information managers ask when collecting data in KoboToolbox or ODK, is “How long did the enumerator spend on each question?”  To track this, you can use a tool called an “audit log” in your XLSForm questionnaire.

It might be easiest if you want to watch a video tutorial on how to use this tool.  Here’s the video showing you all about the simple audit tool:

You can download the XLSForm template I used in the video (with “audit” in the metadata) from here.free template xlsform with audit metadata

Why would you want to know how long questions take in ODK?

  • you might want to see if enumerators are filling in information too fast
    • For example, they might be filling in data from memory after having a conversation with their interviewee.  This might not be a problem for some settings, but for a more robust survey, you might need them to fill in the answers as they go through the survey.
  • you might want to see if enumerators are getting stuck on a particular question
    • For example, is your questionnaire design causing delays?
  • what do you use it for?  I’d love to know!
    • one person told me he needed to track how long people spent on each question because he was running an IQ test using ODK, so needed to track how long people spent per question!

Add an Audit Question to your XLSForm

  1. The first step to take when you want to track the time it takes for a data collector to collect an answer is to open up your XLSForm.
  2. I always put a “metadata” section at the very top of all my questionnaires.  This includes a start time, end time, today’s date, email, username of the data collector, etc.
  3. In your metadata section, under the “type” column, add “audit”.
  4. And then under the “name” column, add “audit”.
  5. If you want the example form I used for the video tutorial, get it here.

template for audit

Put your Form on ODK Collect

  1. Upload your XLSForm to your server (such as KoboToolbox).
  2. On your ODK Collect app (or Kobo Collect app), tap on “Get Blank Form”.  Find the new questionnaire you’ve added to your server and download it to your app.
  3. Now tap on “Fill Blank Form” and you’ll be ready to collect the data.

Fill in your form on ODK Collect

  1. Go ahead and collect your data!

Download your data from Kobo Toolbox (or other server)

  1. Download the normal “XLS” style of data download (the same as you normally would).
  2. Then download the “Zip attachments” for your form.

Open your audit log

  1. In your main data file, you’ll see there’s a uuid column.  Every row of data will have a UUID (a random, unique identifier) for each submission.
  2. In your Zip file that you downloaded – you’ll find folders named with that UUID.
  3. Find the equivalent UUID folder, and find the “audit” file that’s in there.  Open it up.

Add a column named “time-elapsed”in the Audit file

  1. Add a column named “time-elapsed” in your audit log.
  2. Then create a formula like this: 
    • =(d2-c2)/1000
  3. This will give you the elapsed time in seconds that was spent on each question!

 

Good luck – let me know in the comments below how you’re using the audit log, and if you have any questions!

 

YouTube Thumbnail Disability Featured Image

Collect Disability Data using the Washington Group Short Set

Do you disaggregate your data by disability status, whether you’re collecting monitoring data or programme activity data? In this article and tutorial video, I show you how to use the Washington Group Short Set of Questions on Disability and how to programme them into XLSForm so you can collect disability data using KoboToolbox or ODK.

Watch this video below as I show you how to programme these questions into your questionnaire so you can easily collect disability data:

And if you want the templates I use in the video, go get them here.

Follow these steps to create your own set of disability questions in KoboToolbox:

Short Set of Questions to Collect Disability Data

  1. Open up this website and find the Washington Group Short Set of Disability Questions.
  2. It’s a very short 6 questions that gives you the wording to collect disability data around six different health problems:
    1. difficulty seeing,
    2. hearing,
    3. walking or climbing steps,
    4. remembering or concentrating,
    5. difficulty with self care, or
    6. difficulty communicating in your usual language  (or being understood).

1. Create your XLSFORM (using a Field list)

Free template image for disability template

Open up your XLSForm where you’re creating your questionnaire.

In the “survey” tab:

  1. The “label” column is where you want to put the six questions. Copy and paste the six questions into the label column.
  2. Then in the “type” column, make them each a select_one question, and call the choice list “healthproblem” or something similar.
  3. Use the same type of question for all six questions.
  4. In the “name” column, give each of the questions a unique variable name, such as “seeing”, “hearing”, “walking”, “remembering”, “self-care”, and “communicating”.
  5. Now make a choice list under your “choices” tab.
  6. Copy the choice list name from your “select_one” question type, and go into the “choices” tab.

In the “choices” tab:

  1. Create a new list with “healthproblem” as the list_name.
  2. Then in the label column, enter
    1. “No, no difficulty”
    2. “Yes, some difficulty”
    3. “Yes, a lot of difficulty”
    4. “Cannot do it at all”
  3. In the name column, enter equivalent names, except with no spaces, no capital letters, no special characters:
    1. nodifficulty
    2. somedifficulty
    3. lotofdifficulty
    4. cannotdoatall.

Make your six disability questions into a “group”

This set of steps might get confusing – go grab the templates here if you need.

  1. Then, back in the “survey” tab, make the six questions into a group of questions so that they all show up on the same page in ODK Collect or Kobo Collect.
  2. Enter “begin group” in the type column before your set of questions.
  3. Enter “disability” in the “name” column for your group.
  4. And “end group” and “disability” after your set of six questions.
  5. TIP for groups:  I like to colour the Excel rows where I have “begin group” and “end group” so that I can see at a glance where my groups are in my XLSForm.
  6. Now go to the “appearance” column of your group.
  7. Enter “field-list” as the appearance.  That means that all the questions in this group will show up on the same page in ODK Collect or Kobo Collect when you go to collect disability data from respondents.

A couple last finishing touches:

  1. Now, add one more row at the top of your group of questions.  Enter “note” in the “type” column.
  2. Give your note a name in the “name” column, such as “healthproblem-note”
  3. Then copy the introduction statement from the Washington Short Set of Questions and paste that into the “label” column of the note.
  4. Then add a hint.  Enter “Read this statement out first.” in the hint column.
  5. Lastly, make sure that each of the six questions are “required” – so enter “yes” or “TRUE” into each question, under the “required” column.

2. Create your XLSFORM (using a table list)

There is one other way you could display this set of questions using a “table-list”.  It shows the questions a little bit differently on the screen of the mobile device or computer screen when you’re collecting the disability data.  It works better on a larger screen.  If you just have a small mobile phone that you’re using to collect the data, you can try it out and see if it looks good.

  1. So, add another “group” (using begin group and end group – see steps 14-16 above) around the six questions.
  2. Where you’ve entered “begin group”, enter “dis_table” into the “name” column.
  3. Where you’ve entered “end group”, enter “dis_table” into the “name” column.
  4. Then, in the appearance column of this new group, enter “table-list”.
  5. In KoboToolbox or ODK Collect, your set of questions will appear down the left-hand side.  Your possible answers will come across the top.  It looks like a table on your screen, and you have radio buttons that you can quickly select answers for each question.

Hope that was a helpful tutorial about how to collect disability data for your programme.  Please download the template and use it in your programme if you want! Download a free template for collecting disability data

12 Tips for Reporting to Communities - Rethinking Accountability

Reporting to your beneficiaries is important: Lessons about Beneficiary Accountability from COVID19

If you now find yourself in the midst of the coronavirus crisis, let me ask you how much you appreciate getting data and information every single day about the crisis. You probably read daily about 1) how the crisis itself is progressing and 2) what your government is doing (or not doing) to mitigate the effects of the virus and help you and your community.  Is this what “beneficiary accountability” looks like – our governments report to us daily about their progress and their impact!

How many of you visit various websites every day to check on these two indicators for your country:

  • # of new cases
  • # of daily deaths

And how many of you turn on the news each night to listen to your country’s governmental leader to hear an update on:

  • What they’re doing to make sure the health system has enough beds, face masks, or testing kits
  • What they’re doing for families and individuals who can’t work due to the crisis

If you’re like most people, I bet you find this information crucial during this lockdown.

Watch The Video

This article is in video format, hope you enjoy listening!

The data that gets presented to us each day isn’t perfect.  There are mistakes in it.  But that’s okay.  There’s a level of transparency and accountability that our governments (hopefully) have to us.  There’s a level of leadership they feel compelled to show to get us through this crisis.

Beneficiary Accountability and the Typical Flow of Data

Now, what does this have to do with humanitarian data?

In your humanitarian programme, you probably have funding from donors.  Those donors probably require you to report to them every month, every quarter, or every six months.  Because they want to see the progress you’ve made against your agreed plan.  You probably have to do a finance report and report on indicators to show the impact your programme has had so far.

This means – you have to go out and collect data, analyse data, and visualize that data.  And then you can report it to your donors.

Now, I write every week about humanitarian data.  I love data.

But there’s a big problem with the humanitarian data system.  It encourages a one-way flow of information.  We collect the data from people and communities we work in (beneficiaries).  Then we analyse the data and report it internally upwards to our managers.  Then they report it externally to donors.  And that’s it.

Typical Flow of Data in Humanitarian Programmes

Does the data ever get reported back the other way??  If it does, it usually only happens rarely.  And sometimes, we don’t fully think through beneficiary accountability from other stakeholders’ perspectives.  How often do you get a report back from your managers?  Many times, the data collectors (enumerators), who went out gathering all the data for each report, they never see the final report.  And very rarely do communities or households ever get a report about the progress or impact of a programme.

Let COVID-19 Inspire You towards Better Beneficiary Accountability

So I have a question for you:  When your organization works in a community – do you show transparency, accountability, and leadership by reporting your data back to your beneficiaries?  Do you tell them your plans, your progress, your impact?  Or do you only collect data from your beneficiaries and then report that data to your managers and donors?  (see here for another article about adapting the way you collect M&E data during the pandemic.) What’s your true level of beneficiary accountability?

I hope the level of data reporting you get every day in the coronavirus pandemic inspires you to share more data with your communities and people reached through your humanitarian programme.

While you’re stuck working from home, can I encourage you to create a simple report to share with your communities?  I’ve even made a simple example report that you can go and download and modify to use with your own programme.

Simple Community Report in PowerPoint - Free Template

I’m going to give you a few very simple tips for how to provide reports back to the people you work for, and please download the super-simple template, modify it, and start reporting to your own communities.

So, a few tips:

Tip 1. Make the reports visual.

Maybe your donor likes to read big blocks of text (and maybe not).  But I can assure you that most humans don’t read long reports.  We like infographics.  We like pictures.  Use icons and pictures to convey meaning.  One resource that I love is The Noun Project. You can get all kinds of icons (including OCHA’s humanitarian icons) from there for creating your own infographic.

Tip 2. Use the local language.

A lot of humanitarian reporting in the world is done in English.  Which means you’ve got to think through new work-flows if you’re going to start producing short reports in a local language.  The people who maybe write your English reports might not be the same people that should write your local reports.

Tip 3. Only report the indicators that matter to people.

Don’t report everything. Think carefully about the indicators that people most care about.  For example, in the coronavirus we want to know a couple indicators on how bad the crisis is (# of cases, # of deaths).  And we want to know what’s been done to mitigate disaster.  And we want to know what the plan is to get us out of the disaster.  Think about your own programme and what people most need to know.  Only report those things.

Tip 4. Reporting to people is about dignity.

We report to people that we want to show respect to.  To our managers.  To our donors.  We show them respect.  By showing respect, they then show us respect. Our managers let us keep our jobs.  Our donors give us additional funding next year.  Well, what about your beneficiaries?  If we don’t report to them, we show them disrespect.  And that’s when tensions can start to rise.  Show respect.  Share your information.  We’re all human.

Tip 5. Use simple words.

You might be an expert in your sector or subject.  But most other people aren’t.  Don’t use acronyms.    Use easy terms that everybody understands when you write a community report.

Tip 6. Distribute reports to your local stakeholders.

Identify who are key local stakeholders that would benefit by having access to more information. They might include field data collectors, local community governments/leaders, local community households, local clinics, schools, or community centers.  Then ensure that they receive regular update reports.

Tip 7. Choose a reporting frequency you can manage.

You probably can’t report your results on a daily basis.  But what about monthly?  If it’s just a simple report, can you produce it once a month?  The more frequent you can produce it, the better.  Again, just think about how much you appreciate frequent data updates about the COVID crisis…

Tip 8. Personalise the report if you can.

If you are able to report to each community about the impact in only their community, it might make people more interested!

Tip 9. Use a simple reporting software.

Use something like PowerPoint to create a simple, visual report.  If you like Microsoft Word, use that.  Don’t use something complicated.  If you make it simple, you’ll be more likely to get it done.

Simple Community Report in PowerPoint - Free Template

Tip 10. Get creative about ways to share your reporting.

When you report to a donor or manager, they often have reporting templates for you to use.  But your communities don’t have a template!  So be creative in how you reach them!  Community meetings?    What about a report on your Facebook page?  Or delivered to them via WhatsApp?  Or is there a notice board at the local clinic?  Maybe post it there?  Or deliver hand reports any time you’re doing a programme activity.

Tip 11. Even if your data is incomplete, report what you can.

No one has perfect data.  So don’t let this be an excuse of why you don’t report.  Be transparent if you have data limitations.  But still get out there and show these communities your respect by reporting to them.

Tip 12. Your data collection will improve.

I promise you this, when you show people respect by reporting to them, they will help you out with better data collection.  Your field data collectors will be more accurate and thoughtful in their reporting when they see the end-result they help produce.  Your households and communities will contribute more willingly when they know they will be recipients of the final report.

In Conclusion

If you’re already reporting regularly to your local stakeholders, good for you!!  If not, now is a great time to get started. Beneficiary accountability shouldn’t just be “humanitarian jargon”.

Please subscribe to my YouTube channel if you’re still reading.  If you’re a humanitarian or development worker, I try to create useful tutorials and guidance for data collection, analysis, visualization, and reporting.  I’d love you to join!  So please subscribe and share this video and article with anyone else who you think would benefit.

8 Ways to Adapt your M&E during the Pandemic

8 Ways to Adapt your M&E during the Pandemic

So all of a sudden you’re stuck at home because of the new coronavirus.  You’re looking at your M&E commitments and your programme commitments.  Do you put them all on hold? Postpone them until the coronavirus threat has passed and everything goes back to normal?  Or is there a way to still get it all done!?  This article reviews 8 ways you can adapt your M&E during the pandemic.

Here are a few ideas that you and your team might consider doing to make sure you can stay on track (and maybe even IMPROVE your MEAL practices) even if you might currently be in the middle of a lockdown, or if you think you might be going into a lockdown soon:

1. Phone Call Interviews instead of In-Person Interviews

Do you have any household assessments or baseline surveys or post-distribution monitoring that you had planned in the next 1 to 3 months? Is there a way that you can carry out these interviews by phone or WhatsApp calls?  This is the easiest and most direct way to carry on with your current M&E plan.  Instead of doing these interviews face-to-face, just get them on a call.  I’ve created a checklist to help you prepare for doing phone call interviews – click here to get the “Humanitarian’s Phone Call Interview Checklist”.  Here are a few things you need to think through to transition to a phone-call methodology:

    1. You need phone numbers and names of people that need to be surveyed. Do you have these?  Or is there a community leader who might be able to help you get these?
    2. You also need to expect that a LOT of people may not answer their phone. So instead of “sampling” people for a survey, you might want to just plan on calling almost everyone on that list.
    3. Just like for a face-to-face interview, you need to know what you’re going to say. So you need to have a script ready for how you introduce yourself and ask for consent to do a phone questionnaire.  It’s best to have a structured interview questionnaire that you follow for every phone call, just like you would in a face-to-face assessment.
    4. You also need to have a way to enter data as you ask the questions. This usually depends on what you’re most comfortable with – but I recommend preparing an ODK or KoboToolbox questionnaire, just like you would for an in-person survey, and filling it out as you do the interview over the phone.  I find it easiest to enter the data into KoboToolbox “Webform” instead of the mobile app, because I can type information faster into my laptop rather than thumb-type it into a mobile device.  But use what you have!
    5. If you’re not comfortable in KoboToolbox, you could also prepare an Excel sheet for directly entering answers.  Be warned that this will probably require a lot more data cleaning later on.
    6. When you’re interviewing, it’s usually faster to type down the answers in the language you’re interviewing in. If you need your final data collection to be in English, go back and do the translation after you’ve hung up the phone.
    7. If you want a record of the interview, ask if you can record the phone call. When the person says yes, then just record it so you can go back and double check an answer if you need to.
    8. Very practically – if you’re doing lots of phone calls in a day, it is easier on your arm and your neck if you use a headset instead of holding your phone to your ear all day!

5

2. Collect Videos & Photos directly from households and communities

When you’re doing any in-person MEAL activities, you’re always able to observe evidence. You can look around and SEE impact, you don’t just hear it through an interview or group discussion.  But when you’re doing M&E remotely, you can’t double-check to see what impact really looks like.  So I recommend:

    1. Connect with as many beneficiaries and team members as possible through WhatsApp or another communication app and collect photos and videos of evidence directly from them.
    2. Video – Maybe someone has a story of impact they can share with you through video. Or if you’re overseeing a Primary Health Care clinic, perhaps you can have a staff member walk you through the clinic with a video so you can do a remote assessment.
    3. Pictures – Maybe you can ask everyone to send you a picture of (for example) their “handwashing station with soap and water” (if you’re monitoring a WASH programme). Or perhaps you want evidence that the local water point is functioning.

3. Programme Final Evaluation

It’s a good practice to do a final evaluation review when you reach the end of a programme.  If you have a programme finishing in the next 1-3 months, and you want to do a final review to assess lessons learned overall, then you can also do this remotely!

    1. Make a list of all the stakeholders that would be great to talk to. Staff members, a few beneficiaries, government authorities (local and/or national), other NGOs, coordination groups, partner organisations, local community leaders.
    2. Then go in search of either their phone numbers, their email addresses, their Skype accounts, or their WhatsApp numbers and get in touch.
    3. It’s best if you can get on a video chat with as many of them as possible.  It’s much more personal and easy to communicate if you can see one another’s faces! But if you can just talk with audio – that’s okay too.
    4. Prepare a semi-structured interview, a list of questions you want to talk through about the impact, what went well, what could have gone better. And if there’s anything interesting that comes up, don’t worry about coming up with some new questions on the spot . Or skipping questions that don’t make sense in the context.
    5. You can also gather together any monitoring reports/analysis that was done on the project throughout its implementation period. Get any pictures of the interventions you can, too.
    6. Use all this information to create a final “lessons learned” evaluation document. This is a fantastic way to continually improve the way you do humanitarian programming.

4. Adapt your Focus Group Discussion Plan

If everyone is at home because your country has imposed a lockdown, it will be very difficult to do a focus group discussion because….you can’t be in groups!  So, with your team decide if it might be better to switch your monitoring activity from collecting qualitative data in group discussions to actually just having one-on-one interviews on the phone with several people to collect the same information.

    1. There are some dynamics that you will miss in one-to-one interviews, information that may only come out during group discussions. (Especially where you’re collecting sensitive or “taboo” data.) Identify what that type of information might be – and either skip those types of questions for now, or brainstorm how else you could collect the information through phone-calls.

5. Adapt your Key Informant Interviews

If you normally carry out Key Informant Interviews, it would be a great idea to think what “extra” questions you need to ask this month in the midst of the coronavirus pandemic.

    1. If you normally ask questions around your programme sector areas, think about just collecting a few extra data points about feelings, needs, fears, and challenges that are a reality in light of Covid-19. Are people facing any additional pressures due to the epidemic? Or are there any new humanitarian needs right now? Are there any upcoming needs that people are anticipating?
    2. It goes without saying that if your Key Informant Interviews are normally in person, you’ll want to carry these out by phone for the foreseeable future!

6

6. What to do about Third Party Monitoring

Some programmes and donors use Third Party Monitors to assess their programme results independently.  If you normally hire third party monitors, and you’ve got some third party monitoring planned for the next 1-3 months, you need to get on the phone with this team and make a new plan. Here are a few things you might want to think through with your third party monitors:

    1. Can the third party carry out their monitoring by phone, in the same ways I’ve outlined above?
    2. But also think through – is it worth it to get a third party monitor to assess results remotely? Is it better to postpone their monitoring?  Or is it worth it to carry on regardless?
    3. What is the budget implication? If cars won’t be used, is there any cost-savings?  Is there any additional budget they’ll need for air-time costs for their phones?
    4. Make sure there is a plan to gather as much photo and video evidence as possible (see point 2 above!)
    5. If they’re carrying out phone call interviews it would also be a good recommendation to record phone calls if possible and with consent, so you have the records if needed.

7. Manage expectations – The Coronavirus Pandemic may Impact your Programme Results.

You probably didn’t predict that a global pandemic would occur in the middle of your project cycle and throw your entire plan off.  Go easy on yourself and your team!  It is most likely that the results you’d planned for might not end up being achieved this year.  Your donors know this (because they’re probably also on lockdown).  You can’t control the pandemic, but you can control your response.  So proactively manage your own expectations, your manager’s expectations and your donor’s expectations.

    1. Get on a Skype or Zoom call with the project managers and review each indicator of your M&E plan. In light of the pandemic, what indicator targets will most likely change?
    2. Look through the baseline numbers in your M&E plan – is it possible that the results at the END of your project might be worse than even your baseline numbers? For example, if you have a livelihoods project, it is possible that income and livelihoods will be drastically reduced by a country-wide lockdown.  Or are you running an education programme?  If schools have been closed, then will a comparison to the baseline be possible?
    3. Once you’ve done a review of your M&E plan, create a very simple revised plan that can be talked through with your programme donor.

8. Talk to your Donors about what you can do remotely

When you’re on the phone with your donors, don’t only talk about revised programme indicators.

    1. Also talk about a revised timeframe. Is there any flexibility on the programme timeframe, or deadlines for interim reporting on indicators? What are their expectations?
    2. Also talk about what you CAN do remotely. Discuss with them the plan you have for carrying on everything possible that can be done remotely.
    3. And don’t forget to discuss financial implications of changes to timeframe.

7