Kitty.bentvelsen at 16:06, 16 December 2021

2021-12-16T16:06:51Z

Geerte at 12:58, 30 September 2021

2021-09-30T12:58:15Z

New page

Chapter 34 'Monitoring and Evaluation' describes the role of monitoring and evaluation (M&E) in assisting strategic planning and management. Chapter 35 aims to complement the earlier chapter by describing the background leading to the emergence of a WMG tracker in 2016/17 as a response to:

* a growing capacity of WMGs in four key areas: economic development, water management, organisational management and capacity development;
* recognising the value of a dashboard to monitor the adoption of Blue Gold norms by individual polders;
* the value to the TA team of hands-on health checks conducted with specific WMGs; and
* the positive response from water management organisations to participatory monitoring and eventually to self-assessment.
The final sections of this chapter account for the development of a polder dashboard to provide an overview of the progress of individual polders towards the goals of the Blue Gold Program, and how 'health checks' of WMGs gave insights into progress, achievements and challenges, and allowed project resources to be targeted more effectively and efficiently. And finally, a discussion about arrangements for the continuation of self-assessments by WMGs after the project.

== Background ==
The original main purpose of monitoring and evaluation (M&E) in Blue Gold was to monitor and evaluate progress and achievements against key performance indicators of the program at output and outcome levels, with the aim of enhancing evidence to use in planning and management of the program. Information on, and the progress of different activities was collected by technical assistance (TA) sub-teams - then known as ‘components’ - who analysed the data and presented progress and shortcomings against the targets set in the Annual Work Plan - which in turn was based on the project logframe.

From the outset, data related to the different TA activities were collected – at output level as well as at outcome level. This data was then compiled and analysed, and reports were prepared:

* [[:File:TR06 Household Survey Report P22 30 43 2D 43 2F 31mar 14.pdf|TR06]] household surveys for P22, P30, P43/2D and P43/2F (31 Mar 2014)
* [[:File:TN07 FFS Cycle 4 Mar-Sep 2015 23dec 15.pdf|TN07]] FFS Cycle 4 benchmark and end data: Mar-Sep 2015 (23 Dec 2015)
* [[:File:TR14 Phase 1 Baseline Survey Report.pdf|TR14]] Phase I Socio-Economic Baseline Survey Report (28 Dec 2015)
* Master Files<ref group="Notes">The Master Files on these 12 polders were incorporated into the Polder Development Plans (PDPs). For the remaining 10 Blue Gold polders the information assembled for the Master Files was published only in the PDPs. </ref> with geography, present situation, economic sectors and potential value chains for [[:File:MF P2 master file 29jul 15.pdf|P2]], [[:File:MF P22 master file 15jun 14.pdf|P22]], [[:File:MF P26 master file 30sep 15.pdf|P26]], [[:File:MF P29 master file 4nov 14.pdf|P29]], P30, [[:File:MF P31-part master file 6apr 15.pdf|P31-part]], [[:File:MF P43 1A master file 4aug 15.pdf|P43/1A]], [[:File:MF P43 2A master file 26apr 15.pdf|P43/2A]], [[:File:MF P43 2B master file 7jul 15.pdf|P43/2B]], [[:File:MF P43 2D master file 20apr 15.pdf|P43/2D]], [[:File:MF P43 2E master file 24jun 15.pdf|P43/2E]] and [[:File:MF P43 2F 27sep 18.pdf|P43/2F]] (various dates from Jun 2014 to Sep 2018).

Figure 35.1 shows how data was collected, stored and distributed before the introduction of a digital management information system (MIS). The data were collected on paper and entered into spreadsheets, and were then analysed by specific individuals. Access to information by other project stakeholders was only then available through project reports, as processed data .

[[File:Figure 33.1.png|'''''Figure 35.1''' M&E data management system prior to MIS''|alt=|center|frame|border]]

== Establishing a WMG Tracker ==
In August 2016, as part of a refocusing of the scope of the monitoring, reflection and learning team, an activities plan<ref>{{Cite book|last=|first=|title=Monitoring, Reflection and Learning Plan|date=August 2016|publisher=Euroconsult Mott MacDonald|year=|isbn=|series=Working Paper 6|location=|pages=}}</ref> was prepared and discussed with the 2016 Annual Review Mission (ARM). This included a prototype WMG tracker which had been developed in September 2016. During its visit in October 2016, the ARM recommended that the strengthening of WMOs should ''“be supported with an adequate MIS system”,'' and that the TA’s prototype WMG tracker should be converted to a dashboard type MIS with target setting that is open access, with quarterly reports from the MIS provided to all stakeholders. (Recommendations 2.3 and 4.4)

A provisional format for data collection for the WMG Tracker was prepared in January 2017 and given to polder teams for a trial. WMG data was collected by TA polder-level staff, Community Development Facilitators (CDFs). Steps to overcome a difficulty with accurate data entry were agreed in March/April 2017 and the revised format was finalized in May 2017. A first report on the WMG Tracker<ref>{{Cite book|last=|first=|title=WMG Tracker Report – to June 2017|date=November 2017|publisher=|year=|isbn=|series=Working Paper 9A|location=|pages=}}</ref> for the period up to June 2017 was issued in November 2017. WMG tracker data continued to be collected by CDFs on a quarterly basis for each of the Blue Gold WMGs.
== Management Information System (MIS) ==
A vast amount of data is collected from the WMG Tracker and from field surveys and the participatory monitoring system. A rationalisation of the systems for data collection and management aimed to collect and store data from different sources at one location (a central server), which would allow better analysis and data presentation to aid decisions for project implementation. The main objectives of the Blue Gold MIS were therefore to:

* Collect and store information collected by the program in one place/server
* Standardise the information collected by the project;
* Present the data in standard reports that can be shared with program partners; and,
* Make the MIS accessible so that interested users can obtain and analyse data.

The ultimate aim of providing a consistent set of data for analysis, data presentation and reporting, was to facilitate informed, timely and evidence-based project decisions.

Figure 35.2 shows the proposed M&E data management system, in which data collected through electronic device are stored in a single location, the server where data can be analysed and presented in an MIS dashboard. In addition, data can be downloaded from the server in MS Excel for further analysis as may be required – all stakeholders have real time access to data and the dashboard without having to wait for a report to be generated.[[File:Figure 33.2.png|frame|'''''Figure 35.2''' Digital Management Information System''|alt=|center]]

== MIS Design and Development ==
In March 2017, Blue Gold contracted mPower Social Enterprises of Dhaka, to develop an MIS platform. Their specialist team included system design and database analysts, mobile and web app developers and quality assurance engineers. During the inception phase for mPower's project, the technical data requirements of Blue Gold stakeholders were mapped and the project activity chain was agreed. Proposed features of the MIS platform included customizing Open Data Kit (ODK) software so that data could be collected through android devices, and uploaded into a custom-made, cloud-based data collection system with a visual reporting dashboard at the front-end.

The MIS system was ready in October 2017, and output results from the WMG tracker were first [[:File:WMG tracker mPower presentation 25oct 17.pdf|presented]] to BWDB during a workshop on 25<sup>th</sup> October 2017. Training in the design and use of the database was then provided to staff of BWDB and DAE both in Dhaka and in field offices. The MIS system was used from June 2017 to June 2019, when the collection of data was discontinued because of the expectation (in June 2019) that there would be staffing reductions in preparation for closure of the program at end-June 2020.
{| class="wikitable"
|+Box 35.1 ''An overview of the MIS system''
|'''Blue Gold’s MIS system''' is a mobile, web-based solution with built-in forms for the collection and combination of historical data (from previous data sets), baseline data, WMG tracker data, participatory monitoring as well as outcome and impact assessment data. Data is entered through an android-based mobile application using standardised forms which – after completion – can be saved locally and reviewed and then submitted to the server.
Users are able to view and export raw data and review reports. Those with administrative user rights are able to create, assign, edit and build new functions into the system. Data required for reporting can be exported imported into the M&E database from the MIS, providing a quick and simple way to share the data without disrupting the proven MIS processes. All M&E databases and reports are available in the MIS. The MIS can also be linked to a GIS-enabled solution so that information can be depicted graphically on a cadastral map, with interrogation possible at division, District, Upazila, Union and WMG level.
|}

[[File:Figure 33.3.png|center|frame|'''''Figure 35.3''' An overview of the MIS system'']]

== MIS Results Reporting ==
Reports on the gathered data were available through a dashboard as well as in various working papers.

One series of working papers, WP9 (see below and in the File Library), presented an analysis of information of data collected through the WMG Tracker. They provided evidence of growing WMG capacity in four key areas: economic development, water management, organizational management and capacity development.
{| class="wikitable"
|22 November 2017
|[[:File:WP9A WMG Tracker Report without annexes to June 2017 23nov 17.pdf|WP 9A]]
|WMG Tracker Report-June 2017
|-
|15 April 2018
|[[:File:WP09B 11apr 18 WP9B WMG tracker to Dec 2017.pdf|WP 9B]]
|WMG Tracker Report-December 2017
|-
|20 June 2018
|[[:File:WP09C WMG Tracker Report to MARCH 2018 v5 20 06 2018 SKD.pdf|WP 9C]]
|WMG Tracker Report-March 2018
|-
|27 August 2018
|[[:File:WP09D WMG Tracker Report up to JUNE 2018 v4 27 08 2018 SKD.pdf|WP 9D]]
|WMG Tracker Report-June 2018
|-
|26 January 2019
|[[:File:WP09E WMG Tracker Report up to SEPT 2018 iss 26jan 19.pdf|WP 9E]]
|WMG Tracker Report-September 2018
|-
|20 February 2019
|[[:File:WP09F WMG Tracker Report to Dec 2018 iss 20feb 19 opt.pdf|WP 9F]]
|WMG Tracker Report-December 2018
|-
|16 May 2019
|[[:File:WP09G WMG Tracker Report up to MARCH 2019 v3 16 05 2019 SKD.pdf|WP 9G]]
|[[:File:WP09G WMG Tracker Report up to MARCH 2019 v3 16 05 2019 SKD.pdf|WMG Tracker Report-March 2019]]
|-
|30 October 2019
|[[:File:WP09H 30oct 19 WP9H WMG Tracker FINAL Report jun 19.pdf|WP 9H]]
|[[:File:WP09H 30oct 19 WP9H WMG Tracker FINAL Report jun 19.pdf|WMG Tracker Report-June 2019]]
|}
BWDB’s Project Coordinating Director (PCD) used the dashboard, analysed and cross-checked data, and monitored the activities and progress of WMGs. He used information from the dashboard in preparing articles for publication in newspapers, making presentations, and shared information on WMOs with other BWDB officials.

== WMG Tracker Closure ==
Plans to discontinue the WMG tracker were made during 2019, as part of a gradual phasing out of TA staff in expectation of project closure at end-December 2020, and with reductions in TA polder-level staffing (the staff who collected the raw data) planned at end-December 2019, and in the absence of a stakeholder willing to take over its management. The final report was issued in October 2019 based on information collected from April to June 2019.

== Polder Dashboard ==
To provide an overview of the progress of individual polders towards the goals of the Blue Gold Program, ten proxy indicators were identified from project monitoring tools to represent the four broad thematic areas towards empowering WMGs:

* overall achievement
* organisation
* water management, and
* economic development

For each of the ten indicators, targets were set for achievement by end-June 2020 but without defining the modality, thus freeing polder teams to select implementation arrangements which best suited the local environment, in other words, providing ‘a clear destination’ but with some flexibility as to how to get there. Figure 35.3 provides an example of a dashboard summarising results as of June 2019. A short note accompanied the dashboard to identify actions required to achieve the targets, and to identify the achievements for each polder and to recommend corrective actions (for example, see the note dated September 2019).

The polder dashboard was in use from December 2018 to September 2019 but discontinued when information for the WMG tracker - from which some of the proxy indicators were derived - ceased to be collected from end-September 2019.
[[File:Fig 35 3 polder dashboard 30sep 19.pdf|center|thumb|805x805px|'''''Fig 35.4''''' ''Polder Dashboard to end-June 2019'']]

== Polder "Health Checks" ==
In addition to the quantitative data obtained on polder and WMG performance, qualitative polder “health checks” have been carried out by multidisciplinary teams of TA staff (see example output in Figure 35.4). These health checks were intended as a third-eye, qualitative assessment, which in combination with quantitative project data, provides insight into progress, achievements and challenges in specific polders, and help the team to target project resources more effectively and efficiently.
{| class="wikitable"
|+Table 35.1 '''''Health Check for Polder 29 WMG'''''
!'''Topic'''
!'''Snap shot of discussion'''
!'''Outcome & Remark'''
|-
!'''Goal'''
|Members think WMG is for achieving something jointly. They gave emphasis to water management for development.
|This is an organisation based on a temple. It is a close-knit community. Whatever, the goal, they focused mostly on unity.
|-
!'''Leadership'''
|It is a close-knit temple committee. Same leadership since 2005.
|Members think they are well consulted. They are happy with leadership. Seems mostly democratic.
|-
!'''O & M'''
|They performed collective actions for improved water management.
|Collected BDT 4,150 yearly from members. When needed they employ self-labor and raise fund.
|-
!'''Networking'''
|Sound knowledge about potential support they can get from DAE, DLS and LGIs.
|It seems that leadership is well-shaped to network with GoB organisations and LGIs.
|-
!'''Sustainability'''
|They are very confident about sustaining as WMG. They have been working as a coherent group since 2005.
|They are more a temple committee who are ready to work jointly to solve any problems; WMG is just a label for them.
|-
!'''Functionality'''
|They have the potential to evolve / sustain.
|Leadership & members need to strengthen vision.
|}

== Participatory Monitoring ==
Through participatory monitoring of both WMGs (from October 2016 to October 2019) and WMAs (from May to October 2019), WMGs and WMAs had rich experience of using monitoring information to make decisions. Working papers WP8 are a series of reports using information collected through participatory monitoring at WMG level to present self-assessments by WMGs of their performance against potential targets at outcome level – and actions required to move towards full achievement of the targets.
{| class="wikitable"
|28-Feb-17
|[[:File:WP08A participatory monitoring red 28feb 17.pdf|WP 8A]]
|Participatory Monitoring October - November 2016
|-
|20-Nov-17
|[[:File:WP08B participatory monitoring v4 20nov 17.pdf|WP 8B]]
|Participatory Monitoring April-May 2017
|-
|07-Apr-18
|[[:File:WP08C participatory monitoring oct nov 2017 5apr 18.pdf|WP 8C]]
|Participatory Monitoring October -November 2017
|-
|24-Jul-18
|[[:File:WP08D Report on Participatory Monitoring Apr-May 2018 v1 24jul 18.pdf|WP 8D]]
|Report on Participatory Monitoring April/May 2018
|-
|26-Jan-19
|[[:File:WP08E Report on Participatory Monitoring Oct 2018 iss 26jan 19.pdf|WP 8E]]
|Report on Participatory Monitoring October 2018
|-
|10-Jul-19
|[[:File:WP08F report on Participatory Monitoring Apr 2019 v1 10jul 19.pdf|WP 8F]]
|Report on Participatory Monitoring April 2019
|-
|15-Dec-19
|[[:File:WP08G Report on Participatory Monitoring Oct 2019 15dec 19.pdf|WP 8G]]
|Report on Participatory Monitoring October 2019
|}
Working papers WP10 use information collected through participatory monitoring at WMA level to present self-assessments by WMAs of their performance against potential targets at outcome level.
{| class="wikitable"
|05-Aug-19
|[[:File:WP10A WMA Participatory Monitoring May 2019 5aug 19.pdf|WP 10A]]
|WMA Participatory Monitoring to May 2019
|-
|24-Dec-19
|[[:File:WP10B WMA Participatory Monitoring Oct 2019 23dec 19 opt.pdf|WP 10B]]
|WMA Participatory Monitoring to October 2019
|}

== Post-Project Monitoring ==
Whilst the closure of the WMG tracker was because of resource constraints within the TA team, WMAs - the key players for polder water management - had indicated their interest in continuing to operate and use a streamlined version of the self-monitoring system beyond the end of TA involvement. In addition, a meeting held on 20<sup>th</sup> August 2019 with representatives of EKN/BWDB/TA supported the continuation and suggested that the views of potential stakeholders and options for handover should be examined. During this review, BWDB’s Chief Water Management (CWM) and WMO representatives agreed that the monitoring should continue, but because of resource constraints, CWM requested that WMOs take the major responsibility for data collection and management. The WMOs were willing, but unable to handle large amounts of data. To minimise data processing and reporting, an ICT-based data collection and reporting system was proposed, with WMGs and WMAs collecting and entering data on their status and performance.

== Self-assessment of WMG performance ==
WMAs and WMGS are familiar with the process of participatory monitoring. For three years (from October 2016 to October 2019), WMAs and WMGs have assessed their progress and performance against potential development targets - and have used this information in meetings to decide on interventions to lead to a full achievement of those targets. During this participatory monitoring, Blue Gold worked with the WMOs to evolve the process and to observe how the output was used to encourage internal reflection by the WMO – with actions corresponding with the targets. With this experience, there is reasonable confidence that WMOs will continue the self-monitoring using the mobile App. The presence of the TA team in 2020/21 (at a reduced level) will allow some hands-on assistance to establish a quarterly monitoring routine - led and owned by the WMAs.

In January 2020, an IT solution provider, mPower Social Enterprises of Dhaka<ref group="Notes">mPower Social Enterprises had worked earlier with Blue Gold in developing the overall MIS system so were already familiar with the project and its aims and objectives.</ref>, was contracted to develop a web-based system for collating and organising data about the status and progress of activities collected and entered by WMGs and WMAs - and for it to be automatically uploaded to a web-platform. The system has been designed so that WMOs can collect data through, and view results in Bangla on, their mobile phones. The mobile application is used for data collection each quarter. Through self-monitoring, the WMAs and WMGs are able to examine their performance in three key areas of their organizations:

* Legality – Does the WMA, and its constituent WMGs, still comply to the formal requirements of its registration: are there timely elections, are accounts ready for audit, are annual general meetings (AGMs) held?
* Legitimacy – is the WMA recognised within the polder; does it represent 70% of active WMGs and well-connected to LGI, BWDB and others?
* Functionality – Does the WMA do what it is set-up for: ensure catchment management plans are current, and that water management activities are carried out?

The performance assessments are colour coded, and WMAs can also access status reports of their constituent WMGs as well as the status reports of other WMAs (for purposes of comparison).

At the time of writing (in March 2021), two rounds of WMO self-assessment have by been conducted using the IT-based mobile application, and a third is planned for Q1 2021. All WMOs were trained in the use of the App, and most WMOs - 35 out of 36 WMAs and 507 out of 511 WMGs - participated in the self-monitoring exercise. The feedback from the WMOs has been mostly positive, they find the system simple and intuitive. The two main challenges were that: (a) the App couldn’t be installed on some of the WMO mobiles because the configuration was incompatible; and (b) the signal from the mobile network was not always sufficiently strong to operate the App.

The polder teams have reviewed the monitoring results with a number of WMAs and WMGs during meetings for other purposes, during which they encourage the WMOs to reflect on the results and plan follow-up actions. From March 2020 and through the first half of 2021 (when permitted by government restrictions to minimise the spread of COVID), WMA capacity building workshops used the self-monitoring outcomes to help WMAs analyse their strengths and the areas for improvement.

Reports (in English) on the findings of the first two self-monitoring exercises are available in the File Library:
{| class="wikitable"
|28-Dec-20
|[[:File:WP11A WMO self-monitoring Q3-2020 28dec 20.pdf|WP 11A]]
|WMO Self-Monitoring Q3- 2020
|-
|25-Mar-21
|[[:File:WP11B WMO self-monitoring Q4-2020 25mar 21 opt.pdf|WP 11B]]
|WMO Self-Monitoring Q4- 2020
|-
|
|WP 11C
|WMO Self-Monitoring Q1- 2021
|}

== Notes ==
<references group="Notes" />

== References ==
<references />

== See more ==
{{ArtNav|Prev_chap=[[34 Monitoring and evaluation|Chapter 34: Monitoring and evaluation]]|Curr_sect=[[Summary of Section G: Project Management|Section G: Project Management]]|Next_chap=[[36 Environmental Due Diligence|Chapter 36: Environmental Due Diligence]]}}

{{SeeMoreSectionG}}

@@ Line 1: / Line 1: @@
 Chapter 34 'Monitoring and Evaluation' describes the role of monitoring and evaluation (M&E) in assisting strategic planning and management. Chapter 35 aims to complement the earlier chapter by describing the background leading to the emergence of a WMG tracker in 2016/17 as a response to:
-* a growing capacity of WMGs in four key areas: economic development, water management, organisational management and capacity development;
+* a growing capacity of WMGs in four key areas: economic development, water management, organisational management and capacity development
-* recognising the value of a dashboard to monitor the adoption of Blue Gold norms by individual polders;
+* recognising the value of a dashboard to monitor the adoption of Blue Gold norms by individual polders
-* the value to the TA team of hands-on health checks conducted with specific WMGs; and
+* the value to the TA team of hands-on health checks conducted with specific WMGs
 * the positive response from water management organisations to participatory monitoring and eventually to self-assessment.
 The final sections of this chapter account for the development of a polder dashboard to provide an overview of the progress of individual polders towards the goals of the Blue Gold Program, and how 'health checks' of WMGs gave insights into progress, achievements and challenges, and allowed project resources to be targeted more effectively and efficiently. And finally, a discussion about arrangements for the continuation of self-assessments by WMGs after the project.
@@ Line 29: / Line 29: @@
 * Collect and store information collected by the program in one place/server
-* Standardise the information collected by the project;
+* Standardise the information collected by the project
-* Present the data in standard reports that can be shared with program partners; and,
+* Present the data in standard reports that can be shared with program partners
 * Make the MIS accessible so that interested users can obtain and analyse data.

35 Management Information System - Revision history

Kitty.bentvelsen at 16:06, 16 December 2021

Geerte at 12:58, 30 September 2021