In 2020, OECD DC added the criterion of "coherence" to its list of evaluation criteria. OQSI introduced "coherence" in 2016 but the interpretations of the term "coherence" are different in each case and the handling of the guidelines related to coherence are also different in the cases of OECD DAC and OQSI.
The previous article in this series last 35 years has generated enough evidence to suggest that climate change is accelerating. In the context of agriculture, a leading edge domain for innovation in farm planning and project design is oriented towards the effective management of a increasingly dynamic set of circumstances. Today the required configuration and design of optimization farm plans and new projects needs to build in a practical change strategy that has two components. One is substituting the local production systems with production systems that have a higher probability of remaining feasible for at least a decade and perhaps 15 years. This can mean moving current production of selected commodities to different locations in order to preserve the necessary levels of productivity in the near future and to substitute those crops that have been moved, by more suitable crops for the expected local transtion in bioclimatic conditions. These types of considerations are important for many perennial crops that require longer payback times. In the case of mixed annual cropping it is necessary to accept the possibility of building into a project adaptation to change so that at the end of a funding period the project the crops in the production system will be different different from those at the outset.
These changes represent a challenge for policy makers and appraisals of projects whose performance is projected to change during implementation periods. For example there is a need to introduce modified procedures to calculate financial return, the assessment of the ability to sustain real wages and to conserve carrying capacity. This has significant implications for design methods, implementation decision analysis in response to change transitions as well as monitoring and evaluation.
The Gig-commodity economy
|The impact of locational-state on agricultural production|
A cash flow projection is shown as curve CF1 based on the agroecological conditions at the design stage of a project.
Within a normal trajectory of climate change affecting temperature and indirectly water availability, the impact of rising rates of evapotranspiration and water deficit can result in the cash flow outcome becoming CF2.
Where the reproductive cycles are affected, such as in the case of coffee flowers being killed above 33o
C, (Arabica spp.
) the result is zero production and a negative cash flow. These locational-state trajectories occur typically over a period of between 10 and 15 years.
- Allow for dynamic change in all aspects of a project
- Identify and quantify the most likely changes as part of project design
- Adjust project financing arangements to take change into account
- Adjust monitoring and evaluation procedures to this reality
Climate change is creating, in many areas of the world, a GIG-commodity economy where Gig signifies a one time performance. For most agricultural projects this means the general conditions at the inception of a project will not be the same as the conditions at the end of the project. Change is inevitable.
This short article describes how project design needs to be adjusted to accommodate this reality as do evaluation procedures. Therefore, in the preparation of projects there is a need to anticipate the most likely changes in the environment within a relatively short period of the project initiation date and to adjust agricultural project designs accordingly. This approach to project design has become the most single significant agricultural innovation challenge facing world's agricultural sectors, farmers and rural populations. Fortunately this is something the George Boole Foundation has been working on, to find practical solutions, for the last decade.The Log Frame Approach
Log Frames have been applied across projects to record the combined activities of a project that contribute to the output of results; it is a decription of the process with a timeline and demarcation of project phases and deliverables. It is used to manage the sequencing of work as well as conducting monitoring and evaluation by comparing planned outputs with those achieved. Some development agencies have introduced modifications in terms of what is included in LogFrame information. For example, some have introduced resource budgets for each activity.
During the last 35 years it has become evident that an increasing number of projects "fail" because of the lack of additional funds in contingencies result in an inability of teams to make the necessary adjustments in response to changes in conditions. The default "solution" is a rapid monitoring and evaluation (M&E) visit which invariably results in a necessary downgrading of performance indicators if the project is to continue within the budgetary envelope assigned. This can result in project delays. On several occasions desk officers in development agencies, under pressure from their administration to maintain adherence to financial agreements, can terminate projects because the active period of financing has been over-run or is about to be. This can result in what were good projects with good teams, failing. Of course, an external M&E team might review the outcome by making use of the original LogFrame and decide that the performance was not good. They might refer to the lack of funding to cover neccessary adjustments required due to changed conditions. Often, because M&E teams are working for the funding agency the role of LogFrames in this situation is seldom questioned.
LogFrames tend to work well in controlled enviroments such as a processing plant or agroindustrial operation where climatic conditions do not impact the actual process directly. However, in rainfed agricultural production, LogFrames have proven to be too simplistic because of the fact that the final project LogFrame is often associated with a fixed budget. So, the LogFrame becomes the single description of a fixed output to be delivered for a fixed price. Any continencies tend to be arbitrary percentages of the actual total budget of a project. Clearly any major change in environment or market conditions will cause a poor fit between the project "intent" and the current requirements.
One of the definitions of a decision, in the discipline of decision analysis, is that it is an irrevocable allocation of resources to a stated course of action and that any change in a decision will result in the need for additional resources. So a LogFrame contains the stated course of action but when a decision needs to be changed to respond to changed conditions so must the LogFrame and budget if the original objectives are to be secured.Required changes
There is a need for more of the project "identification" to be concerned with a "design effort" within which the different potential impacts of likely changes are assessed in quantitative terms. Because Sustainable Development Goals embed a range of social, economic and environmental considerations the potential impacts of a single factor can generate differential impacts on these considerations. However, it is necessary to identify those circustances where the trade-off between these considerations become too constraining on the ability to obtain desired results. In order to avoid these types of circumstance, changes in the project configuration, often linked to technology, techniques and organization, can sometimes result in improved resilience in the sense of reducing the impact of predictable changes on fundamental performance requirements. Amongst the more obvious performance requirements, within the context of the current Agenda 2030 portfolio, we can include the current areas of really poor peformance which need to be improved, including:
- ERR - Aspects of Economic Rates of Return (financial and real incomes)
- financial return - linked to SDG 1
- real income disparity- linked to SDG 8 and SDG 10
- RRE - Aspects of Rates of Returm to the Environment (carbon footprints and equilibrium)
- attenuation of climate change- linked to SDG 13
- carrying capacity- linked to SDG 12
Formal performace criteria
|Adapting to change ...|
Sustainable Development Goal actions involve complex cross-relationships and the Agenda 2030 portfolio performance is poor. There is an urgent need practical approaches to improve performance. There are two systems for policy and project performance evaluation, the OECD DAC Evaluation Criteria and the OQSI Design for Performance System. How do they compare?
Currently there are two sets of performace criteria applied to projects of which the OECD DAC Criteria are the most widely applied. In the latest update1
the criteria are shown in the left hand column of the table below. This includes the addition of "coherence" added in the criteria update made in 2019. Also included in this table is the OQSI recommended criteria. The OQSI had added "coherence" in 2018 but also they added "resilience" as a test of the degree to which a design represented a minimization of risk linked to possible changes in conditions.
|Comparison of OECD and OQSI project evaluation criteria|
Comparison of systems
Coherence & resilience
The OQSI use of the term "coherence" refers to the degree of tranparent logical links between locational state variable data, agricultural systems and the information use for decision analysis in project design.
The OQSI use of the term "resilience" refers to the degree that a project's onjectives have been tested against likely changes in locational state variables to identify project designs that will suffer the lowest negative impacts under conditions of change.
First of all there are differences in the interpretation of the criterion "coherence" on the part of OECD and OQSI.
OECD apply this in a general way to complementarity between projects within a country related to some given objective. There is also another interpretation related to the internal compliance of a projects activities with norms, legal frameworks and regulations.
The OQSI covers complementarity and legal compliance in separate specific procedures that are part the project due diligence design procedures (3DP) where such alignments are sorted out. The OQSI's interpretation of coherence is linked to the comparability, transparency and complementarity of the data, data types and metrics used in measuring project performance and comparing them. Without this level of coherence it is difficult to develop decision analysis models to simulate options in optimising project designs.
Linked to this is the OQSI's additional criterion of "resilience" which appears to be the equivalent of sustainability under the OECD scheme. However, under the OQSI approach, resilience is specifically related to the level of tolerance a project design has for impacts from different levels and types of change. It is a measurement of stability determined on the basis of risk analysis. Sustainabilty is a more all-embracing state involving the balance of the tradeoffs of all performance criteria that can support the desirable state over time.
The OECD has maintained the OECD DAC criteria for many years but only provide very general orientation on how to apply their evaluation criteria but this does not include any specific cross referencing to performance criteria. Under Sustainable Development Goals the specific performance criteria which the evaluation criteria relate to have yet to be defined.
On the other hand, the OQSI was established in 2010 by the George Boole Foundation to develop project design and implementation recommendations as part of the Foundation's Decision Analysis Initiative (DAI 2010-2020). The OQSI breaks projects down into distinct phases where the nature of evaluation focus changes:
- project design - review of evidence presented to justify design and project plan
- operational setup - state of readiness of equipment, tools, manpower training, procurement timing etc
- operational performance - each actvity compared with plan benchmarks
- operational decisions - and their results under conditions of change
- post funding adjustments for sustainability - adjustments intorduced to ensure sustainability
The OQSI states quite specifically what is being evaluated in each phase.
On the other hand, The OECD documentation refers to the fact that all DAC evaluation criteria can be used to evaluate before, during or after an intervention and in a footnote it states "However, the way the criteria are defined here reflects the dominant practice of interim, final and ex-post evaluation
" but the focus in each case is less clear.
The OQSI's recommendations are very much founded on the importance given to careful project design so as to minimise subsequent issues. Part of the procedure here is to undertake a wide range of analyses on scenarios so that if conditions change a team is more prepared to take decisions under change with the support of such groundwork data. This is a basis for securing improved overall performance. The OQSI have a greater focus of the practical need to adjust projects to ensure post-funding sustainability.
The OQSI's recommendations are based on the maximised application of modern information technology generally available in low income countries. and a central database where all design and subsequent events are recorded so that the notion of an interim, final or ex-post M&E does not exist. On completion of any project activity OQSI recommends internal evaluations take place to record results as well as decisions in cases where activities face difficulies. This builds an effective real time monitoring and evaluation system that collects relevant information throughout the design and implementation phases. Because this data is collected by the tam using specific software the quality of project data can be of a high standard and all information is accessible to any external M&E assignment personnel.
What is evaluated?
The OQSI's specific division of evaluations according to project phases. At all stages evaluation, under Sustainable development Goals, needs to maintain a "handle" or objective assessment of the following performance criteria:
- real income
- carbon footprints
- carrying capacity
| Open Quality Standards Initiative Project Phase Evaluations|
| The evaluation of performance over the project cycle|
|Performance criteria|| Principle project evaluation phases|
|Design||Operational setup||Operations||Operational decisions ||Post-funding sustainability|
- real income
- carbon footprints
- carrying capacity
To what extent does the project design correlate with the evidence provided to justify the project configuration with respect to the feasibility of meeting stated performance objectives?
To what extent were all requirements in place on schedule to intiate operations including the delivery and inspection of required supplies and equipment and manpower?
To what extent has the overall project performance met the original or modified objectives in the case of all relevant performance criteria?
To what extent have implementation decisions and their outcomes contributed to performance in meeting planned objectives as well as any justified and modified objectives?
|Record any actual or planned post-funding modifications in operations. To what extent does this modified configuration correlate to the evidence of the project's overall performance and future objectives? |
In the next article in this series, a pracical solution for project teams and farmers will be described to coincide with an annoucement from the Sustainable Development Facility. This will cover the details of their new grant scheme specifically designed to support the provision of advanced project management and farm planning tools for farmers and extension servies through local networks covering the 143 lowest income countries.
|Posted: 20220310||We welcome questions and feedback:|
To submit questions or comments on the contents of this article please contact the author or main reference source by email.
The relevant emails are provided below:
Author: Hector McNeill: email@example.com Source: OQSI: firstname.lastname@example.org