Home page

Leading Issues

Development Intelligence Organization

Open Quality Standards Initiative's project due diligence design procedures - are they practical?
Part 2

Reproduction authorized by DIO and APEurope.org

The Development Intelligence Organization's annual "Leading Issues" workshop took place in Alexandria, Virginia, 9-11 August, 2019. The Keynote address by Hector McNeill of the George Boole Foundation (GBF) was extensively discussed. This outlined a set of new evaluation criteria for projects, programmes and policies, in the context of Agenda 2030 Sustainable Development Goals. In particular, these have been designed to accommodate and adjust project design and implementation management to support Reduced Inequality (SDG10), Responsible Consumption and Production (SDG12) and Climate Action (SDG13). Hector McNeill, the Director of GBF, is a leading development economist who has worked in this field now for over 50 years. He is the world's leading developer of the Real Incomes Approach to economic development having initiated this work in 1975. This specific topic was covered in a separate interview that took place in 2015.

This interview, is divided into three parts. The first part deals with the problems the OQSI work has been designed to solve. These relate to gaps in current practice that exacerbate the state of affairs of SDGs 10, 12 and 13 as well as many others. Once this background is clarified it is fairly easy to understand the purpose and likely impact of the OQSI's new project due diligence design procedures (3DP). Below is the content of Part 2 covering some of the shortcomings in common administrative arrangements and procedures applied to project cycle management and Part 3 will cover the OQSI proposed solutions where I assess their practicality.

Nevit Turk
Economics Correspondent
Agence Presse Européenne, Centre Montparnasse
email: nevit.turk@apeurope.org
January, 2020    

Hector McNeill
Nevit Turk: I would like to relate what you have referred to as issues needing to be resolved with respect to the disappointing performance of SDG 10 reduced inequalities, SDG 12 responsible consumption and production and SDG 13 climate action, to the contributions of the OQSI procedures and criteria recommendations. I want to understand how practical this is in terms of ease of introduction of these recommendations and to assess their likely impact. In particular I want to maintain the linkage to the way in which these contribute to resolving the issues posed by poor performance of SDGs 10,12 and 13 In short, how can the new OQSI recommendation contribute to gaining better performance.

The significance of Project Memory

OECD-DAC evaluation criteria that are commonly applied during evaluation assignments can be traced to the “process approach” (ISO:9000 and BSI series) applied by organizations, manufacturers and product and service development teams perfecting the design and delivery of processes, products and services.
A coherent project memory

Functional overlaps increase coherence of information and records
The successful applications involved tight knit groups within single organizations, collaborating closely as a cluster of specialists, all dedicated to working on a design and testing cycle to perfect some product or service. This approach is the foundation of successful quality management and zero defect manufacturing.

The process diagnostic approach

This is a diagnostic procedure that is reiteratively applied to the improvement of the design of a product or process through testing and evaluation. The testing and diagnosis of the outcome of a development cycle is in reality an evaluation. In this system evaluators have a central role in contributing, as members of the development group, to the perfection of the product or service under development. On this basis, “lessons learned” are identified and applied immediately to the next part of the design cycle and not after the product or service is launched. These close-knit teams consist of funder representatives, designers, evaluators, sometimes potential users, workshop personnel and other technicians. This system has proven to be highly successful in terms of the development of reliable minimum defect products and services. This operation is illustrated below left. The most significant benefits of this approach are:

1. The establishment of a coherent project memory.

2. Those with a critical function in this continuous process are evaluators.

3. The repetition of the same agreed analytical methods covering the cycle steps

4. The high standard of monitoring record keeping and documentation as a basic quality assurance procedure which provides any newcomer with a comprehensive review of procedures and any aspect of a system’s development

The focus on project memory is an important aspect of OQSI development work. It signifies that the level of detailed knowledge of the process and product is high so that someone within the product/service cluster will be able to answer almost any question on any aspect of the current or past status of the cycle. This is because of the team’s close association and necessity for a joint understanding of all tasks in the cycle.

Conventional economic development project approach
Atomized project memory

The lack of functional and communications overlaps undermine the coherence of information and records

In the case of economic development projects general field of economic development, this type of tight cluster seldom exists. Very often different people work on each part of the cycle, sometimes from different departments and even organizations. An example of typical participants are shown on the right

This type of arrangement can result in two challenges:

1. A common occurrence is that some members of these groups, during the course of a project cycle, leave for other types of occupation or move within their organization and no longer participate in the project. As a result the parts of the joint project memory can be lost.

2. Compensation for loss of project memory is sometimes difficult because project documentation is of an inadequate quality. This is often the result of the multi-departmental or organizational involvement in different parts of the project cycle. This often prevents the production and maintenance of a single coherent centralized and fully up-to-date documentation on the whole project cycle to date.

Extract from, "The relevance of DAC criteria for the evaluation of development assistance", OQSI, 2018.

Note: Hector McNeill mentioned that this reference is being updated, extended and will be re-issued to include the latest OQSI recommendations discussed in this interview series: N. Turk.
Hector McNeill: In the last session we reviewed some of the reasons several critical SDGs are not performing related to economic policies, project failures resulting from design deficiencies, the needs to balance economic rates of return with rates of return to the environment and the distance between policies and the content of projects and programmes. I would like to elaborate on some fundamental problems associated with procedures commonly applied in designing, implementing and evaluating projects and programmes. In this context I refer to a section in the 2018 OQSI report entitled, "The relevance of DAC criteria for the evaluation of development assistance". In this report there is a specific section that explains what the OQSI refer to as project memory.

(Inserted note: The relevant section of the report referred to is contained in the box on the right - N. Turk)

Project memory is a documented record of all of the cumulative experience, analyses, narratives and data generated by all activities contributing to the projects identification, design and implementation, including implementation decisions and their results. A review of thousands of projects by the World Bank Evaluation Group (2010) reported that the completion of tasks by different departments and institutions led to a fragmentation of information on a project, there was a tendency for documents covering previous phases to be missing at evaluation assignment stages. In some cases the results of economic rate of return calculations were missing. A particular issue linked to this poor documentation is the damage done to project memory when a key member of a project team leaves at any stage in the cycle. Those taking their place have no good documentary sources to come up to speed. Naturally this situation also results in evaluation assignment personnel not having access to complete sets of documentation.

In short, the current administrative arrangements are inadequate both in terms of oversight of use of funds, the quality of design and of management of project, programme and policy implementations.

Nevit Turk: But what are the alternatives? I realize this is a bit bureaucratic but how does this relate to Agenda 2030?

Hector McNeill: In addition to administrative structures working against a useful and coherent project memory, the live communications are also inadequate. In spite of W3 (world wide web) capabilities we still observe very slow reporting cycles to fit in with internal administrative procedures within government, development banks and other donor organizations which is exacerbated by the distance of these organizations from the project level activities. For example, if a portfolio manager is asked about the progress in any particular project, reference is made to reports that often have passed through a highly tiered reporting structure. This not only takes time but also can contain out of date and even distorted information.

Nevit Turk: Why is this information distorted?

Hector McNeill: In highly-tiered reporting structures each administrative level adds information to project reports which reflects the interests of those vetting reports at each level. Because performance at any level is a reflection of the effectiveness of management of that level there can be uses of words more designed to divert attention from the possible identification of the facts surrounding performance deficiencies. Also, sometimes a project can be delayed because of internal administrative procedures. For example, Ministries of Finance often operate on the basis of a standard accounting year, and don't coordinate payment transfers to agricultural projects to fit in with the agricultural year. As a result funds can be made available at inappropriate times. I have seen cases where projects have been delayed because of this type of inefficiency causing projects to face ver a year in delays in implementation. There are also examples of international agencies who supply the funding, following established time lines for budget line activity and simply closing the budget lines of perfectly good projects, with competent teams, so as to end up with a failed projects. Such failures can be the result of failures in communications caused by inappropriate administrative procedures. The closing of budget lines can be related to tidying up officials' portfolios to show efficiency in completing budget allocations and completion of internal admin cycles, with no insights or concern with the project level circumstances. In other words emphasis on internal adminitrator's performance can take precedence over the substanative issue, project performance.

Just as an example of another administrative constraint. Procurement procedures are quite often not handled by a project team but by specialized sections of Ministries of Finance or Agriculture. Project teams seldom have any direct experience with the legal frameworks, regulations or procedures involved in procurement. They are just expected to comply with the regulations and procedures. The details are managed by separate agencies. A common oversight in project designs and proposals is to not allow sufficient time for the normal procurement cycles to be completed. These should include, for example, calls for tenders, assessment of tenders, contracting and then time to product and service delivery to the project, commissioning at setup ready for use. Many research projects become delayed because of this simple oversight and production seasons can be lost in the case of agricultural production projects with essential equipment turning up afer long delays.

Nevit Turk: But isn't that sort of thing picked up by evaluation teams in lessons learned?

Hector McNeill: Well, you are highlighting what is part of the problem, rather than the solution. What is the point of waiting for this to happen and then organizing an evaluation assignment only to be informed this happened and classify it as an issue to be avoided in the future? The system needs to be organized so that this could never happen. It should also be organized so that any problems and lessons learned should be applied in real time so as to benefit the project in question. Being informed about such matters after the event and sometimes after a project has been completed is hardly an effective use of resources. For example by including financial and procurement administrative procedures as part of the identification phase constraints analysis, such risks can be identified and be accounted for in designs and project plans.

Nevit Turk: How can this be achieved?

Hector McNeill There is a need to short-circuit the process between on the ground analysis of gaps and needs, identifying priorities, designing solutions and then assigning funds to those that have been identified as priorities. As OQSI work has identified, in the multi-organization system, there is a problem with communications and lack of coherence in records but also nationally, this is exacerbated by the vertical distribution of qualified human resources with too few working at the design and implementation levels. As a result, with the growth in complexity associated with SDG solutions design and management, the issue of the number of people available to undertake the necessary work of an acceptable quality is a, so-far, hardly mentioned problem.

We know the rate of project failure under the current system is costing some $75 billion each year, so there is a need to solve this issue while also improving the relevance of all projects to solving the climate crisis. The evolving crisis in climate action requires more competent practitioners to work directly on projects so as to ensure sustainabilty and climatic positive projects through sound ERRs (Economic Rates of Return) combined with sound RREs (Rate of Return to the Environment).

An essential factor in successful projects in the involvement of stakeholders in the whole project cycle. Stakeholders need to include both potential beneficiaries as well as those who might be prejudiced by the growth of some activity. An early involvement of all is an important aspect of gaps and needs analysis so as to fine tune the identification of project targets and thereby clarify the existing constraints facing those targets. Too often, stakeholder involvement is almost cosmetic but, from my own experience, such involvements require more effort because this helps to avoid over and under-ambition and help reduce likely risks. OQSI procedures do make provisions for the involvement of stakeholders including oversight of project performance.

Nevit Turk: So we not only face an administrative constraint but also one related to the numbers and distribution of qualified manpower. But how can these issues be addressed within the sort of periods set out as time lines for projects, after all, Agenda 2030 has set the year 2030 as a milestone in progress whereas 5 years into the programme we are going backwards and there are only 15 years left. How can the necessary changes be accomplished in time to have any positive impact?

Hector McNeill Well, you have summed up the nature of the crisis facing us in your question. This is the reason OQSI altered its focus after 2015 to ensure that the issues of technology and know how transfer to more people can be accomplished within a short period. Fortunately modern technologies and the W3 (world wide web) offer an effective solution. The focus has been on the development of good practice meaning introducing procedural methods that facilitate the identification of priorities by providing support for project designers. The foundation of this development is a set of due diligence design procedures (3DP) that cover all relevant factors in order for these to be given due consideration. Rather than leave required analysis as narrative in a guideline the OQSI system is accompanied by analytical tools to complete necessary calculation, simulations and projections. This lays down the foundation upon which to build project memories to which to add all design information, proposal appraisal information, implementation events, decisions and results and evaluations. An important requirement is that project memory remains as an accessible resource to all with an interest in a project.

Nevit Turk: Thank you for explaining these administrative constraints that need to be overcome and the direction OQSI is going. It isn't clear to me yet how this fundamental issue with macroeconomic policies, being an important part of the problem facing SDGs, will be addressed. There would appear to be issues associated with the quality of project design and a major issue related to the need to upgrade the capabilities of the people involved in managing project cycles. Potentially this must involves an enormous number of people but who currently are not in place or do not exist. In the last part of our interview I look forward to hearing how the OQSI recommendations address these issues in a practical manner.

Posted: 20200410
We welcome questions and feedback:

To submit questions or comments on the contents of this article please contact the author or main reference source by email.
The relevant emails are provided below:

  Author:   Nevit Turk:     nevit.turk@apeurope.org         Source:   Hector McNeill:     hector.mcneill@boolean.org.uk