Agricultural Innovation

  Home page

Part 4: Impact analysis - why are so many project results disappointing?

Hector McNeill
George Boole Foundation
Hector McNeill
Where it all began

The work leading to the Real Incomes Approach started in Rio de Janeiro, Brazil in 1975. Initially focusing on the impact of the petroleum crisis on the Brazilian economy. Some of this initial work made use of the resources provided by the Getulio Vargas Foundation located in Botafogo, Rio de Janeiro.

The Real Incomes theory and policy propositions emerging from this work have since become a new front in macroeconomic policy analysis, constitutional & development economics.

The reasons why agricultural project results appear to be associated with reduced impacts are generally linked to:
  • Deficiencies in the measurement and analysis of relevant biometric variables

  • The increasingly inappropriate methods applied to project design that reduce the practical effectiveness of project execution

The handling of natural complexity and heterogeneity in biometric variables such as rainfall, temperature, soil texture's influence on soil water holding capacity relate to agricultural production in specific ways and I will cover these dimensions in a following article.

In this article I want to review how the operational frameworks of agricultural sectors have changed, especially with respect to agricultural extension systems. There has been an associated change in the nature of what is included as critical path activities related to training in projects and this has led to difficulties in measuring impacts. This is of importance in the light of the significant failings in the Agenda 2030 project portfolio and it is a matter of urgency requiring a solution.

The agricultural system

My undergraduate training took place at the School of Agriculture of the University of Cambridge. In 1965/1966 we were introduced to agricultural economics. This topic helped us as students begin to tie together all of the other disciplines we had studied in the previous two years in crop and animal production, physiology, genetics, histology, reproduction, pathology, weed and pest control, plant ecology and soil science (pedology). What was quite exhilarating was how the apparently dull subject of "farm planning" led all of us to begin to fully comprehend how so many considerations that go towards crop and animal production were brought together into an operational "economic model" which after a time became extremely easy to understand, in spite of the complexity of all of the contributing components or "variables" concerned.

This approach to farm planning was in reality a systems approach3 that applied a particularly coherent form of decision analysis but neither of these terms was ever mentioned, it was just "farm planning".

However, since this was our first exposure to "economics" I don't think any of us realized, at the time, how different and practical this training in economics was compared with other form of training in economics. The reason the training at the School of Agriculture being so practically orientated and effective was that our lecturers were mainly economists, most with advanced degrees, who worked as extension officers in the School's Farm Economics Branch. The Farm Economics Branch was the data analytical centre for the East Anglia region of the National Agricultural Advisory Service (NAAS). Part of our training involved having to visit farms, ask farmers questions and then return to try and work out the best farm plan for the coming year for the farmer concerned. The student who worked out the most profitable plan was then required to explain it to the farmer and attempt to advocate its adoption by the farmer for his next year's production. The student was accompanied by the lecturer to these meetings who simply introduced the student to the farmer but during the meeting did not contribute anything; the student was wholly responsible for all explanations. Although this was throwing us in at the deep end it was an amazing experience because we were actually applying all of our knowledge to the resolution of practical issues and the challenge of convincing an experienced farmer was significant.

In the rare week that I came up with the best plan I duly visited the farmer concerned. I thought he would be pleased that by a series of clever maneuvers I had doubled his net profit. However, the farmer, who was very patient, explained why although what I had come up with appeared to be feasible there were other issues of importance to him such as not wishing to oversee operations for more than a given number of hours in any day and some of my suggestions would have had him working overtime.

Because the linear programming technique we used (Swedish Method6) had the objective of maximizing aggregate gross margins, there was an underlying wrong assumption, on my part, that a farmer would adjust his own personal priorities to squeeze out that additional profit. This of course was wrong. I was quite disappointed in the fact the farmer had turned down my plan, albeit in a very gentle manner. But the main lesson I took from this ordeal was that just because something can be achieved does not mean it will be and there is a need to avoid supporting a sort of economic or technological arrogance that ignores the needs of those expected to bring about change. In short, stakeholder needs are all important. This balance between society's needs, such achieving net zero carbon emissions and conserving carrying capacity of the globe, all rely, in the agricultural sector, on farmers having the motivation and means to change.

Bringing about change

Policy traction

Having completed my under graduation in Agronomy I embarked on a post graduate Diploma in Agricultural Economics at the School of Agriculture. This required the compulsory completion of courses in Biometry and Project Evaluation. Project evaluation was an excellent course that made use of World Bank projects as case studies where we were required to assess and complete cost-benefit and other analyses including environmental impact assessments. We were also required to attend lectures on micro and macroeconomics at the School of Economics, Sidgwick Site at the University and sit examinations on these topics as well as complete a dissertation.

I was somewhat taken aback at by the reverence proffered by students by such professors as Joan Robinson, a colleague of John Maynard Keynes, whose lectures I found to be bordering on assertion. There seemed to be less questions raised by students of the lecturers when they produced some rather weird and wonderful graphs based on mathematical models, many of which were counter-intuitive. E. A. G. Robinson who gave a course on development economics had what I would consider to be a more practical transparent approach to economics.

It is notable that Robin Matthews, the Drummond Professor of Political Economy at Oxford University had recently published the results of a detailed enquiry into why the United Kingdom had enjoyed unprecedented growth during the period 1945 through to 1965. The Cambridge economists tended to believe that this was all part of the "Golden Age of Keynesianism". However, Matthews drew the conclusion that this rise in real incomes was not the result of Keynesian policy because none was applied. There was no government borrowing with the government's current account running a surplus in all years and policy, if anything, was highly deflationary. The good economic growth, therefore, had nothing to do with Keynesian economics. The practical reason was an amazing growth traction gained from there being no policy to get in the way and economic units were left to advance their own allocation of resources according to their capabilities and their development of tacit and explicit knowledge in contributing to carrying out post-war reconstruction.

This significant difference between agricultural development economics and macroeconomics theory as taught at Cambridge and later I noticed the same was true of lectures at the Economics Department at the University of Stanford, led me to realize much of this theory did not work in practice and this led me to embark in 1975 on the work leading to the real incomes approach. I will cover in a future article in this series.
The other early lesson that arose from our training in what I would consider to be an "extension environment" was that we also received instruction in agricultural policy. It was not at all difficult to reason that to achieve agricultural policy objectives that these could only be realized if farmers change their practice in some way. For example a policy to raise physical productivity to augment national supplies and lower unit costs of production and also raise import substitution would require rises in productivity at the farm level. In other words the policy impact was wholly reliant on the changes brought about at farm level. Extension services were in reality part of an important national agricultural learning system where demonstrations on farms as well as experimental stations allowed farmers to witness the performance of new crop varieties or the introduction of some new equipment. Where there was some physical action required to some manual technique, extension officers would invite
Knowing and knowing about - tacit and explicit knowledge

In 1949 Elton Mayo observed that,

"A simple distinction made by William James in 1890 has all the significance now that it had then; one can only suppose that its very simplicity has led the universities to brush it aside as obvious, which is true, or as of small account, which is not true. James pointed out that almost every civilized language except English has two commonplace words for knowledge-connaitre and savoir- knowledge-of-acquaintance and knowledge-about. This distinction, simple as it is, nevertheless is exceedingly important; knowledge-of-acquaintance comes from direct experience of fact and situation, knowledge-about is the product of reflective and abstract thinking."

Thus, knowledge-of-acquaintance is tacit knowledge and knowledge-about is explicit knowledge.

In the case of the "technical progress model" it was only in 1957 that Robert Solow created a technical progress model of the US economy that estimated that about 80% of the economic growth in US output per worker was attributable to technical progress. Robert Solow did not concentrate on technique but obviously without this element technology will not be operational. Nicholas Kaldor also developed models exploring the impact of technology.

The learning by doing aspects of technological progress was advanced by Kenneth Arrow in 1962 who developed analyses to show that the process of technical progress was the result of learning by doing, thereby explaining the mechanism for achieving technological advance (innovation) and economic growth being intimately tied to the accumulation of tacit knowledge.

The contribution of learning to the accumulation of tacit knowledge and the associated rise in productivity was first established as the foundation of the learning curve by Theodore Wright in a paper in 1936 analyzing the factors affecting the cost of aircraft frame manufacturing.

See: McNeill, H. W., "Tacit and explicit knowledge", Real Incomes.
farmers to try out the technique themselves so as to "get a feel" of what was required to introduce change. In that period the estimated contribution of extension services to agricultural productivity in the United Kingdom was 2% each year or about 22% each decade. Given the importance of the agricultural sector to family budgets and in a post-war period with food rationing having ended just 12 years before, this was a very significant contribution to the growth in real incomes (purchasing power) of families. The lesson here is that policy can be effective if the details of how change on farms have been worked out and supported on what is essentially considerations all on the supply side of the economy.

So how did this happen?

The fact that economic development and real income growth arises from learning has been implied by many for centuries but in economic circles was only formerly established in the 1960s. The process of learning is made up of the accumulation of tacit knowledge (see box left) by individuals and who become increasingly competent are different tasks as a result of repetitions and intimate operational acquaintance with the techniques involved. The result of this learning curve is that tasks take less time, they are associated with less mistakes and waste and with falling unit costs.

The other aspect of learning and communication is explicit knowledge in the form of measurements and data sets which are used to analyse and communicate performance, to diagnose performance and suggest better ways to achieve objectives. This leads to change in practice and innovation as the leading edge of technological and economic development.

Monitoring and evaluation

One of the changes in project cycle management that occurred during the last 50 years was that initially training of those project stakeholders was not included by some donors in project implementation budgets. Then as from around the mid-1990s, training became a specific task more widely included in projects. The majority of training and education in schools and universities tends to be academic and highly theoretical. The word academic in fact signifies:

“relating to schools, colleges, and universities, or connected with studying and thinking, not with practical skills”.

As a result, an increasing proportion of so-called “training sessions” included in projects did not contain practical orientation of how to accomplish required changes in a practical way. For example, attendees would be provided with lectures on sustainability and the benefits of water conservation but there were no practical field sessions providing the opportunity for farmers, for example, to carry out the procedures in practice. This is an essential step in establishing the necessary level of confidence arising from having carried out the procedure and internalized the necessary actions, even for a short period. This is sufficient for each individual to initiate their own tacit knowledge development pathway on their own accord improving their operations in the meantime. The cognitive ergonomics of training needs to emphasize the accumulation of tacit knowledge and the only way to achieve this is to have instructors who show farmers what they need to do in their own fields and then provide the farmer with the opportunity to actually carry out the physical procedures necessary themselves; not in theory but in practice.

Analysis by SEEL, our decision analysis unit, has concluded that this misalignment of course content with project objectives is why many M&E reports increasingly showed1 multiple “successful completions of training” as well a high degree of “comprehension on the part of attendees". However, when it comes to the actual changes in operations as the objective of the projects, many M&E reports concluded that, for example, the degree of application of necessary conservation practices, was either very low or even zero. So the training and even understanding of necessary principles of the action required does not, in general, translate into practical action and therefore impact. Since the objective of projects is the quantitative impact which the M&E is meant to measure, it needs to be concluded that low impact is the result of poor project design or in the case of ongoing sector operations involving farmers, low impacts are the result of a failure to emphasize tacit knowledge development and therefore very practical forms of training.

This very simple problem which was overcome by extension services many years ago by keeping their focus on the importance of tacit knowledge through “farmer field demonstrations”. Where project designers do not possess the required applied knowledge to offer such an approach it is therefore necessary to arrange field demonstrations for farmers, preferably organized by the farmers, and inviting practitioners to carry out the training. However, emphasis needs to be given to an applied hands-on approach so the farmers leave with a new capability as opposed to just new concepts. In terms of transfer of know-how-do, the time taken for an individual to acquaint themselves with a new practical technique is surprisingly short if they are allowed to "do it" and this has more effect than several days of lectures or what used to be called "chalk and talk".

Project impacts

Jumping some 25 years, from undergraduate days, we arrive at a time where because of the short term vision of politicians, the world over, caused by the desire to show "performance results" within their relatively short administrative periods, there was a general defunding of extension services. An instinctive ideology emerged linked to the Bretton Woods economic development philosophies promoted by the IMF and World Bank. This defaulted to a perception of public services, such as extension services, being "inefficient" in terms of cost-benefit, even when no cost-benefit analyses showed this to be true. So extension services were defunded in the expectation that market forces would supply farmer's requirements. This resulted in two trends. The loss of highly experienced extension personnel and linked research centres and a decline in the number of visits made by advisors to farms. Input suppliers made use of private "extension agents" who in many cases ended up as commission sales agents for the companies concerned.

The other outcome of this trend was the marginalization of small farms where because of lack of cash flow private extension agents would not spend time visiting such farms because of the likelihood of not being paid or not making a commission sale. Several companies overcame this by offering production inputs finance but because of the natural variations in yield associated with weather as well as lack of effective advice, a large pool of poor farmers with rising debt developed over this period. This remains a serious problem.

By the 1990s we see that more than 35% of internationally-financed agricultural projects were failing and the percentage of projects authorized by the World Bank Board having been subjected to cost-benefit analyses fell from around 85% in the 1960s to around 20% in 20104.

In the early 2000s a move was taken to replace the extension services in Mozambique with an NGO-based system. Naturally when the funding for this "initiative" was terminated, Mozambique farmers found themselves without extension support since most of the former agents had to leave the government service as a result of the NGO-based initiative displacing them from their former operational tasks2.

If we now move on another 30 years we find that the 2019 Sustainable Development Report highlighting the fact that there is a negative correlation between "economic growth" and the following Sustainable Development Goals:
  • SDG 10 Income disparity
  • SDG 12 Sustainable production and consumption
  • SDG 13 Climate action
To advance Agenda 2030's Sustainable Development Goals requires a dedicated extension system

The European Union faced similar difficulties in advancing the environmental agenda embodied in European Law and Regulations because farmers found it difficult to balance economic viability with the introduction of environmental regulations. Many farms were facing sanctions because the farmers did not know how to adjust their production practice to comply. The same is currently the case under the Agenda 2030 regime but the scale of the problem is global.

In 2013 the European Commission adopted legislation to create a Farm Advisory System (FAS) (Regulation (EU) No 1306/20137).

The FAS helps farmers to better understand and meet the EU rules for environment, public and animal health, animal welfare and the good agricultural and environmental condition (GAEC). The FAS provides information about:
  • obligations at farm level resulting from the statutory management requirements and the standards for good agricultural and environmental land conditions (“cross-compliance”);
  • agricultural practices beneficial for the climate and the environment and maintenance of the agricultural area (“greening”);
  • measures at farm level provided for in rural development programmes for farm modernization, competitiveness building, sectorial integration, innovation and market orientation as well as for the promotion of entrepreneurship;
  • requirements for water protection, efficient and sustainable water use;
  • use of plant protection products;
  • integrated pest management
A need for action

So far, too little action has occurred on this front in the Agenda 2030 developments, as the insignificant impact on the performance of the Agenda 2030 project portfolio demonstrates. The EU FAS is not exactly what is needed for Agenda 2030 because it is based largely on information exchange (explicit knowledge) and lacks the essential farmer field demonstration components, but it is an example of an action taken to address the same sort of issues facing Agenda 2030.

It is so important for action to be taken on this front that it would make sense to have a dedicated SDG solely for the purposes of establishing this vital infrastructure to support the transformation of the agricultural sectors to comply more easily with the needs to support several SDGs. As we see population numbers expanding and carrying capacity of renewable natural resources declining as a result of their over-use and climate change we, as members of the world community should not accept this state of affairs but should advocate for change.

More recent developments linked to impact analysis

The record on project impacts has led to several attempts, since 2000, by several organizations to improve the diagnostics of project outcomes and this has led to an assumed association between the lack of project impacts to psychological, knowledge, literacy, attitudes and cultural reasons. Since 2010, this has given rise to elaborate schemes such as the World Bank’s Evaluation Group’s CrI2SP set out in the publication, “Evaluating Behaviour Change in International Development Operations : A New Framework 2016”) and the 238 page report by Wageningen University’s (M4SDi) approach set out in the document entitled “Managing for Sustainable Development Impact”.

To review these approaches would take up too much space in this article but I will prepare a review of each one. However, at first sight it is notable that neither of these approaches emphasizes the importance of tacit knowledge and for projects to establish practical changes in practice rather than "talk about it". It could be that the intent is not to include agricultural projects. The M4SDi Report mentions tacit knowledge but seems to consider it to be something other than what is generally understood and therefore its significance is not grasped. M4SDi is stated to address Agenda 2030 SDGs, so agriculture in the context of food security, food and nutrition is an important component of solutions to several SDGs. However, these approaches do not appear to address the specific issues of how to change agriculture so as to drive agricultural innovation to address sustainability issues in a practical fashion and to address Agenda 2030's Sustainable Development Goals.

It is apparent that there is a gap that has appeared over the last 50 years where the traditional more down-to-earth (literally) extension type approaches to project activities and to impact assessment have been lost.

A specific underlying theme of many project design guidelines is the so-called Theory of Change (TOC) which appeared some years ago but which does not have well-developed methodologies to determine risks, probabilities, deterministic models of farm production options and how to ensure post-funding sustainability. We have worked in this field for a decade under the Decision Analysis Initiative and the OQSI. If the change trajectory is based the change on tacit knowledge, there is a known starting point for productivity and by applying normal learning curve relationships and performance benchmarks within a project and then ensuring that farmers, as beneficiaries, receive the correct applied experience, then there are fairly precise methods to estimate the expected level of achievement attained according to throughput. In this approach the output is a Logical Project Option that describes a change strategy which in reality is the structure and purpose of the whole project. This approach needs to based project design applying an appropriate due diligence procedure5.

The Médecins Sans Frontières’s (MSF) KAP (Knowledge, attitudes and practice) procedures, specifically developed for biomedical domains, including nutrition, are more relevant to the biomedical issues related to family and in particular mother awareness of what is required to purchase and prepare nutritious foods for their families or take necessary precautions in relation to hazards related to domestic hygiene and the access to and handling of fresh water. However, this approach does not seem to relevant to the process of change required in farm practice.

It is worth mentioning that the former extension systems were organized on the basis of farm visits by a technical extension officer accompanied by a domestic economist, usually a woman. The technical extension officer spent more time with the person responsible for the farm operations, usually the farmer to review production performance. The domestic economist spent time with the farmer's wife and was there to inform the wife on such issues as home economy, nutrition, ways to manage the budget for foods, health, hygiene and other issues of importance to healthy families. At the same time the emphasis was on practical demonstrations on how to handle food, and other practical issues that were the subject of visits. At other times these agents would hold community meetings in village halls and other local venues easily accessed by members of the local population.

Traditional extension systems

The traditional agricultural extension system is an effective communications and innovation framework and it would seem that all countries would be advised to re-establish or expand existing services so as to reinforce a service to support the needs of all sizes of farms and in particular those in the lowest 40% of income and performance. The diagram below shows a structure of the elements of a traditional extension service which encompasses contributions from many actors.

1 "The impact of deficient project design on impacts", Decision Analysis Initiative (2010-2020), SEEL-Systems Engineering Economics Lab, September, 2019.

2  Gemo, H. and Eicher, C. K., "Mozambique' Experience in Building a National Extension System", Michigan State University, 2005. Personal communications from Helder Gemo the author on 25th August 2006 in Maputo.

3  I was introduced to both the systems approach and decision analysis as disciplines a few years later at Stanford University in applied systems development courses under "Space Systems" organized by the Electrical Engineering School. These courses were outstanding because they, like "farm planning", required students to apply their knowledge to practical real world issues such as the development of whole systems for natural resources survey and marine management systems.

4  "The Impact of trends in the international development environment on project performance", OQSI, Decision Analysis Initiative, 2018.

5 "3DP-Due Diligence Design Procedure - for the development of logical project options", OQSI Recommendation 1 Release June, 2020, OQSI.

6 "The Swedish Method" was a very convenient "back of envelope" method of emulating the SIMPLEX linear programming algorithm which, with practice, could generate crop land allocations that were very close to the computer-based SIMPLEX farm plans generated by the Farm Economics Branch. It was used by Swedish extension services. This system features in Alasdair M. McFarquhar's paper in the Journal of Agricultural Economics, "Research in Farm Management Planning Methods in Northern Europe", 1962..

7 Regulation (EU) No 1306/2013 of the European Parliament and of the Council of 17 December 2013. The farm advisory system should cover at least the obligations at farm level resulting from cross-compliance standards and requirements. That system should also cover the requirements to be respected in relation to the agricultural practices beneficial for the climate and the environment and the maintenance of the agricultural area under Regulation (EU) No 1307/2013 of the European Parliament and of the Council and measures at farm level provided for in rural development programmes aiming at farm modernization, competitiveness building, sectoral integration, innovation, market orientation and promotion of entrepreneurship.

Posted: 20200721
We welcome questions and feedback:

To submit questions or comments on the contents of this article please contact the author or main reference source by email.
The relevant emails are provided below:

  Author:  Hector McNeill:         Source:   DIO: