|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Green Plan Evaluation
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
BackgroundThe Canada-Ontario Agreement of the Agricultural Component of the Green Plan, an equally-shared Canada-Ontario program, was designed to encourage and assist farmers with the implementation of appropriate farm management practices within the framework of environmentally sustainable agriculture. It was delivered over a five-year period from 1992 to 1997. Under the Agreement, the federal component was delivered by Agriculture and Agri-Food Canada's Market and Industry Services Branch, from the Ontario regional office in Guelph. The three objectives of the Green Plan were:
The Agriculture Green Plan in Ontario consisted of seven program areas, namely, Technology Transfer, Research, Best Management Practices, Rural Conservation Clubs, Wetlands/Woodlands/Wildlife, Stewardship Information Bureau and Environmental Farm Plans.
Evaluation ProceduresThe program evaluation was conducted using an Evaluation Framework specified by the client. A work plan was developed and followed in conducting the data collection procedures. The collection of data for the evaluation involved: a review of records, files and reports; and the use of 14 questionnaires with a total of 314 respondents. The personal, telephone and group interviews and mail surveys which were conducted are outlined below in the Data Collection Summary.
Data Collection Summary
The data were tabulated and analyzed to answer the various evaluation questions identified by the evaluation framework.
Evaluation FindingsAlthough the focus of the evaluation was the Green Plan, it became apparent that data would have to be collected, not about the Green Plan per se, but about each of the seven programs. The responses of those individuals who designed, delivered or participated in each program were presented for each evaluation issue. Only the overall summary for each evaluation issue is outlined below. For further details of responses regarding each program, see the body of the report. The general findings for each evaluation issue were as follows: 1.0 Rationale
A large majority of respondents had a copy of or claimed they were aware of their program goals. They believed their program goals were clearly specified, consistent with Green Plan goals and were appropriate. The consistency of responses indicates an unusually high awareness of and support for the program goals. The program goals identified for the seven program areas were consistent with the Green Plan environmental objectives. We also determined that a high degree of awareness of these goals exists throughout the program delivery staff.
The respondents overwhelming believed that the operational objectives of their work plan were clearly specified and consistent with program goals. They also claimed a high awareness of their work plans.
A need exists for programs which provide original research on agronomic and environmental problems as well as a system which communicates this and existing information to farmers and landowners. Farmers need both technical and management data. While the specific programs have not been evaluated, it is obvious that there are a very large number of landowners who have not yet adopted many of the known means of enhancing the environment. The EFP, BMPs, Conservation Clubs Type B and the WWW projects were innovative and demonstrated new ways to encourage farmers and landowners to better manage their soil and water resources.
2.0 Objective Achievement
The operational outputs and measures were clearly specified but the actual numbers of participants were only specified for the Technology Transfer and EFP programs.
Based upon the evidence available, primarily self-reports, the programs were generally carried out as planned. A number of program delivery problems were identified but most did not jeopardize the overall success of the programs. The SIB was an exception in that it was not carried out as initially planned.
A majority of respondents interviewed from each program believed there was adequate flexibility in responding to project needs and in making necessary adjustments.
A relatively wide range of opinions were found as to the percentage of the program objectives expected to be achieved by the end of the program. About one-fifth of the respondents did not know or would not make an estimate. Most believed 71% or more of the objectives would be achieved. The most optimistic were the researchers, all of whom believed over 90% of their objectives would be achieved. The members of the three Committees were less optimistic than the others as half of those who made an estimate said 70% or less of the objectives would be achieved.
The responses of the program participants provide very strong evidence that the farmers interviewed and those farmers observed by the program planning and delivery staff consider environmental issues when planning their farm activities. The consistently high level of consideration of environmental issues shown by the various respondents, especially by the farmers, indicates this objective is being achieved.
Participation in Green Plan programs had a major influence on the adoption of environmentally sustainable agricultural practices among participants. Among the Conservation Clubs and WWW participants, approximately 90% said the program influenced their adoption of new practices. The reports of over 80% of the Conservation Clubs staff and EFP local county committee members confirmed that their programs had also influenced farmer adoption of new practices.
3.0 Program Design and Delivery
The various programs were delivered in different manners, some directly by AAFC, such as Research and Rural Conservation Clubs, some by third parties such as WWW, EFP and SIB. The others, Technology Transfer and BMPs, while partly financed as third party delivery activities, were completed by committees made up of AAFC and OMAFRA staff with the help of consultants. Alternatives to various programs were proposed. The majority of respondents did not believe their program could be delivered at a lower cost. The Committee members suggested reducing the cost of the Research, EFP and BMP programs. The interviewees, including managers, staff and participants, generally believed that their program delivery approach and the actual on-site delivery were quite effective. In most cases, a few individuals dissented. One of the most frequent complaints was that there was either too much administration or the administrative costs were too high.
While we believe the priority setting and fund allocation are reasonable, there is no empirical means of determining, at a reasonable cost, the optimum environmental benefits. Additional funding of research on adoption and diffusion theory and training in extension methods was discussed. The failure of the EFP program to meet their participation objectives and under utilization of grants could be considered the single largest error in allocation. These monies were later allocated to other programs mainly related to ground water management. The SIB was almost terminated early in the program but a decision was made to continue the Bureau. Closure would have had relatively limited impact on either the overall program achievements or budget. The Green Plan evaluation activities were not optimized because they were: late in starting; disjointed by the use of three consulting teams and changes in the evaluation design; underfunded; and, in the case of the benchmark survey, extremely rushed. The result is three evaluation studies which lack continuity because they each address very different issues and do not facilitate detailed measurement of the environmental benefits of the Green Plan or the seven programs.
On the basis of the relatively consistent comments of the respondents, we conclude that the programs provided adequate technical information on various innovations. The generally perceived inadequacy of economic information is symptomatic of a long standing emphasis on the biological and technical characteristics of innovations at the expense of the economic costs and benefits. Greater emphasis should be given to the profitability of innovations as a means of encouraging their adoption.
The Green Plan programs did not address all barriers to adoption. The areas most frequently ignored were economic in nature. The lack of information on economic returns or implications of the practice, the cost of investing and the sources of financing were all cited by the respondents interviewed. The results are consistent with those found for the previous question.
The respondents were able to differentiate between indicators of the success of an environmental program and those which are indicators of environmental improvements at the farm level. The answers provided tend to overlap quite a lot with slightly more specific innovations being identified as on-farm indicators. The most frequently cited indicators of general and on-farm successes were: the level of adoption of specific innovations; measures of improvements in the environment such as water quality and achievement of program objectives. There was no consensus among the respondents on either question. The most obvious indicators appear to be measures of the extent to which a new practice or behaviour has been adopted or tried by the farmers and measures of changes in the environment.
4.0 Other Impacts and Effects
Many unintended program effects were identified by the respondents. Others were apparent to the evaluation consultants from reading various reports and meeting with program personnel. Most of the unintended impacts were positive in nature and flow from the success of specific programs or activities. Some could have been anticipated but were not identified in advance because they were not objectives of the specific programs. We believe the following positive impacts have occurred:
These developed within the agricultural industry, within rural communities, between rural and urban groups and between landowners and various government agencies and non-government agencies. These occurred within the three Accord Committees, during several programs and as a result of specific projects. Third party delivery provided existing organizations with financial benefits and prestige. Rural Conservation Clubs and WWW projects gave local groups the opportunity to undertake useful projects. The OFA and OSCIA gained experience and credibility. This empowerment helped establish the socio/political environment which made the Agricultural Adaptation Council possible. The Rural Conservation Clubs and WWW Clubs brought people together at both the organizational and operational levels. A few of these will continue to undertake similar or new projects. The web site established at London by AAFC provides a new scientific source of environmental information which is available to rural and urban populations. The negative unintended impacts tended to be related to project failure or to be very situation specific. The impacts were relatively minor such as unhappiness with a contractor, excess geese and planting trees improperly.
5.0 Alternative Delivery Systems
The management system was, as stated by the respondents, excessively complex, diffuse, slow and cumbersome. The system of three Committees plus an administrative officer, but not a manager was designed to provide high level involvement by the many stakeholders. We believe the program would have benefited from the designation of a single program administrator who would have had primary responsibility for managing the Green Plan.
The opinions of the respondents regarding third party delivery were mixed with many people providing responses which indicate a limited understanding of all the potential cost factors or the level of actual costs. One cannot determine the incremental costs of third party delivery until such time as benchmark studies have been conducted on government delivery costs. The extent to which third party delivery is effective depends upon the type of program being delivered and who the third party is. Third party delivery works best when the clients identify more closely with the third party than the first party. It also works better when the program is primarily action oriented, requires on-site delivery and is of intermediate technical requirements. One consideration must be the degree of accountability. The first party needs to develop precise, measurable objectives which are agreed to by the delivery agency. The work activities of the third party should be monitored to assure performance. Accountability should be maintained in all situations even those of a political nature.
6.0 Future Directions
Overall, the subjects or issues receiving the greatest number of mentions by the respondents were: soil conservation, management and quality; water quality, both surface and ground; reduced pesticide use; waste management; and the protection of natural areas, habitats and the environment.
The respondents perceived the most important role of each to be:
Conclusions
Recommendations
Green Plan Home
Created: 16 November, 1996 13:26:59 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
