Educational Technology Evaluation Design
Introduction
Sun Associates, an educational technology consulting firm, specializes in designing customized evaluation and assessment procedures to evaluate and assess the impact of educational technology on teaching and learning. Our evaluation work is designed to take schools and districts beyond their current strategic technology plans and into an area where they can begin to assess their progress in meeting the vision and goals of these plans. Our work centers around a process which combines practice profiling, the development of indicators of success, and the creation of data collection procedures which are to be used to perform a formative progress evaluation of a school or district's educational technology initiative. In many cases, our clients choose to combine our work in designing an evaluation procedure with an audit of current technology implementation, and various reporting options. These options might include a final written report which details the developed evaluation strategy and provides a summative evaluation for the activities of the school/district's technology efforts.
Framing the Task
Many of Sun Associates' clients have made dramatic commitments to implementing and integrating technology in all areas of their educational environments. These schools and districts are making steady progress towards fully integrating technology tools in all areas of teaching and learning. Nevertheless, as with all well-managed implementation efforts, questions of performance must be asked. These questions are necessary not just to find out what is and is not working, but formative evaluation of performance is a critical factor in fine-tuning and improving future implementation efforts and planned activities. In this way, evaluation is a key component of all educational technology planning and implementation efforts. The question then turns to one of how to best accomplish this evaluation.
Activities
Sun Associates works with school districts across the country to design and implement evaluation strategies which inspire reflection and change. Our process for doing this is rather simple and roughly breaks down into the following steps:
1. Development of evaluation priorities with the participating district.
We understand that the evaluation should focus specifically on the activities of a school or district's current technology initiative. In many cases, this means that evaluation will be taking a hard look at the impact that staff development has had on technology implementation. In all cases, we resist looking at only technology infrastructure. Districts are fully capable of "counting computers" on their own. Our task is to help schools and districts articulate their original vision and teaching and learning goals into viable evaluation priorities and questions. In other words, we help districts answer the question "What difference is all this technology making in terms of teaching and learning?"
2. Work with a representative team of district staff - classroom teachers (users), administrators, trainers, and others as identified by client - to develop practice profiles of what constitutes effective implementation of the chosen priorities.
This is the heart of the evaluation process as it defines the indicators to be used to evaluate success in the chosen priority areas.
For example, if we focus on just one aspect of technology implementation - staff development -we may ask "What does truly effective teacher use of technology look like?" In this way, the team identifies its optimum desired outcome for use of the particular technology. One of the major roles of Sun Associates consultants, as facilitators in these discussions, is to insure that the team does not under or over estimate their desired outcomes. This is why the process requires a facilitator experienced in both the realities of the chosen district and the full range of possibilities for any particular technology.
Indicator development does not stop with defining indicators for optimum success. Rather, the team is asked to define indicators which describe "less than optimum" levels of achievement that when arranged in order would culminate in the optimum success. Arranged in a matrix, these indicators form a rubric which will be used to measure the school or district's success and progress in meeting desired outcomes.
3. Develop data collection procedures
Once indicator rubrics have been developed for all desired elements of priority areas, the school or district evaluation team develops lists of data and other supporting evidence that needs to be collected to measure the district's performance within the rubric. We facilitate the development of data collection process, and are sure to add in various data collection procedures that we are aware of and that the team might overlook.
It is important to note - and this is a point that always needs to be made to the project team - that data is not restricted to quantitative information and/or simple survey data. Rather, data can and should include things such as interviews, observations, and a variety of stories about the experiences involved with technology integration.
In addition to the above, Sun Associates can create a final written report that collects all of the indicators, rubrics, and data collection devices/objectives. These materials can be organized into an evaluation plan for the district and presented as such to the district's Technology Committee, School Committee, or other policy/oversight board.
Summary
In summary, Sun Associates works with schools and districts to create and implement educational technology evaluations which have the following features:
The evaluation methodology makes use of a number of indicators that measure degrees of success in all key technology program areas. In particular, we focus on helping our clients understand and evaluate the impact of technology on teaching and learning.
Under the guidance and facilitation of Sun Associates consultants, a representative team of school or district personnel work to develop their own unique indicators of success. This team is also trained to develop data collection priorities and requirements.
The indicator development process is one which is designed to inspire considerable reflection as to what constitutes "effective" technology integration. By requiring stakeholders to set their own criteria for success - as guided and monitored by a knowledgeable facilitator and other outside informational resources - this evaluation process has the effect of strengthening a school or district's understanding of technology integration and specifically how it impacts teaching and learning
.
Sun Associates' Qualifications
Sun Associates is an educational technology consulting firm specializing in issues related to the integration of technology in primary and secondary schools, districts, and classrooms. We take a broad approach to technology implementation which emphasizes the need to link the efforts of teachers, students, administrators, parents, and other community members in a common understanding of the uses and value of classroom technology. This approach is realized through services centered around:
Assistance to school and district strategic technology planning committees
Evaluation of school and district educational technology implementation
Assistance to educators on choosing and using educational software and hardware
Assistance to educators conceptualizing and creating on-line information, servers, and other electronic publications
Several examples of the firm's recent work are listed below:
Evaluation of Lexington (MA) Public Schools Year Two Technology Plan Implementation evaluation of the impact and effectiveness of the unique train-the-trainers approach that this district is taking for developing basic technology skills in all of its elementary and middle school teachers. The evaluation process involves developing indicators of success and rubrics which the district will use in its evaluation and on-going technology planning efforts.
Evaluating the Impact of Technology: A Workshop Series for Kentucky School Districts- an on-going series of two-day workshops for school districts in Kentucky to develop procedures and plans related to the evaluation of technology implementation efforts. Held in partnership with the Appalachia Educational Laboratory, these workshops provide hands-on experiences related to understanding the basic types of educational program evaluation; the links between strategic technology planning and the evaluation of technology implementation; and developing processes and procedures for evaluating district educational technology implementation.
An Evaluation of Educational Technology Implementation for Warren County (KY) Independent Schools - a six month study, with a 150-page final report and various presentations for a 15-school district in south-central Kentucky. This study examined the current use of educational technology across the district. The goal of this project was to assist the district in assessing the effectiveness of current technology use and the implementation of its district technology plan.
Virgin Islands Territorial Five Year Master Technology Plan - guidance and technical assistance to the territorial government in the creation of a master plan for technology implementation in schools and administrative offices. Specific services have included providing assistance in the overall design of the plan, facilitating planning committee and subcommittee meetings, editing and critiquing the overall plan, and conducting Territory-wide workshops on school technology planning.
West Windsor-Plainsboro (NJ) Public Schools Educational Technology Plan - guidance and technical assistance in the creation of West Windsor-Plainsboro's district-wide technology plan. Specific services include facilitating the development of a vision for educational technology; developing and administering a district technology needs assessment; developing a professional development plan; and overseeing the development of all key strategic plan components for submission to and approval by the New Jersey state department of education.
Timeline
The time a school or district takes to develop and implement an educational technology evaluation varies greatly. Significant factors that determine the amount of time required are:
The size of the school or district
The scope of the evaluation effort - i.e., how many evaluation questions/indicators are to be developed and explored?
The amount of time/effort a district is able to devote to technology evaluation and on-going planning
Most of Sun Associates' evaluation projects take approximately six months to complete. During this time, a school/district evaluation committee would meet monthly. The first several meetings are full or half-day events and are used to orient the evaluation committee to the process and to begin to develop indicators. Later meetings are generally short, after-school, meetings and are used to check progress and coordinate efforts. Data collection takes several weeks and is usually dispersed over several months.
Budget
Like the evaluation timeline, the budget for an evaluation project varies. Significant factors that determine budget are:
The size of the school or district
The amount of time/assistance Sun Associates provides in the data collection phase
The degree to which Sun Associates assists the evaluation team in writing a final report
.
