Section 3: Evaluating, Implementing and Managing Instructional Programs and Projects

Section 3: Evaluating, Implementing and Managing Instructional Programs and Projects 

All too often instruction is developed with little thought as to how evaluation of learning or the effectiveness of the instruction will take place. When evaluation is considered on the front end of the instructional design process, it is often limited to evaluating whether the instructional design is more effective than traditional methods.

For this week's reflection activities, I would like for you to concentrate on the following:

Chapter 10 discusses evaluation in instructional design and provides you with two evaluation models, the CIPP and Kirkpatrick models for evaluation. Search for at least two other models used for evaluation and summarize these models. Describe how you would use them to evaluate your instruction.

Reflect on what other questions that instructional design evaluation should address besides whether the instructional design leads to comparable amounts of learning and learner satisfaction as traditional methods. What else would be useful to know?

Rossi’s Five Domain Evaluation Model allows the user to tailor the “evaluation to fit local needs, resources, and type of program,” Trends and Issues in Instructional Design and Technology (p. 98).  This model focusses “on one or more of the following domains: needs, theory, process/implementation, impact, and efficiency,” Trends and Issues in Instructional Design and Technology (p. 103).  From a teacher’s perspective, this method will allow the teacher to modify the evaluation to fit the needs of the class and the objectives.  Binkerhoff’s Success Case Method instructs the evaluator to compare the success/failure of similar programs to your program “so that success can become more widespread in the organization,” (103).  By combining Rossi’s and Binkerhoff’s models, an evaluator can focus the evaluation on particular needs while comparing programs to ensure success. 

I am currently working on my masters to become a certified school librarian.  One goal of librarians is to help with the implementation of technology in the school.  Rossi’s Five Domain Evaluation Model can be used to evaluate the effectiveness of iPads in the library while Binkerhoff’s Success Case Method can be used to compare similar programs.  As the librarian and evaluator, I will need to determine if there is a need for iPads in the library and what will be the purpose of the iPads.  Both models address these issues in step one.  I will work with teachers and administration to determine how the iPads will be used in the library and in the classrooms. 

According to Rossi’s model, I will need to conduct a program theory assessment to determine if “the program is conceptualized in a way that it should work,” (p. 98).  I can do this using Binkerhoff’s method of comparing programs.  I plan to locate and contact other libraries currently implementing an iPad program.  I will also research peer review journal articles for documented procedures and outcomes.  Once a plan is devised, a visual impact model can be created using Binkerhoff’s Success Case Method.  The model will be constructed using the comparison of other iPad programs.  The model will include learning objectives for students and teachers in column one, including: using the iPads for research, e-books, and apps.  The middle column will explain how the students and teachers are expected to access the iPads.  The iPads will be used in the library and can be checked out for classroom use.  The initial iPad instruction will be by the librarian either in the library or in the classroom.   The last column will illustrate the goals of the iPad program: students can independently access the iPads to contribute to learning in all subject areas.  The visual impact model is a plan that all participants can see and follow.  

After implementing the program, evaluation is still necessary.  We will need to conduct an impact assessment using Rossi’s model.  Was the iPad program instructionally effective?  Has the student’s intrinsic motivation improved?  To assess the impact, the librarian can compare learning outcomes from the previous year to this year on similar projects with and without the iPads. 

In a school district with a limited budget, Rossi’s last domain, efficiency assessment, is crucial.  If a goal of the iPad program is to spread to all the schools, efficiency must be proved.  This task is difficult when the ROI is not in dollar amounts but is determined by student achievement.  A few possible cost benefits could be: the use of electronic databases instead of paper encyclopedias, sharing a set of iPads instead of buying iPads for all classrooms, and the mobility of the iPads compared to students computers. 

Binkerhoff’s Success Case Method includes a write-up to communicate the evaluation outcomes with the teachers and administration.  This will explain the details of the program, how it was successful, and how the success can be continued long term. 

            The instructional design evaluation models discussed in this chapter include both formative and summative evaluations to assess planning decisions, processes, and results.  Binkerhoff’s and Rossi’s evaluations did not address behavior like Kilpatrick’s Training Evaluation Model.  It is important to determine if the learner will be able to transfer the knowledge learned to an on the job application.  For my iPad program, it will be important to determine if the students will be able to use the iPads in subject areas other than what was specifically taught.  As the students move on to future grades, will iPad use continue in future classes?   

Chapter's 12 & 13 focus on project management and how to manage projects when resources are scarce. You have been assigned to develop a series of professional development sessions focusing on technology use in the classroom for teachers during a time of economic decline. How will you use Situational Leadership to facilitate this project and manage scarce resources? 

            The school district I most recently worked for experienced all stages of the economic cycle.  I was hired during economic growth and then experienced the peak, decline, and trough.  The cycles were due to an extreme increase in population that leveled off over the years.  During the time of growth, two elementary schools and a high school were opened which caused an increase in staff.  An influx of technology also occurred to support the expectations of the growing community.  When the district was entering the trough, resources were becoming limited in all aspects including, people, time, and money however, learning and innovation must continue.

            One positive element of the situation was that an enormous amount of technology was purchased during the growth and peak so every classroom has a Promethean Board, Elmo, students in 5^th grade and up have personal laptops, and every school is equipped with a computer lab.  Since technology is already in place, money will only be an issue concerning salaries. 

            The team will consist of one teacher from each grade level, curriculum facilitators, and a technology facilitator.  Since people and time are limited the team will meet in person initially to discuss the details and expectations of the staff development but further communication will be through email.  Communication is a major component of situational leadership.  The communication during the first meeting will be very direct and the leader will take a supervisory role to ensure the team understands the plan to reach the desired goal and what each person’s role is in the planning process.   

            Conducting discussions through email will save everyone time but the leader must be very clear when conversing through email without any hints of sarcasm.  It is impossible for a group of teachers to meet during the day and the district cannot pay the teachers for their time spent after school working on the project so individuals will be expected to work on their own time.  Phase two of situational leadership becomes very important now.  The leader must ensure the team “buys in” to the project.  To boost morale, the leader will meet with each team member individually when it is best for the member.  During this time, the leader will listen openly to questions, concerns, and ideas while providing clear feedback and encouragement.  After meeting with team members, the leader will compile the information and email the entire team to keep everyone updated.  This process will continue until completion of the project. 

            Phase three requires the leader to focus on results and ensure the team is rewarded for effort.  A reward for effort can be difficult when money is limited.  A reward for a teacher could be extra planning time during the day while an aid covers her class or recognition in front of her peers for her effort in creating the project.  Since the facilitators are not responsible for a class of students during the day, they can be rewarded with an extended lunch break. 

            The team will meet face to face at the end of the project to clarify any concerns and tie up loose ends.  During this meeting, the leader will not have a supervisory role but will monitor the discussion instead.  The leader will step back and allow the team to work.