CAREER CONNECTIONS TO TEACHING WITH TECHNOLOGY (CCTT)
The research reported herein was supported by the U.S. Department of Education, Grant Award #R303A970001. The opinions and findings are those of the evaluator and do not necessarily reflect those of the U. S. Department of Education.
EXTERNAL EVALUATION, YEAR 3
Karen C. Cohen
Karen C. Cohen and Associates
9 Cliff Road
Weston, MA 02493
1.0 Introduction, Background and Methodology
When the CCTT project was proposed and funded, there was no evaluation component. At the end of the first year of the Grant, the external evaluator was hired as a consultant for less than 1% of the budget. In Years 2 and 3 the consulting and evaluation role has increased, and external evaluation is now an ongoing part of the project. With the help of hub site managers, a good deal of baseline data and implementation data has been collected for the project. We use surveys developed for the project, OERI funded surveys, along with site visits to all 6 hub sites during the first half of the project, focus groups, project records, and full participation/observation in all semi-annual consortium meetings. As the Consortium has strengthened, a true collaboration has evolved. A shared consensus among all of the stakeholders as to evaluation plans and needs for full documentation of goal attainment, impacts, and outcomes, has developed. We will be using a broader variety of evaluation strategies as the CCTT Project’s infrastructure, development process, review process, and published materials and resources are deployed not only in the six original hub site schools and their counties and states, but also nationwide.
Two evaluation surveys were used during Year 3:
(1) A Teacher Survey (See Appendix A) which was used with about 80% of the participating teachers in the six hub site schools (N=50). This survey covered their involvement and reactions to the project in detail, and the results provide the project with a good deal of formative feedback as well as external documentation of outcomes and this mid-point in the project (March, 2000).
In addition, in Year 3, we also included:
(2) A Staff Survey (See Appendix B) was used with consortium staff. This survey was developed adapted from the RAND Sourcebook, funded by OERI. [It was used at the beginning of the evaluator’s involvement in the project with all teachers in all hub site schools, and we continue to use it in a variety of contexts for comparisons over time. We were requested, however, by US Department of Education to report only data collected prior to June 8th in this evaluation report.]
The remainder of this report is organized as follows:
- A focus group conducted with all project staff and consultants in January, 2000.
- Participant Observation in all Consortium Meetings.
- On-line review of units and lessons.
- Written reports from each hub-site manager.
- E-mail dialogue and telephone interviews with all hub-site managers.
- Personal and telephone interviews with content specialists and strategic partners.
- Student record data provided by hub sites.
2.0 Attaining Project Goals and Objectives
3.0 Recreating a Consortium: The Technological and Human Infrastructure
4.0 Impacts of the Project
5.0 Impacts on Students
6.0 Impacts on the Teaching-Learning (Classroom) Environment
7.0 Impacts on Teachers
8.0 Summary and Conclusions
Appendix A Teacher survey [Used March, 2000]
Appendix B TICG Participant Survey [Used after June 8, 2000—reported in Yr 4]
2.0 Attaining Project Goals and Objectives
The project has attained or is on the way to attaining all of the goals and objectives articulated in the originally proposed grant appropriate for this point in time and in addition has created some originally unanticipated contributions.
GOAL 1. STANDARDS AND CAREER CONNECTIONS: Identify and utilize national curriculum standards and career connections to core academic subjects:
Objective 1: Identify and incorporate into curriculum development the national reading, writing, mathematics, and science achievement standards, benchmarks and accountability measures for students.
GOAL 2: CURRICULUM DEVELOPMENT AND DISSEMINATION: Develop curriculum products and disseminate in an appropriate environment suitable for posting, editing, reviewing and sharing educational resources.
Results Year 3:
(a) Student academic gains: Students are using materials developed by teachers participating in the project. However, field testing of materials in classrooms other than that of the authors has only just begun. Teacher reports of student involvement and gains are appearing, however. (See Section 5.0 below). Evaluation Activity: The external Evaluation is establishing a data base including information from student school records to assess such gains in years 4 and 5 where sufficient implementation will have taken place and we can attribute possible gains to use of project materials. This will be done in collaboration with each participating school district/hub site, but common factors across the Consortium will be explored. We will also re-use instruments developed and used by students and teachers in the hub site schools and their districts involved as CCTT materials are implemented in classroom settings in Years 4 and 5 of the Project.
Evaluation findings for year 3 are supported by interviews with content specialists, surveys with teachers, focus groups with all site managers, review of units, lessons and courses produced in 5 of 6 sites, and participant observation with the entire staff at all full staff project meetings.
(b) Incorporation of National Standards CCTT has made strong progress incorporating national standards. It has done so both by utilizing content specialists who, in some instances, created and refined the national standards in their own fields. [It should be noted here that some of these sets of standards are, themselves, being rethought and refined.] The project also has started to use content and standards curriculum experts in areas covered by the project who are state-qualified content specialists and are also participating in the on-line curriculum development and review process CCTT has created. This multi-tiered review and collaborative development process is geared toward developing peer and specialist reviewed quality materials. The original proposal suggested using national content specialists who were on the committees that created the standards in each field to review developed units and lessons. This has not materialized successfully, but involving qualified external content specialists who are qualified both in the standards and instruction in each field has replaced this original expectation as a more viable option. Hub site managers and TIES consultants have been active in recruiting these people to add their expertise to the Consortium, and this involvement is strengthening the process.
Objective 2. Create a career connection to integrated core academic studies that increases the relevance and authenticity of learning.
The project has established an advisory panel and plans for this goal attainment. One hub site reports using career planning activities as part of assessing impact on students. [Collecting evaluation data for this goal is appropriate for Years 4 and 5 of the project. Interviews, participant involvement, and surveys will be used.]
Objective 3. Train teachers and students to access and utilize existing technologies to create original instructional materials, and to collaborate with business partners in co-development of resources. In Year 3 of the project, CCTT has continued to use and develop on-line professional development strategies as well as local workshops in each of the six hub site schools and geographic areas involved in the CCTT Consortium. The project’s teachers, hub site managers, reviewers, and consultants have produced almost 100 units and over 300 lessons. These are in varying stages of consortium and content specialist review. In addition, complete courses have been developed at some sites, and rich web-sites in Physics and Chemistry have been developed by individual teachers and are also available to the consortium. Although implementation has varied at each of the six sites, communication and collaboration across sites has been maintained, utilizing, strengthening, and leveraging the Consortium’s resources. Teachers find peer-reviews very beneficial (See Project and Teacher Impact sections below).
3.0 Recreating a Consortium: the Technological and Human infrastructure
Evaluation findings here for Year 3 are based on:
(1) lengthy and detailed paper-and-pencil surveys returned by nearly all of the teachers at the participating hub site schools who are involved with the project;
In addition, the project has fostered and supported material and resource development beyond the originally envisioned curriculum units and lessons. These resources include a website replete with resources for full teaching and tracking students’ involvement and achievement for courses in several levels of physics, as well as courses in Graphics Design, CAD, and Criminalistics/Forensics. Evaluation of these additional resources has been formative to this effort to date, but results obtained in collaboration with the teacher-developers indicate strong impacts of these technology-rich resources on student achievement, recruitment, and attendance, among other factors being tracked by the project and the evaluation for years 4 and 5.
(2) participant involvement in the planning of workshops for Years 4 and 5 of the project;
(3) reports of hub site managers;
(4) on-line review of developing and developed materials;
(5) telephone interviews with participants and consultants;
(6) focus groups at selected locations and staff meetings;
(7) Corroborative documentation reports are in process as internal project efforts by consultants from TIES who are heavily involved in the development of the project curriculum development process.
Objective 4. Contribute to national educational networks using telecommunications to disseminate products and best practices. The project has made major headway in this area. It has developed and implemented authoring systems and templates for its own use. By becoming the CGLi node (Challenge Grant Learning interchange) housed and supported at no cost to the project and to end-users by Apple Computer Corporation, the project may be resolving issues of authoring systems in state of flux along with incompatible equipment configurations both within sites and across the consortium. Via the CGLi network, teachers can and will develop, publish, and disseminate CCTT materials. CCTT can also disseminate other materials, although it has to resolve a mechanism for doing it. This resource is designed to serve as a national and international locus for other individuals and projects developing similar and or related materials to be approved, publicized, and made available.
Year 3 evaluation of this activity includes ongoing review of systems used and materials developed; feedback from technical specialists associated with and supported by the project as hub site managers or full-time partners--depending on the site; and participant observation by 3 different external participant observers at staff conducted training workshops.
It is important to note that the people who envisioned and obtained funding for the initial CCTT grant proposal, i.e. the original stakeholders, are in not the majority of the people who are directing the project. Unfortunately, the original project director died a few days before the project was to begin, and Marshall Ransom became director of the project and the Consortium on three days’ notice. It is to his credit and the continuing and replacement hub site managers, that the consortium has faced many challenges as a group, handled its original partnerships appropriately, and is fulfilling original project expectations. The original hub sites are all involved and are productive. All hub site managers, technical counterparts, project consultants, and project administrators participated in a focus group on January, 20, 2000 to reflect on their experiences. We present their responses:
3.1 What were the biggest Challenges in the first half of the project?
4.0 Impacts of the Project
- Changing vision and formats/ and presenting those to faculty
- Reforming a common vision
- Setting up contracts and payment procedures
- Overcoming distances and diversities
- Dealing with problems arising from some of the original partnerships
- County Dynamics/District Networks
- Establishing and maintaining deadlines
- Shift from delivering content to becoming curriculum specialists
- Becoming curriculum creators not locators/greater role in instructional design
- Loss of the original grant writer - What was George thinking? New project director
- Establishing a dynamic between technology and pedagogy
3.2 What were the Most Outstanding Successes of the Project to date?
3.3. Anticipated project outcomes (Year 5).
- The group coming together to collaborate and share the decision making process
- Creators of content / Finding direction out of chaos
- Using technology totally to further project’s goals
- Introducing teachers to new technologies and seeing them use them
- Teachers changing from accessing curriculum to creating it
- Gaining access to resources and personnel from our geographically diverse sites
- Dissemination/ presentations/ promoting the grant and DOE
- Moving from a static product to a dynamic support process
- Development of the process for creating standards-based, quality materials
In brief, the CCTT Consortium, as a group, feels that the impact on teachers of involvement with the project and its processes along with its use of its technology-rich materials will fulfill the vision of the original proposal, and that this vision is now structured clearly.
- Product validated by an outside group of teachers and students
- A “boat load” of quality product
- A model product available to more teachers for developing standards-based, technology-infused teaching and learning
- Empowerment of teachers to be innovators and agents of change
- Evidence that technology, money and time are critical for creating change in schools
- User-friendly system for development and distribution of content
- A plan for sustainability
3.4 Infrastructure change(s)
The change from DCI to Apple (from an intranet to an Internet-based authoring and potential publishing facility) has been successfully implemented, hopefully to the future benefit of the project.
3.5 The RAVEN simulation algebra tutor
The failure of UCF’s Institute for Simulation and Training (IST) to develop a functioning simulation (RAVEN) as an algebra tutorial is being handled on two levels. IST has offered to provide simulation and training resources for on-line professional development and most logical use in physics courses at little or no additional cost to the project. Materials in algebra are being considered that have been developed by others if support is available, or that will be developed by new teachers joining the Consortium as it continues its diffusion, dissemination, and writing workshops. Two participants in the first dissemination workshop have offered to recruit teachers and teachers-in-training at the University of Florida (Gainesville) in mathematics and a second at St. Petersburg High School, also in Florida.
Plans for sustaining and furthering the project and consortium financially, i.e. support and availability for its units, materials, and other resources (including on-line professional development opportunities) are still not concrete. Individual hub sites have leveraged their involvement with CCTT, however, and have received related or co-related funding from public and private groups to augment and further the overall effort to use technology and develop innovative materials and resources to improve teaching and learning for themselves and others.
4.1 The sample 50 teachers responded to a detailed survey about their participation in the project, their units and lessons, project support for what they were doing, and support staff help (See appendix A). These surveys were administered to teachers only in the hub-site schools participating in the project. (Far larger numbers of teachers are involved in developing materials for the project, but we were piloting this survey in an open-ended form and are developing surveys we can easily score in future years based on this year’s input.) The surveys were administered and returned anonymously.
5.0 Impacts on Students (evidence)
Distribution of Respondents
|Advanced Technologies Academy, Nevada
|Ft. Leavenworth Unified School District, Kansas
|Mainland High School, Florida
|Manual Arts High School, California
|Omaha North High School, Nebraska
|Sprayberry High School, Georgia
Despite their geographical diversity, the teachers participating in the project were remarkably similar:
The teachers at Fort Leavenworth taught K-8; all of the other teachers taught at the high school level. (A small number of teachers taught more than one subject.)
- They averaged 11 years teaching experience;
- Half (or 25) have BA or BS degrees and half had M.A. or M. Ed. degrees; Most had professional certification far beyond their degree levels;.
- The year in which they received their highest degree was 1990 (with the range far greater); and
- they had been with the project and average of 1.8 years.
The subjects taught were
9 -- Social Studies
These teachers reporting had developed 125 Units and 370 Lessons for the Project by March, 2000. In addition, some teachers (not included in this sample) had developed complete courses (including lessons and units).
8 -- English/Language Arts
6 -- Mathematics
6 -- Chemistry
3 -- Physics
3 -- Science
3 -- Computer Science
3 -- Reading
2 -- Applied Internet Basics
2 -- Biology
2 -- Spanish
1 -- Drama
1 -- Physical Education
1 -- Integrated Science
1 -- Earth Science
1 -- Library Science
1 -- Life Science
1 -- Home Economics
4.2 Challenges in developing Units and Lessons
The most prominent challenges for teachers involved in the project were:
Less frequently mentioned but other frequently mentioned challenges were “Using the template”, “Using CGLi”, etc.
- Narrowing their concepts into units and lessons; refining a topic
- Finding the time to get everything done during the school year, and
- Determining or coordinating with the standards (National and State)
As one teacher effectively stated: “Getting it all together. Learning to create in lessons that followed several objectives/goals such as national/state standards and be creative and fun.”
Other less frequently mentioned problems involved overcoming fear of technology, having technical problems, not getting timely review help or not getting reviews at all, and not having a chance to pre-test what they had done before sharing it with others to use.
4.3 Challenges in Implementing Units and Lessons
For those teachers who had implemented there were three major problems:
4.4 Most Successful Outcomes from Units and Lessons
- Time—time to teach and time in class
- Technology—availability and reliability, and
- Teaching students Technology skills in addition to materials
Teachers reported successful outcomes for their students and for themselves. For their students, they reported:
- a huge increase in student motivation and interest
- successfully using a new medium for students to use to learn (technology)
- students expanding their learning and research with access to the Internet
- greater variety and depth of learning
- collaboration and team-work with small groups of students
- better organization for students—objectives they could understand, meet and master
- self-pacing options for slow and fast (all) learners
- one teacher: “…could see girls figuring it out.”
5.1 Concepts and Understanding
6.0 Impacts on the Teaching-Learning Environment
In all sites, some or all of the teachers reported evidence of mastery by student scores on the assessment activities built into the units (with 85-100% mastery) as the strongest indicator along with project reports and products their units involved. In some instances these assessment or mastery items were in units used by teachers in the consortium that were developed by other teachers in the consortium. Several teachers in one site reported student ability to relate what they were learning to career paths was clear evidence of concept mastery and project success. Other teachers reported longer attention span, retention over the year, and impact of visual presentations for “visual and kinesthetic learners”.
5.2 Student Attitudes
Over half of the teachers responding said that their students were excited, engaged, and very much enjoyed the CCTT developed units and lessons. They “loved it” when they saw the projector set up for a power-point presentation. Some teachers felt that students liked the better organization of the lessons—they knew what was coming, what to expect, and were willing to learn. Two teachers reported students using these resources at a distance to keep up or make up for absences from school.
5.3 Student skills
Student increase in technological skills was the overwhelming result of using and implementing CCTT materials. Collaborative or group work was also prominent across all sites.
5.4 Behavior in class
Teachers reported that students were almost all engaged, focussed, motivated, interested, on task, and more dependable—fewer discipline problems in general. Some teachers reported frustration that each student did not have his or her own computer, and some frustration with time “off-task” when primary resources (e.g. the Internet) were not available to them.
5.6 Reactions of others
Teachers reported strong parent enthusiasm and excitement (at open houses and parent conferences when this was discussed). One parent reported that “I could not get my child to go to the movies, he was so busy working with the computer at home!” In two sites other teachers loved the units and were also using them as well as the new equipment. Teachers also reported varying interest in their schools, but noted peer interest and support, principal support, district support, and invitations to make national presentations.
6.1 Classroom Activities
7.0 Impacts on Teachers
Most teachers reported (as would be expected), enormous use of computers. They also reported group work and interaction between and among students far more than in their traditional classes. They reported use of CAI tests and tutorials, hands-on activities, all students involved, smoother/more planned classes, and in some instances students leading lessons rather than the teacher. A small minority of teachers reported no change, but even some of them were already using computers as a regular part of their instruction.
6.2 Instructional Materials Used
Beyond universal use of computers and heavy use of the Internet, teachers reported using projectors, powerpoint, texts, and television along with a variety of laboratory materials and utilities, other software utilities, and calculators.
6.3 Your (Teacher’s) role
Teachers overwhelmingly reported being facilitators, guides, mentors, and problem solvers. They “set the stage” and “answered students’ questions” when implementing their units.
6.4 Student Engagement and Participation
Teachers reported on professional development activities, project staff, project resources, and personal impacts.
8.0 Summary and Conclusions
7.1 Professional Development Activities
Professional development activities varied quite a bit by site, with each hub site manager organizing and presenting sessions, workshops, or on-line courses for participating teachers. Sessions with speakers were well-received, although the speakers were in some instances presenting too much information at one time. Nearly half of the teachers had NOT attended or participated in professional development activities at the time of responding to the survey.
7.2 Content Specialist Help
Teachers generally felt that the help they received from their hub site managers or peer reviewers was very helpful. Most teachers had not had content/standards specialist help, and a few who had did not find it helpful. One felt it was not required, and there were complaints about the lag time (e.g. waiting from August, 1999 to February, 2000 for reviews).
7.3 The Template
Most teachers found the template easy to use, a good organizer, and as one teacher said “Just seeing it helps me get over the first stumbling block.” Teachers complained that changing the template three times in three years was too much, but those newer to the project found it helpful. A few found technical problems, e.g., saving information, uploading files, and not allowing formatted worksheets to be uploaded, that have to be solved. [The new version of the CGLI website allows uploading of attachments.] A minority felt that it was not good.
7.4 Curriculum Development Process
Teachers felt that time and getting started were the most difficult problems, with finding the time to do the work being the most difficult problem. Some felt that the process was clear, but two felt that they were “just flying solo”. Others felt that it was teacher initiated, good, and one felt that the CCTT review process doesn’t exist.
7.5 Project Staff
Project staff (in this project Hub site Managers) were uniformly helpful and supportive.
7.6 Use of Units and Lessons in the future
95% said they were using their CCTT materials now and would continue doing so. Further, some were using materials other teachers in the project had developed and were planning to use more in the future. The collegiality developed among this virtual community is strengthening CCTT and should expand as the project continues.
7.7 Personal Impact
Beyond finding CCTT as a community where teachers can contribute, receive feedback, develop and share materials (see above), teachers generally reported (in decreasing frequency):
To quote three teachers:
- major person gains in technological literacy and skills
- general growth and development as teachers
- changes in how they deliver instruction
- changes in how they think about their teaching
- learning to manage their time
- overcoming fear of technology
- “I actually can’t believe how excited I am about so much extra work!”
- “I can generate electronic classroom presentations over a weekend that rival what only a Hollywood producer could have done two decades ago.” and
- “It caused me to think through the planning process differently when I know others would be using my unit.”
Although not without problems, the CCTT Project and Consortium is a cohesive consortium. The current stakeholders are fully invested and are guiding and directing the project through initial problems of development of a technological infrastructure for authoring and disseminating materials on-line; its units and lessons are being implemented in all hub sites and regions geographically covered by these hub sites and are also being shared across the sites. The peer review process is successful. The problems with standards and content specialist reviews are being addressed. Student motivation, engagement, is strong. There is reported evidence of student achievement,, interest, enthusiasm, collaboration, changes in teacher role (facilitator), and classroom environment (collaborative and group work). There are some reports of gains in test scores, increased attendance, and increased enrollment to be followed systematically as the project matures and is fully developed and deployed. In addition to Units and Lessons (the core of the originally envisioned curriculum development materials process), complete courses and technology rich web sites have been developed by individual teachers with project support. Years 4 and 5 will test more fully the viability of this model, the impact of its materials, and its contribution to improving education K-12 nationally.
Copyright © 1997-2003
Career Connection to Teaching with Technology
USDOE Technology Innovation Challenge Grant
Marshall Ransom, Project Manager
All rights reserved.