Navigating the Evaluation Landscape

Many players participate in course evaluations. Each of these players has a particular focus and set of needs that an evaluation system must meet to be successful.

Through close cooperation with a number of universities and researchers in higher education, we have identified some of the requirements inherent to an automated system for evaluating instruction.

Executive Officers

An institution’s executive leadership wants to provide:

• Continuous quality improvement
• Support for “learning from the best”
• Support for accreditation measures
• Open procedures
• An ability to compare faculty results and make organizational development more effective

These leaders want to minimize additional work for faculty members who support a service-oriented concept for implementing the evaluation. They would like an automated process that enables surveys to be carried out on a regular basis so they can find trends and tendencies, strengths and weaknesses between instructors and across quarters, semesters, or even years. They then want to use this information to determine necessary action. And last but not least, administrators want a modern quality management system, giving them an edge over their competitors.

Deans, Deans of Studies, or Department Managers

Department heads would naturally like to have an overview of the data collected for their department (such as comparative reference data for course subject and course type). They would like to retain subject-matter relevance, even when surveys are conducted across departments. In addition, they would like survey results that allow them to see what action(s) may need to be taken to improve their courses.

Instructors

Faculty members are interested in improving their teaching—they want prompt and specific feedback immediately after a course is over. However, they are also concerned that the survey instrument is fair and that it provides data security. Many instructors would like to be able include their own questions in the survey and would like to be able to make anonymous comparisons to their colleagues.

Students

Students want to be taken seriously and to see that they are part of the change process. They are concerned that their honest and, at times, critical opinions might be used against them, so anonymity and security of responses are crucial. Like instructors, they, too, would like to see what has been done with their responses in a survey. And they would like to be spared unnecessary questions or surveys—otherwise they will stop providing course feedback.

Computer Centers/Institutional Research Groups

Computer centers and institutional research groups generally have little spare capacity and would like a procedure that makes as few demands on their resources as possible. In this case, a browser-based solution available institution-wide as a web service is the best solution. The head of the computer center is happy when there is as little administrative paperwork (e.g., setting up user accounts) as possible. They want to set up data security and data access security once, and then have them run automatically. Institutional research groups are happy because they data they need to analyze and recommend improvements is available in a single, easily-accessed location.

Administration

The administration needs a system that conforms to data security regulations as well as university evaluation policy and regulations. They may want some of their services—over and above teaching—to be evaluated as well.

From this discussion, we can see that a functional and effective evaluation system must include a variety of questionnaires and support a variety of needs. Further, we can see that comprehensive and continual evaluation requires a set of data to model the organization, including all of its departments, courses, and instructors. This structure forms an important foundation for creating data reference norms and Quality Measurement processes, as well as allowing for convenient re-use of previously developed materials. To guarantee as much flexibility as possible, consider making mobile, online, and paper surveys available for deployment. The responsibility for implementing surveys, their costs, and the access rights to the resulting data must be finely layered and adaptable to the needs and contexts in your organization.