Measuring Course Quality: Exploring Basic Issues

Student course evaluations involve assumptions about what makes good teaching and what performance standards in teaching can be expected.

Faculty members must be willing to be evaluated by students and must have a basic trust in the reliability of the procedure and its results. Without this cooperation, the goal of student course evaluation—optimizing instruction—remains out of reach. If instructors don’t have confidence in the accuracy of the results, they can hardly be expected to reflect on their teaching, or to consider making changes.

The choice of survey instruments—along with the organizational, trust-building measures to embed evaluations into a Quality Management process—plays a decisive role. Our clients using Class Climate have deployed a wide variety of questionnaires; there are effectively no limits to the surveys that can be developed and administered.

Legal Framework, Regulations, Data Security

Before beginning your first survey, decide who will carry what responsibility. You cannot underestimate the importance of trust in the procedure. Discuss the evaluation procedure’s goals in a broad-based forum:

• Which courses should be evaluated, and when?
• Who takes on the position of trust as evaluation coordinator/administrator?
• What access rights will deans and deans of studies have?
• What kinds of report should be generated?
• How should instructors and departments be involved in creating their own questions?
• What equipment is to be used?
• Who receives which reports?
• How will students receive feedback?
• What kind of security should students have?
  • Should you use individual passwords for response tracking, or should you forego passwords and risk “stuffing the ballot box”?
  • Should handwritten comments be transcribed from paper surveys to avoid having an instructor recognize a student’s handwriting?
    • How high should the transcription threshold be set so that large—and inherently “anonymous” courses—are excluded from this expensive procedure?

Most universities develop and adopt specific policy regulations regarding these issues. You can then configure a system like Scantron’s Class Climate according to these specifications.

Developing and Managing Flexible Survey Instruments

Your evaluation system should free you to design single- or multi-page questionnaires containing numerous questions (items), which you can summarize individually or in the form of indicators. Scantron’s solution, Class Climate, supports the following types of questions:

• Scaled questions
• Open questions (free response format) with variable size response fields
• Single choice questions
• Multiple choice questions
• Matrix fields (for numerical data such as age, year or zip code)
• Differently scaled response formats such as grade values from A to F a rating scale (e.g. 1–5 where 1 is Excellent and 5 is Needs Improvement)

A strong evaluation system should include a question library and questionnaire templates so you can store and re-use your work. You should be able to save questions and organize them to suit your needs, grouping them by department, subject matter, or whatever hierarchy helps you find questions faster and create questionnaires more easily.

The item library integrated in Class Climate enables you to organize related questions in up to 10 layers of categories.

In addition, you can create templates with pre-defined questions.  Once you have a suite of templates (say, for a particular subject such as history, or type of class like a lab), you can easily customize the template each time you use it to include new or additional questions. Using templates to start the process dramatically speeds up your ability to create evaluations that are targeted and specific, giving you more accurate —and useful—responses.

It should be easy to create questionnaires in your evaluation system. Looks for a solution with a graphic layout, so you can see what your questionnaire looks like as you’re working on it and make changes faster. Class Climate provides two easy-to-use editing options:

• A simple, web-based editor you can use to design basic questionnaires.
• (As a system add-on) Visually create complex forms using VividForms Designer’s drag-and-drop design tool. Includes predefined templates for multiple columns, free-form layout for both portrait and landscape orientation, graphical items (such as lines and images), and other advanced design options.

Both editors support robust design functions and automatically adjust questionnaires for mobile, online, or paper distribution. Drag-and-drop questions from the question library or create questions on the fly using a simple question wizard for a clean, professional survey look. Once you’ve completed the wizard, you can always shift questions, exchange question groups, or correct typographical errors by double-clicking the question item. Simply and easily define font type and line spacing, among other design elements.

In addition to supporting a graphic layout process, you should be able to group questions by broader indicators. As well, your survey system should include a variety of additional information so you can analyze results and quickly identify appropriate changes to make to instruction. For example, Class Climate provides rules for creating norm reference values or quality standard objectives as well as details governing access rights to a questionnaire (domain, period, etc.).

See And the Survey Says…? Understanding Questionnaires in Class Climate for more details.

Defining the Organizational Structure

Each institution organizes departments, courses, and classes differently. An effective evaluation tool supports your organization structure, rather than forcing you to adapt and adjust a pre-define hierarchy. In addition, the most useful solutions enable you to develop and process questionnaires in batches to match your departmental needs.

The central evaluation approach in Class Climate provides a variety of functions for processing questionnaires in automated batches. For these functions to be effective and provide an overview of the different types of questionnaires to support the hundreds of courses you’re likely evaluating, you can define an organizational structure that matches your needs. You can:

• Set up subunits, such as departments in a college or colleges in a university
• Assign addresses, logos, and corporate design guidelines
• Specify the group of individuals who are responsible for implementing the evaluation

By providing a variety of user roles, Class Climate can accommodate the conditions of your particular organizational structure. Depending on the extent to which you follow a centralized evaluation model, you can authorize specific data access rights based on various roles.

Defining Evaluation Cycles and Interaction Times

Typically, your institution will deliver instruction using a semester- or quarter-based system. Some colleges and universities structure monthly classes to support returning education students. A strong solution supports your schedule so you can evaluate classes routinely without a lot of extra work.

Regardless of the term structure you use, you only need to set up the schedule once in Class Climate. If you have a blend of schedules (for example, your main campus follows a quarter system, while your Extension campus follows a monthly schedule or has classes of varying length), you can define individual periods for special programs or other unique circumstances.

While you prepare the organizational structure, also decide when you would like to evaluate during the cycle. Some institutions conduct course surveys towards the end of the term, so that instructors have the results in time to discuss them with their students. This creates a useful balance between including only students who have attended enough classes to be able to adequately assess a course and ensuring there is enough time remaining in the cycle for lecturers to reflect on the results and make changes if necessary.

You may also want to give instructors time at the beginning of the term to add their own questions to the survey (specifically relevant to their subjects or specific courses) so that these can be included when you produce the questionnaires.

Course Data and Instructor Addresses

Because of the high degree of automation provided, more and more universities are beginning to comprehensively evaluate all of their courses with a unified system like Class Climate. Metadata on the courses and their instructors (such as course names and numbers, faculty names and E-Mail addresses) provide electronic course catalogs.

Class Climate offers two procedures in order to enter the courses to be evaluated into the system: manual entry or import from CSV/Excel or XML data. Our Professional Services team can work with you to define an automated process. See the Product Training, Implementation, and Professional Services brochure for a complete list of the services we can provide.