Program-Level Assessment - Section IX
Program-Level Assessment
Introduction
Department chairs are responsible for ensuring that assessment of student learning is taking place at the program level. All departments are expected to complete at least one assessment activity each year that provides insight into how well graduate students, majors, minors, and/or certificate-earners are achieving the learning outcomes that departmental faculty have developed for the program(s).
It is important to remember that program-level assessment is not about evaluating individual students or faculty (although assessment activities may lead faculty to rethink course content, assignments, or pedagogical approaches). Rather, program-level assessment is gathering data on the effectiveness of the current curriculum in helping students achieve the learning outcomes established by the department, and using the findings to continuously improve students' learning experiences. Because departmental faculty are the resident experts in their disciplines, they are the most qualified to make judgments about what learning outcomes to assess and the most meaningful ways to measure whether the desired level of learning has taken place.
Program Learning Outcomes
Every unit in CLAS has developed learning outcomes for its undergraduate (and, when appropriate, graduate) degree-earners. These outcomes are not set in stone; most departments will continue to add, remove, and revise outcomes as their assessment activities progress and their disciplines and program offerings grow and change. Some programs have as few as four broad learning outcomes, while others have more than twenty that are much more specific. However many learning outcomes your department has developed, it is important that they be measureable, and that your department has some kind of mid- to long-range plan to eventually measure each of them.
One of the most important functions of program-level learning outcomes is to help students understand and articulate the skills and knowledge they have gained through their experiences in your program. Make sure that your department is taking advantage of every opportunity to communicate its learning outcomes to students. This could include posting them in the department office and on the program's website, including them in orientation, recruitment, and advising materials, and listing relevant program-level outcomes on course syllabi.
Assessment Methods
Fortunately, you do not need to be an expert in statistical reasoning, social science, or education research to conduct effective program-level assessment. There are many relatively straight-forward methods that programs can use to assess student learning successfully, but most fall into one of two broad categories: indirect and direct.
Indirect Measures
Indirect measures of student learning are those that rely on self-report or other secondary sources of information to gauge what students have learned. They include methods such as:
- Surveys of students orfaculty
- Ficus groups with students or faculty
- Analysis of syllabi (to get a sense of the course content to which students have been exposed)
Indirect measures can provide important insight into students' experiences of the curriculum and the degree to which they or their instructors believe they have had opportunities to achieve a learning outcome. Students also often appreciate the opportunity to provide feedback on their experiences in a program and any gaps they feel are present in the curriculum. However, on their own, indirect measures are not considered sufficient for a long-range program-level assessment plan. Although they do not need to take place every year, all departments should make sure to include at least some direct measures of student learning as part of their long-range assessment planning.
Direct Measures
Direct measures of student learning are those that examine actual student output or work samples to determine how well students have mastered particular skills, concepts, or bodies of knowledge. Direct measures include analysis of the following kinds of materials:
- Student writing samples, such essays or reflective papers
- Non-written work, including presentations, performances, creative works, or digital media
- Compilations of student work, such as portfolios or e-portfolios (online collections of student work, often in the form of a student-designed website)
- Performance on quizzes, tests, or examinations
These measures are considered to be more objective, reliable indicators of student learning than student self-report or anecdotal faculty impressions.
Please note that, for both indirect and direct measures, your department does not need to collect data from every student in the program. Rather, you are collecting data on a sample of the students, either through survey and focus group responses or through a batch of student work that accurately represents their range of performance.
Assessing Student Work Samples
The first step in assessing student work samples is to identify a key course or assignment in which students will demonstrate the degree to which they have mastered the learning outcome your department is assessing. In many cases, this will be a course toward the end of the curriculum, such as a senior seminar, which is a logical place to measure students' cumulative learning in the program. In other instances, the department may be interested in assessing what students gain in a particular course that is intended to address one or more of the program's key learning outcomes (for example, a methods course). In some situations, if the department wants to know more about the foundation being laid for students at beginning of the program, it might target an introductory course for data collection. The most appropriate site for gathering student work samples depends on the learning outcomes being assessed or the pressing questions about student learning that the department wants to answer.
When assessing student work samples, be they papers, poster presentations, or portfolios, it is important to remember that this kind of assessment is different from grading. Rather than assigning a single letter or percentage to the work sample as a whole, the "readers" should be operating from a rubric that breaks down the criteria for successful achievement of whichever learning outcome(s) they are assessing (click here to see examples of effective rubrics). If, for example, your department is using a sample of student essays to assess understanding of cultural diversity, the readers should be scoring the work samples with a rubric that focuses on the pertinent diversity-related skills, concepts, and knowledge relevant to your discipline. Other factors that might be taken into account when grading such a paper, such as grammar and spelling or mastery of citation, should not affect the way readers score the paper for understanding of cultural diversity.
Before assessing student work samples, make sure to remove all names, both to preserve students' confidentiality and to help protect against reader bias (a particular concern in small departments where faculty readers may know many of the students whose work is included in the assessment).
Levels of Analysis
Faculty sometimes wonder why the grades they give students and the course evaluations students complete are not considered sufficient for program-level assessment. The reason for this is that grades and course evaluations are operating at different levels of analysis than program-level assessment. Course grades reflect individual students' performances within a course, and evaluations reflect students' self-reported experiences of a particular course (and the instructor teaching it). Program-level assessment is intended to measure the degree to which students are collectively achieving learning outcomes across many courses with several different instructors in a multi-year curriculum. While program-level assessment activities often focus on a particular course for data collection, this is generally done to determine whether that course is performing its intended function within the curriculum, not to evaluate that course's instructor.
Human Subjects
Collecting student data for internal assessment purposes does not require a human subjects review process or informed consent from the students involved. However, if there is any possibility that someone in the department might want to use the data in a publication (other than basic assessment reporting to the University), the department must submit a request for human subjects approval to the Office of Research and Sponsored Projects (ORSP). Most types of assessment research will qualify for waived review, the quickest and easiest approval granted by ORSP. Templates of human subjects requests and letters for informed consent that can be adapted to a range of assessment methods are available through the Center for Academic Excellence (click here). If you are planning to publish any of your assessment data, approval from the Compliance Specialist at ORSP must be obtained before data collection can begin.
Who Conducts Assessment
Different departments take different approaches when assigning responsibility for program-level assessment. In some small departments, the Chair conducts the assessment, consulting with other faculty as needed. In some cases, the faculty as a whole decide on an assessment activity, and the faculty teaching the relevant course(s) take responsibility for collecting the data. Many larger departments appoint an assessment committee, which is tasked with planning, conducting, and reporting the findings of program-level assessment back to the department. Finally, some departments have faculty who specialize in the scholarship of teaching and learning in their discipline, and these faculty take on program-level assessment with the intention of publishing on some aspect of this work. Rewarding faculty during the promotions and tenure process for participation in assessment activities and publication on teaching and learning will encourage more faculty in your department to participate and help shoulder the responsibility for program-level assessment.
Closing the Feedback Loop
The most important part of the program-level assessment cycle is "closing the feedback loop"-that is, using the findings of assessment activities to make specific changes to the program that will improve student learning. Although the department chair or assessment committee can make recommendations, the conversation about how to make use of assessment findings is one in which all faculty should participate. Some departments make it an annual practice to discuss their assessment findings at an end-of-year meeting or faculty retreat. Documenting this evidenced-based decision-making is part of the annual assessment reporting process.
The Annual Assessment Cycle
Portland State operates on an annual assessment cycle. Just like hiring and scheduling, budget, faculty review, and promotions and tenure, there are important deadlines on the assessment calendar that now take place every year, and that department chairs are responsible for meeting.
Planning Assessment
Most departments have a multi-year plan for assessing their programs' learning outcomes. This plan might be highly detailed, with each learning outcome linked to clear assessment measures and courses from which to collect data several years in advance. Other departments have looser plans, and make many of their decisions about outcomes, methods, and courses to target for data collection year to year, depending on their most pressing assessment concerns. Regardless of the level of detail in your department's long-term assessment plan, you should be developing specific ideas for the upcoming academic year's assessment activities by late summer. By the beginning of fall term, you should know:
1. What learning outcome(s) you will be assessing
2. What term(s) and course(s) you will be collecting assessment data from
3. What measure(s) you will be using to assess the learning outcome(s)
4. Which faculty will be responsible for collecting the data
5. Which faculty will be responsible for analyzing the data
6. Which faculty will be responsible for reporting the data
You will be asked to report your assessment plans for the academic year to the Dean's Office at the end of fall term.
Conducting Assessment
When your department conducts its assessment activities will depend on the course(s) from which you are collecting data: although most departments conduct their assessment activities during winter or spring term, some key courses in your curriculum may only be offered in the fall. Regardless of the timing, it is very important to make sure that the faculty teaching the course(s) and the faculty conducting the assessment are aware of their responsibilities, and are prepared to distribute surveys, conduct focus groups, or collect student work samples during the necessary window(s) of opportunity. It is okay if there is some lag time between when responses or work samples are collected and when they are analyzed; however, particularly with data collected during spring term, faculty should be cognizant of the amount of time required to process the data, and must complete this work in time to present results to the department and determine how to close the feedback loop before the University assessment reporting deadline in June.
Reporting Assessment
All units are expected to report on their assessment activities annually, with the reporting deadline in June. Program-level assessment reports are submitted through the online Digital Measures interface (link to Digital Measures coming soon). While your department is free to write a lengthier report on assessment activities for its own purposes, the Digital Measures reporting form is brief:
[Screenshot]
The key pieces of information required for program-level assessment reporting are:
- Which learning outcome(s) you assessed
- Whether these outcomes fall under one or more of the following Campus-Wide Learning Outcomes:
- Disciplinary and/or Professional Expertise
- Creative and Critical Thinking
- Communication
- Diversity
- Ethics and Social Responsibility
- Sustainability
- Internationalization
- Engagement
- The term(s) and course(s) from which data were collected
- The names of faculty directly involved in the assessment activity
- Methods and measures (i.e., what you did)
- Findings (i.e., what you learned)
- Closing the feedback loop (i.e., how you will use your findings to improve your program)
Feedback on Assessment
Every summer, the Dean's Office and the Associate Vice Provost for Teaching, Learning, and Assessment in the Center for Academic Excellence (CAE) work together to review each department's recent assessment activities and long-range assessment plans. This committee locates the developmental stage of each unit's program-level assessment to date on a three-point scale: Early-Stage Development, Mid-Stage Development, or Established. A department's placement on this scale is based on the following criteria:
- The establishment of clearly articulated, measurable program-level learning outcomes
- The degree to which program-level learning outcomes are being clearly communicated to students
- The quality and meaningfulness of the assessment activities that the department has conducted
- The extent to which assessment findings have been used to improve the program
- The proportion of the department's program-level learning outcomes that have been assessed so far
After determining the developmental stage of your department's assessment activities to date, the CAE will work with you to identify areas for improvement as you begin planning your program-level assessment for the upcoming year. With this ongoing feedback, CLAS hopes to see all its academic units reach "Established" assessment levels by 2015.
Where to Go for Help with Assessment
If you have questions about program-level assessment planning or methods, contact Leslie McBride, Associate Vice Provost of Teaching, Learning, and Assessment in the Center for Academic Excellence.
