Year Two - Learning Outcomes
There are two themes for the second year or phase of design and implementation. The first is developing learning outcomes, which I define as specific statements of what students will be able to know and do as a result of engagement in a narrowly defined programmic interventions (e.g. a workshop, a counseling session, an internship, etc.). The second is matching appropriate assessment tools to your learning outcomes.
As you plan your time, I suggest the following:
- Plan 2-4 hours at a staff retreat to teach your staff how to write learning outcomes and how to begin working with assessment tools;
- Each participating staff member should set aside 1 hour a month for preparation time - writing student learning outcomes and planning assessments;
- Also plan 1 hour a month of staff meeting time to review progress;
- Plan 2-4 hours at the end of the year/phase to work with an external evaluator and to develop your year-end report.
Page Contents
Developing Learning Outcomes
Select Leveraged Programs
Begin by selecting leveraged programs or learning interventions to work with during the year. By leveraged, I mean programs that:
- Are highly visible;
- Are run by capable leaders who buy into the assessment process; and
- Can most easily be assessed.
Do not choose too many - no more than one a month per staff member. Though you are now in your second year of the process, you still want to build positive momentum with early wins.
The Basic Learning Outcome Formula
As with most elements of the assessment process, there are numerous approaches to the process of writing learning outcomes. At its most basic, a learning outcome can simply consist of the following:
SWiBAT + Behavioral Activity
SWiBAT is an acronym for "Student will be able to" - it is a nice way to get the outcome started in a manner that leads naturally to cognitive or behavior activities. Some practioners will cut SWiBAT to SWi, or "Students will..." They complete the phrase with the activities that students will complete during the exercise. While this works well for some, I prefer the full SWiBAT. For example, when the DIRECTV guy set up my TV, he showed me how to program the TiVo functions and even had me do a couple for myself. While this guy left my house being able to say that the "Customer will...program his TiVo box," he could not exactly say that the "Customer will be able to...program his TiVo box." An hour later, I was on the phone to DIRECTV with a question. While doing is a stronger learning strategy than observing, doing by itself does not guarantee learning.
When using this basic formula to write a learning outcome, I suggest that you start simple and then make sure that the outcome is clear, specific, and measurable. For example, if I am planning a workshop in which I want students to learn to develop an effective resume, my progression might go as follows:
- SWiBAT develop a resume.
- SWiBAT use "power words" when developing a resume.
- SWiBAT develop a resume using at least five power words.
Oftentimes, student affairs practitioners try to start at #3, and with practice, that may be a reasonable goal. However, even after doing this for a number of years, I still typically start with just the skeleton of a learning outcome, and then I strengthen it to make it more clear, specific, and measurable. Over time, I've also learned a some additional strategies for using various theories or models to write stronger learning outcomes.
Hierarchy of Outcome Activity
In the most basic version of the formula, learning outcomes are behavior. While there are many behavioral psychologists who know more than I do who will suggest that behavior is not the most basic of human functions, I place behavior at the base of my outcome hierarchy. However, our learning goals can and should become more advanced than just addressing behavior. I suggest the following four-part hierarchy moving from the most basic to the most advanced:
- Behavioral - SWiBAT do something.
- Cognitive - SWiBAT know something.
- Affective - SWiBAT feel something.
- Spiritual - SWiBAT be known by something.
In light of this simple hierarchy, one could strengthen a set of learning outcomes by adding cognitve outcomes to behavioral outcomes, or affective outcomes to a set that is solely behavioral and cognitive.
I will note, however, that writing affective outcomes is tricky - one needs to first carefully check her or his biases when moving into this realm; this is even more true of spiritual outcomes. For the purposes of this web guide, I will only discuss behavioral and cognitive learning outcomes.
Bloom's Taxonomy
A second means of strengthening learning outcomes is to insert a "Bloom word" into the formula as follows:
SWiBAT + Bloom Word + Cognitive/Behavioral Activity
"Bloom words" (click here for a list of Bloom words) are action verbs that one can associate with one or more of the six levels of Bloom's taxonomy of thinking - knowledge, comprehension, application, analysis, synthesis, and evaluation. For example, in Bloom's taxonomy, comprehension is a relatively basic level of thinking, but synthesis is a higher order thinking skill. I developed the first of the following two learning outcomes using a comprehension-level Bloom word and the second using a synthesis-level Bloom word. Note how the second outcome reflects higher order thinking skill.
- SWiBAT discuss her/his internship experiences during a professional interview.
- SWiBAT generalize her/his intership experiences into 3-5 themes to discuss during a professional interview.
More Bloom Word Examples
Imagine that you are a resident director, and you have decided to use the six parts of Greenleaf's wellness wheel as learning objectives for your residence life staff. You are planning a workshop to teach the wellness wheel to your staff, and you would like to develop three learning outcomes for the workshop, each one demonstrating a deeper understanding of the wellness wheel.
For the first learning outcome, you want your staff to learn the six components of the model, which is the knowledge level of Bloom's taxonomy. Begin with SWiBAT. Then add a knowledge level Bloom word (list, define, describe, identify, show, label, collect, examine, quote, or name). Finally, clearly state what participants should know or be able to do as a result of participating in the workshop:
- Students will be able to + list + the six components of Greenleaf's wellness wheel.
Second, you want your staff to be able to use the model to develop programs in the residence halls. This is Bloom's application level. Given time restraints, you might limit the scope of your expectations, or you might have students work in groups.
- Students will be able to + apply + three wellness wheel components to residence hall programs.
- Students will be able to, while working in small groups, + demonstrate + how to use each of the six components of the wellness wheel to develop a residence hall program.
Finally, you want your staff to use more than one component of the wellness wheel in some programming efforts and you see an opportunity to help them build their synthesis skills.
- Students will be able to + integrate + at least two components of the wellness wheel in at least one programming effort each semester.
Stop and think for a moment. How long did this process take? It might take a little while the first couple times, but once you get the hang of it, this process should take no longer than 10-15 minutes. Granted, with time, you may wish to refine some of these learning outcomes to make them stronger, clearer, or more measurable, but these are fine outcomes for professionals beginning the process.
Developmental Instruction (DI)
Using William Perry's (1968) forms of intellectual and ethical development as a foundation, Lee Knefelkamp and Carole Widick created the Developmental Instruction (DI) model to Perry's theory to practice (See Knefelkamp, 1984). The DI model presents four variables of challenge and support than one can use to help students move from less or more mature positions in Perry's model. The four variables are:
- Structure: the amount of direction provided for students - more sophisticated efforts involve less structure.
- Diversity: the number of alternative answers or perspectives that students must consider - more sophisticated efforts require student to choose from a larger set of alternatives or to consider a larger number of perspectives when making decisions.
- Experiential Learning: the concreteness of learning - the most basic efforts involved students directly in the learning experience, while more advanced efforts require students to consider their learning in the abstract.
- Personalism: the safety of the learning environment - more advanced efforts challenge students to take greater risks.
I believe that the DI model is an excellent resource when designing both learning outcomes and learning interventions; the DI model also fits well with assessment efforts that utilize rubrics, as I will discuss later in this section.
When your goal is to stimulate greater critical thinking among students, consider the following example applications of the four DI variables as you write your learning outcomes:
- Structure: instead of requiring students to follow instructions or apply protocols to scenario, challenge students to generate their own instructions or protocols.
- Diversity: instead of simply responding to a case study with a suggested strategy, require students to develop three possible response strategies and then choose among the three.
- Experiential Learning: first require students to apply a given theory to their own experience, then call for them to evaluate their application process when applying it to a case study.
- Personalism: in the context of a dialogue, require students to assume the stance of a devil's advocate and defend that position.
Student Development Theory
In one of his now classic works, Alexander Astin (1993) critiqued many assessment and evaluation efforts because they only considered the outputs in reference to a set of criteria, but not in comparison to student capacity at the beginning of a learning process. For example, if two students both get 95% of the problems correct on their final exam, one cannot assume that both students learned as much in the class. It may be that one of the two could have earned an 85% at the beginning the class while the other could have only earned a 65% - in this case, the second student learned much more than the first.
When taken in its most basic form, this is a limitation of outcomes-based assessment. However, there are means of writing learning outcomes that allow for both input and output assessments. Student development theories, especially theories that use stages, statuses, positions, etc., can be helpful. While broad scale applications typically require some form of a "pre-test post-test" design - thus, violating the Keep It Simple Stupid principle - smaller efforts do allow student affairs practitioners to assess student learning development on more of an individual basis. I will return to this concept later when discussing rubrics below. Levels of Assessment
Once you have developed your learning outcomes for a specific learning intervention, the next step is to plan an assessment strategy. In other words, plan how you will demonstrate whether or not students achieved the learning outcome, or to what degree they mastered the learning. For most of your learning objectives, your assessment will fit into one of three levels: checklists, scales and ratings, rubrics.
Checklists
Assessing many programs involves little more than examining a checklist. Activities such as dances or attending athletic events have great value - they help students to build social relationships and gain a sense of belonging on campus; they also provide students with a healthy release that can promote retention and even eventual financial support for the institution. That said, the learning outcomes for these activities are difficult to write in clear and measurable statements, and they are even more difficult to measure.
My recommendation is to be happy with a checklist for such activities - the assessments may be as simply answer "did we get it done?" and "how many people participated?" More advanced checklists might outline a series of tasks to complete or topics to address. These simple assessment questions provide valuable information for future decisions about which activities to provide, and how to organize them.
This leads to an important segue. While the learning outcomes for students who participate in a dance may be more generic, the learning outcomes for students who organize the dance can be quite specific. These students have opportunities to learn important lessons about budgeting, planning, organization, teamwork, marketing, etc. Further, student affairs professionals can write clear learning outcomes for these learning lessons, and then develop specific means to assess student learning.
Scale and Rating Assessments
While checklists are a nice start, for the second year, I suggest you develop scale or rating assessments for all major programs listed in your first-year matrix. At its most basic level, this simply involves follow-up after each programmatic effort for reflection and to assign a likert scale type rating. The result at the end of the year will be a revised matrix. Instead of simply providing checkmarks that connect or map learning experiences to learning objectives, you will replace these checkmarks with likert ratings. At the end of the year, you will be able to go beyond simply reviewing frequencies (how often did you address each learning objective) to a basic assessment (how well did you address each learning objective). Check out the following example:
At some point during the process, several individuals will disagree about the rating for a certain program. A standard likert scale of 1-4 or 1-5 is helpful, but there is no clear sense of what each number means. For example, most of us have seen surveys in which someone rated everything as a perfect 5 on that 1 to 5 scale. This may be a nice compliment, but as an assessment, it is little more than a checklist.
One means of improving the likert scale is to add criteria values to each number on the scale. These could be as simple as "1 = poor," "2 = basic," "3 = competent," and "4 = accomplished." More advanced criteria might set numerical values for participation (e.g. 1 = 0-15 participants, 2 = 15-30 participants, etc.) or assign Bloom-like levels of sophistication for thinking to each number on the scale (e.g. 1 = knowledge/comprehension, 2 = application, 3 = analysis/synthesis, 4 = evaluation).
Rubrics
Rubrics are similar to criteria ratings, but they differ in that they are tailored to fit specific programs or outcomes. Consider the "know and do" activity from the first year of the process. The learning outcomes for that program were:
- Participants will develop a set of 3-6 cognitive/behavioral learning objectives for a student affairs program or department.
- Participants will write the learning objectives in a manner that is clear and measurable.
- Participants will develop learning objectives that are consistent with the department's/institution's mission and that integrate student learning/development theory.
The accompanying assessment rubric was:
Poor |
|
Basic |
|
Competent |
|
Accomplished |
The group failed to synthesize brainstormed ideas into fewer than seven cohesive learning objectives.
OR
The content of the brainstormed ideas lacked sufficient depth to be organized into at least three distinct learning objectives. |
|
The group developed 3-6 distinct learning objectives, but failed to present them in a manner that, at its face, is measurable.
OR
The group failed to distinguish between cognitive/behavioral and affective objectives. |
|
The group developed 3-6 clear and measurable learning objectives. Each of the learning objectives was cognitive/behavioral in nature.
The group did not clearly develop connections between their learning objectives and the mission of the institution/ department and/or with student development/ learning theory. |
|
The group developed 3-6 learning objectives that were clear, measurable, and cognitive/ behavioral in nature. The group further drew clear connections between their learning objectives and the content of the department's and/or institution's mission statement as well as to relevant student learning/ development theories. |
This rubric helped to mark the group's progression from the first to the third learning outcome. For example, a group that mastered the first two learning outcomes, but not the third would fit in the "competent" range of the rubric.
At the more advanced level, one might develop a separate rubric for each learning outcome (see the cocurricular lesson plans below for examples).
Assessment Tools
On the overview page for this assessment guide, I critiqued the tendency of practitioners to begin discussions about assessment with talk of surveys and questionnaires. Now that you have some more context, let me restate this more boldly:
I would rather have an assessment program that involves nothing more than writing learning objectives and outcomes and assessing these with intuitive professional judgments than an assessment program consisting of a comprehensive set of expensive standardized surveys and questionnaires that lacks a foundation in student learning objectives and outcomes.
The best option, of course, is to first set the foundation in place, and then draw on a broad set of assessment tools to collect data for the assessment model. While this should involve some surveys and questionnaires, I suggest a number of additional means for collecting data that are less expensive and more practitioner friendly.
Newsprint
This is one of my favorite assessment tools. Student affairs practitioners often ask students to break into small groups, write their ideas on a large sheet of newsprint, and later use that when reporting back to the group. The assessment key is simple - keep the newsprint (or take digital pictures of it), and then compare the content of the newsprint to your learning outcomes, and ideally to your rubric.
Formal Observations
When conducting an activity with a partner, make sure that someone has a notepad at all times and is writing down observations - student comments, non-verbals, questions, etc. Later, compare these written observations to your your learning outcomes or your rubric.
Note Cards
Hand out index cards to participants and ask them to write down three things they learned on one side and three things that were unclear or missed on the other side. You can also have students list what they would like to learn more about in future activities or rate their own learning on a scale of 1-5 and explain their rating.
Online Discussion Boards
I'm amazed at what students will say to a computer that they would never speak out loud. This is especially true for traditional age students coming out of high school. For your assessment tool, set up an online threaded discussion or blog about your event - tell students that for each post to the discussion, you will add their name to a drawing for a bookstore gift certificate. You will be surprised at the results, and you will gain meaningful data from your more reflective students who are not as comfortable sharing feedback on the spot.
Photos
Pictures can be worth a thousand words, especially for activities with social or relational learning outcomes. If you spend just a few minutes explaining your learning objectives to you photographer in advance, she or he may be able to capture some interesting moments when this learning is or is not happening.
Video
Many students come to college with high levels of competency using video - utilize this. For example, consider a new student orientation program that includes several relational learning outcomes framed in terms of meeting faculty, staff, and other students. Ask one or more returning students working with the program to use a videocamera to tape these relationships as they happen; ask this student to keep an eye out for students standing or sitting by themselves in the corner of a room. Use the video later to see which kinds of interactions seemed to work best and which ones did not; also note where students seemed to fall through the cracks. You may be surprised at what you see!
Stories & Quotes
Carry a hand-held tape recorder. During or near the end of an activity, walk around and interview students about what they learned and how they will use it. You can also ask them to share thoughts about how to improve the program next time.
Interviews & Focus Groups
Invite a graduate intern who is completing a qualitative research methods course to conduct a series of in-depth interviews and/or focus groups with a target population, and then ask for a copy of the final paper. Better yet, offer the student a small honorarium if she or he will write up a brief executive summary in addition to the 10-20 page paper he or she is writing for class.
Questionnaires
As I'm using the term here, questionnaires are short, informal surveys that are used only once or twice. Though they require planning, they do not require the same level of piloting and/or testing as do surveys. While I urge you to use these sparingly, they can be a valuable assessment tool.
Surveys
As I'm using the term here, a survey is a formal instrument that practitioners use on a regular basis, typically on an annual basis. Unless there is someone on your staff who has special interest and/or expertise in developing surveys, I suggest that you utilize a graduate student in a research methods course to help develop your surveys when possible. You will find that surveys become more valuable with time as you begin to collect longitudinal data.
Cocurricular Lesson Plans
When working with graduate students and new professionals, I require them to write cocurricular lesson plan to help them become more intentional about learning, theory, and assessment in their work. When introducing a new assessment effort, I also recommend that each professional develop at least one cocurricular lesson plan as an exercise. Below, I provide a potential template for these lesson plans and a list of several example lesson plans that I have developed over the years.
Note that there is a space at the end of each lesson plan for assessment results. You can easily add photos, numbers, or written summaries from any of the assessment tools above directly into the file to create electronic documentation system for tracking your results. You can later print these or upload them to a website as part of the documentation portfolio for accreditation visits.
External Evaluator
The final step for the first year is to invite an external evaluator in to review your system. Just as the initial baseline CAS self-assessment helped you to review your progress and identify potential gaps in your data in year one, the evaluator can serve a similar function at the end of the second year. You should search for someone with a reasonable degree of expertise in assessment and in this specific area of the cocurriculum. Ask your evaluator to review your print materials, meet with staff, and meet with students to learn about your process. If time works out, draft your year-end report (see below) first, and have the external evaluator review your report as well. At the conclusion of the visit, your evaluator should provide you with an oral summary of her or his findings, and later with a concise written report.
Year-End Assessment Report
The year-end report for the second phase of the process will be similar to the report for year one, though with several additions (see italics in the list below). That said, the report will probably take less time to write now that you have some experience.
- Summary of major assessment process accomplishments
- Program strengths and areas of improvement
- Areas for growth
- A couple examples of assessment successes
- Current gaps in your assessment evidence
- Recommendations for improvement in year/phase three
- Recommendations for next steps in your assessment process
You will also want to include supporting materials including the following:
- Learning objectives for the program
- Example learning outcomes from leveraged programs
- A copy of the revised matrix
- Any other supporting materials that you believe are relevant
As with the first year assessment report, be sure to use the content of this report to inform your planning and decision-making for the coming year. Ideally, you should complete this report with several ideas about how to improve practice and do a better job of promoting student learning in the coming year.
|