Next Steps and Other Considerations
Page Contents
Repeating the Cycle
I shared early in this web guide that one of my goals is for assessment to shift from being something extra that student affairs professional do to an integrated part of what we do and, to a degree, of who we are. It is in this spirit that many assessment professionals have described the process as iterative. Some have incorrectly interpreted iterative as synonymous with cyclical, but it is more than that. Assessment as an iterative process is a cycle that builds on itself so that each cycle changes and evolves because of lessons learned in earlier cycles. This three phase model spends one year focusing on learning objectives, a second on learning outcomes, and a third on a comprehensive program evaluation. That could naturally progress into a three-year cycle, or it might evolve into something more. If nothing else, future cycles will need to give greater attention to how practitioners use the results from earlier cycles to promote and improve student learning.
Multiple Measures
Another important element of future cycles should be developing multiple measures to assess your learning objectives. Let me draw upon two metaphors to explain what I mean:
A quantitative researcher wants to study self-esteem, but quickly recognizes that there is no direct means to measure this concept. So she develops and tests a quantitative instrument with a long series of items, or questions. Her assumption is that if there is such a thing as self-esteem, it should consistently influence how participants answer the various questions of her instrument. If she wants to be able to draw the strongest possible conclusions about self-esteem, she will need to do several things: 1.) use a large number of tested and reliable questions to measure her self-esteem construct, 2.) administer her instrument to a large and diverse sample of participants, and 3.) continue to administer her instrument to new populations over time.
A qualitative researcher also wants to study self-esteem, but recognizes limitations in measuring the concept. He decides to conduct a series of qualitative investigations. In order to ensure that the results of his study are trustworthy and good, he draws upon a number of different qualitative approaches including direct observations, interviews, and focus groups. He also conducts document analyses of works written by his subjects. He then conducts a number of member-checks in which he goes back to the participants of his study to make sure that his conclusions are consistent with their understanding of self-esteem.
Just as no one measure of self-esteem will do for either of these researchers, no one measure of student learning will work for student affairs professionals engaged in an assessment process. Drawing from these metaphors, I suggest the following regarding assessment practice at its best:
- Assessment professionals should draw upon a broad and diverse set of student participants;
- Assessment efforts should employ different kinds of tools or measurements (e.g. newsprint, interviews, surveys, etc.) to triangulate results;
- Practitioners should involve students in the process and check with them to ensure that the results fit with their own perceptions about their learning;
- Assessment efforts should be ongoing and longitudinal in nature, considering both current results and long-term trends.
Other Considerations
One of my purposes in developing this website was to help student affairs professionals design and implement quality assessment programs at a low cost. While neither of the following are free, they may prove to be a worthwhile investment.
Consultants
There are many reasons to consider hiring a consultant to help you implement your assessment program. I will highlight three:
- There are a number of great assessment conferences available to student affairs professionals, and I encourage you to attend if possible. That said, a good consultant may be cheeper than financing a conference trip for you and several members of your staff.
- While the design and implementation process is not overly complex, there will be times when confusion or frustration develop. A good consultant can help you to anticipate these moments and diffuse them before they become counterproductive.
- Lastly, the "100 mile rule" often works - a consultant may not bring any new ideas to the table, but some people will listen to the ideas more carefully if they know they are paying for an out-of-town "expert" to come in and show the way.
Standardized Assessment Tools
One might conclude from the content of this web guide that I am not a big fan of standardized assessment instruments. This is not the case - they can be highly valuable. That said, before I suggest strategies for using standardized tools, let me share a couple concerns:
- Many institutions use standardized instruments as a substitute for a comprehensive assessment program grounded in student learning objectives and outcomes. As I have noted elsewhere, I would rather have a solid learning foundation and no assessment tools than many tools with no foundation.
- Many institutions make poor choices about the standardized instruments that they use. Rather than strategically selecting an instrument or a set of instruments to fill evidence gaps or to triangulate data regarding core learning objectives, many professionals choose instruments because of price, a recommendation by a colleague, or because they received a marketing phone call.
- Too often, institutions administer standardized instruments, but then fail to use the results. Sometimes, a single department hoards the data. In other cases, professionals simply read the executive summary, but never dig in and ask deeper questions by disaggregating the data. In other cases, administrators rigidly interpret data, allowing the results (or perceived results) to make key decisions for them rather than informing decision-making processes.
Having stated these concerns, the implications for the effective use of standardized tools are rather straight-forward:
- Strategically choose the instruments that you will use - make sure that they either fill evidence gaps in your assessment plan or triangulate data regarding your most important learning objectives.
- Consider using a cycle of instruments. For example, you might choose a three year cycle in which you administer a different instrument in each of the three years of the cycle.
- Read more than just the executive summary of results. Be sure to order the actual data in addition to summary information and then work with someone with the appropriate expertise to disaggregate and analyze your data in greater detail.
- Disseminate your results -- be sure to communicate the results to a wide array of professionals across the university and make data available to others who have a reasonable interest.
- Use the results to inform decision-making processes.
- Keep your results and examine trends that emerge over time. Remember that the return on your investment for a standardized instrument may come many years in the future - trends of how student learning changes over time are often more valuable than information about learning in any one given year.
|
|
Contents
|