Closing the Loop
Closing the Loop
Planning involves writing outcomes considering: What do I want students to learn?
Doing is teaching and learning considering: How do I teach effectively?
Checking is evaluating student work considering: Are my outcomes being met?
Acting is revising and reinforcing considering: How do I use what I have learned?
Image credit and above text modified from: https://www.westminster.edu/academics/accreditation-assessment/cycle.cfm
Closing the Loop Resources
Closing the Loop Strategies
Celebrate! Reward! Share! Keep Going!
When your assessment findings reveal weaknesses, consider the following:
- Do you have the right learning goals? Do you have too many learning goals?
- Do you need to improve your assessment methods?
- Take a hard look at your curriculum:
1) Course content and requirements
2) Program goals and requirements
3) Sequencing and prerequisites
4) Teaching methods and existing assignments
5) Faculty/staffing support
6) Admissions criteria
7) Placement criteria
8) Advising
9) Tutoring
10) Co-curricular activities - Develop concrete plans for implementation of changes. Develop a reasonable timeline, usually multi-year.
- Develop a plan to assess (measure) the impact of those changes.
- Sometimes it really is the students’ fault.
- Keep going!
From Linda Suskie, Assessing Student Learning: A Common Sense Guide (Jossey Bass, 2009)
Closing the Assessment Loop
Faculty Enrichment Session: March 4, 2015
Presenters:
- Pam Pennock, Campus Assessment Coordinator
Presentation powerpoint and remarks - Yi Maggie Guo, Associate Professor, College of Business
Presentation - Nadja Rottner, Assistant Professor, Art History
Presentation - Troy Murphy, Associate Professor, Public Communication and Culture Studies
Closing the Loop Program Examples
Program faculty administered a survey to students in the introductory course for the major (AAAS 300) to measure students’ perceptions of how well the course was meeting the program’s learning goals. Additionally, the program assessed students’ aptitudes on a major research paper in an upper-level course (AAAS 470).
Closing the Loop action: The program will consider splitting the intro course into two courses because evidence shows one course cannot adequately address all the program goals, as intended. Evidence from the AAAS 470 assessment led faculty to conclude that in research/writing courses they should create shorter scaffolded assignments so students can develop research and writing skills in preparation for the big paper.
The program assessed Anth 101 by examining student performance on key exam questions that are linked to particular program learning outcomes.
Closing the Loop action: The faculty member will change the exams to introduce short answer questions that require the students to flesh out key concepts. These questions will be used a benchmarks in future assessments.
The program assessed its Critical and Creative Thinking learning goal by examining student performance on the field trip paper assignment in two 100-level art history courses.
Closing the Loop plans: A question that emerged from this assessment is: can we more thoroughly convey the objective of this assignment to students prior to the museum field trip? The program will consider adding an exercise such as having an in class discussion of an unfamiliar object and asking students to interpret the meaning and possible function of the object based on comparisons to covered material. This in- class practice will better prepare students for the critical analysis required for the museum field trip paper.
Biology, BS (2015-2016)
Closing the Loop actions: Based on prior assessments, the Biology program has implemented the following practices: 1) Adopted a benchmark that students are achieving a learning goal if 80% of the students in a course meet or exceed expectations; 2) Shared assessment rubrics; 3) Encouraged the common use of A Short Guide to Writing about Biology in order to improve writing skills.
The discipline assessed their program learning goal related to biochemical research methods. In BIOL/BCHM 474, the instructor found that 40% of the students did not meet expectations for this goal on an assignment that required them to locate information about genes and their growth and employ that information in experiments.
Closing the Loop action: The faculty member will introduce a new assignment into the course that requires students to write a grant proposal summary after they conduct their literature search. This assignment will help him identify weaknesses in student performance earlier in the term and allow him to intervene and assist.
Program faculty assessed students’ written and oral communication skills in their course projects and lab reports, and their oral presentations of their findings.
Closing the Loop action: Faculty concluded that this year’s introduction of one-on-one meetings with students to provide specific feedback was effective in improving their communication skills, and they will maintain this practice.
Closing the Loop actions: Based on prior assessments, the Biology program has implemented the following practices: 1) Adopted a benchmark that students are achieving a learning goal if 80% of the students in a course meet or exceed expectations; 2) Shared assessment rubrics; 3) Encouraged the common use of A Short Guide to Writing about Biology in order to improve writing skills.
The discipline assessed their program learning goal: "Ability to communicate scientific information in writing." In her evaluation of lab reports in BIOL 303, the instructor found that most students made great strides in their scientific communication skills over the semester, but that many of them struggled to making arguments based on data in their conclusions.
Closing the Loop action: The faculty member create an assignment to specifically address the common problems in the lab report conclusion section for future offerings of the class.
To assess the Business program’s ethics learning goal, the program measured how well students could apply an ethics framework to a case study scenario in an Organizational Behavior course.
Closing the Loop plans: In the future, the faculty member will reorganize the instruction of this lesson to provide students with more guidance before they undertake the exercise. Furthermore, the faculty will consider having the students take the Ethical Lens Inventory in advance of the assessment exercise.
To assess the Business program’s technology knowledge learning goal, the program measured students’ performance on a multiple choice exam testing six key concepts, administered in an Information Technology Management course.
Closing the Loop plans: The faculty member will re-evaluate and consider re-phrasing the exam questions to strengthen its usefulness and fairness as an assessment tool for this learning goal. Additionally, the faculty member will implement more in-class assignments to ensure students have more practice and are better prepared for the exam.
The program assessed the effectiveness of a new, scaffolded approach for evaluating students’ lab reports in two upper-division courses.
Closing the Loop plans: Finding that the new method produced student work superior to prior classes, the faculty member will endeavor to convince his colleagues who teach these courses to adopt this method.
The program assessed several learning outcomes, including the ability to apply knowledge of computing and math, and an ability to communicate effectively with a range of audiences. Instructors used student projects and exams to assess students’ proficiency.
Closing the Loop plans: The program recommends that CIS 200 become a prerequisite course for CIS 275. In CIS 4951, the instructors will improve the timeline logistics for the research project to improve student performance. Further, the instructor plans to eventually implement two new programming protocols, UDP and distance vector.
Using a rubric, the faculty member assessed students’ quantitative reasoning skills (a program learning goal) in their research projects.
Closing the Loop plans: The faculty member will introduce new homework assignments for students to practice skills needed in their research projects and consider having students peer-evaluate rough drafts of the project.
English assessed the following program learning goal: “The ability to conduct and incorporate research into writing about texts, providing appropriate documentation.” Faculty found that students on the whole were proficient in research skills and attributes this good result to the incorporation of research assignments throughout the curriculum. Nevertheless, some students’ research skills can be strengthened and reinforced in upper-level courses.
Closing the Loop plans: 1) Reemphasize library research skills in upper division courses, for example by visiting the library. 2) Place more emphasis on teaching students to incorporate source quotations into their writing. 3) Engage in more discussion among English faculty as to what constitutes research-intensive pedagogy in order to learn techniques from each other and achieve greater consistency.
Two faculty members evaluated students’ learning of key ecological concepts on the final exam in the Ecology course (ESCI 304).
Closing the Loop plans: The program plans to implement more assignments and quizzes throughout the term, especially assignments that require critical thinking. They will encourage students to take advantage of Supplemental Instruction, possibly by offering extra credit for attendance at SI sessions. They will also require students to complete Science Learning Center modules. On the final exam they will include broader questions in order to better capture students’ conceptual learning.
In BIOL 545 (Restoration Ecology), to examine the program goal "Ability to understand underlying concepts and principles associated with environmental aspects of biology, chemistry and geology," the instructor designed several exam questions to challenge students with the "choices," based on political, economic, and environmental factors, going into an ecology project. The exam questions then asked students to estimate the long-term value to the region and its population.
Closing the Loop plans: Evaluation of the students’ essay responses alerted the instructor to many students’ “rationale bias” based on the timing of their hypothetical ecology projects. In the future he will try to discover student rationale bias by giving them a case study defining the time (e.g. restore to 1800). He will also do the reverse: e.g. if we wish to restore an area to minimize frog and turtle migration casualties, what would be the level of success and what period in human occupation here would correspond to that success rate?
To assess the expectation that MSF students are knowledgeable about ethical issues in finance, the program assessed relevant essay questions on an exam in FIN 581.
Closing the Loop plans: Although most students met the goal, the program plans to supplement the lesson with a case study where students need to examine an ethical dilemma.
The discipline administered a pre-post quiz to assess various aspects of students’ Spanish language abilities in SPAN 101. Faculty found surprisingly high scores on the pre-quiz, which revealed many “false beginners” in the class -- students who should be taking a more advanced Spanish course.
Closing the Loop plans: Spanish faculty used this data to successfully press CASL for the implementation of a placement exam for all freshmen.
In an introductory U.S. history course, the program assessed students’ ability to analyze primary sources.
Closing the Loop plans: The assessment exercise found that most students excelled at analyzing sources when guided by the instructor, but many floundered when presented with a source to analyze on their own. In the future the instructor will build more opportunities in the classroom for students to understand how to situate primary sources within their historical context. Besides implementing activities that hone their skills in primary source searching, the instructor plans to design in-class activities where students are given primary sources (not from the syllabus) and asked to describe connections to the lecture material.
To assess how well students grasp the knowledge necessary “to design, install, improve, and evaluate large integrated systems,” the program assessed students’ performance on homework problems and projects in two courses.
Closing the Loop plans: The assessment results in IMSE 501 show that the program goal has been met for all the assessment questions except for homework problem 1. In problem 1 several students had difficulty identifying real risk measures based on the description of the risk factors learned in class. Next year, the course will include more cases and examples on how to classify risks based on common risk factors.
In MIS 649, the program assesses students’ data research projects to measure how well they grasp the learning outcome “Students will use data to provide solutions to business questions.”
Closing the Loop plans: The program found that the students need extra guidance in reporting their projects. The instructor will share the rubric and a suggested outline of the report with the project instructions.
Faculty examined their students’ performance on the Michigan Test for Teacher Certification. Passing rates on the English and Language Arts tests have fallen. Faculty studied the students’ coursework and determined that too many of the courses required for the Language Arts major do not sufficiently cover core language arts content and that there are too many electives.
Closing the Loop actions: Review the major’s requirements and make changes in the curriculum to better address core content for English language arts. Changes may involve redesigning courses and substituting courses in the major.
To assess students’ ability to understand and apply basic principles in microbiology in the area of microbial diversity and ecology, the program assessed their performance on relevant exam questions in the course MICR 405.
Closing the Loop plans: To improve students’ understanding of this program goal in the future, the instructor will reorganize the three-hour lecture block to include more opportunities for active learning, content summaries, and breaks in between the delivery of new material. The instructor will also provide short video lecture of difficult concepts to allow students to review later as needed.
The discipline assessed its goals related to Mathematical and Computational Physics: “Understand and utilize the mathematical tools commonly used by physicists, including calculus, differential equations, linear algebra, Fourier analysis, and numerical methods” and “Know how to use a variety of techniques to organize, display, and analyze experimental data.” Faculty found the students lacked prior experience and aptitude on these skills.
Closing the Loop action: To address this need, the discipline developed a new course, PHYS 314, Computational Physics. The course is targeted at physics majors about to begin their upper level physics courses.
Physics faculty assessed the following learning goal: “Understand, utilize, design, and construct scientific instruments and data-collection systems for the experimental study of physics.” Faculty examined current instruction methods for use and configuration of a scientific instrument, the digital multimeter, and found the methods deficient.
Closing the Loop actions: 1) Complete revamped the Science Leaning Center module on the use and configuration of the digital multimeter, and replaced old equipment with equipment that matches the equipment used in the PHYS 126L and PHYS 151L experiments. 2) Require students in PHYS 151L to complete the renovated SLC module before attending the lab in which the multimeter is first used.
Psychology graduate faculty assessed several program goals (Psychotherapy Skills Acquisition, Assessment Skills Acquisition, Testing Skills Acquisition, Research Methods) using a graduate student exit interview. Faculty found that students have achieved these skills, but they often have difficulty with case conceptualization and treatment planning.
Closing the Loop action: 1.) Add an additional case conceptualization to Psych 552, the second psychotherapy class, to make it more of a capstone class for that competency. 2.) Standardize the process for collecting and analyzing data across goals and classes.
Communications faculty assessed the following program learning goal: “Integrate the theory and practice of communication in multiple contexts.” Faculty found that students were more proficient at integrating theory and practice in personal and family contexts than in the context of public discourse.
Closing the Loop plans: 1.) Increase discussion of democratic deliberation and citizen action in small-scale settings through incorporation of specific texts and films in certain courses. 2.) Increase the program’s offerings in topics such as health communication that bridge the gap between personal and public identities and understandings. The program has hired a professor who specializes in health communication.
Faculty in two courses used exam questions and projects to assess students’ ability to apply mathematical knowledge and solve engineering problems.
Closing the Loop plans: Faculty will conduct additional mathematical review at the beginning of the term, and provide more TA and recitation support for students.
Using several courses and assignments, the program assessed how well students can: 1) design a system to meet desired needs with realistic constraints; 2) communicate effectively; 3) use engineering skills and tools.
Closing the Loop plans: The program recommends including more emphasis on software engineering tools and user interface design and creation in CIS 285 to better prepare students for CIS 375. In CIS 375, the students need more instruction in citing and using references in their documents to better prepare them for CIS 4961. To improve the capstone research seminar (4961/2), the logistics of scheduling, class size, team size, and instructor consistency need to be carefully considered in order to maximize student performance.