ILP Norming Session Findings
The Teaching, Learning and Assessment (TLA) Committee sponsored a norming session at SNL’s July 2015 faculty meeting in order to clarify ILP assessment guidelines, establish consistency among competence domain assessors, and identify recommendations faculty assessors have for the ILP assessment form. The process for norming involved domain specific groups of faculty members assessing a single ILP by competence (N=8 for Arts & Ideas competence; N=5 each for Scientific World and Lifelong Learning; N=4 for Human Community).
Once assessors completed their ratings, they announced their scores for each criterion in their domain specific group and the group’s note-taker (TLA member) recorded each score for each criterion. These ratings were then used to calculate the interrater reliability by competence domain. In addition, faculty competence specific assessors discussed the following questions:
a) If this student had more time to work on this ILP, what would you recommend he/she do to improve it?
b) Are there any other items or criteria would you add to the assessment form?
The complete findings of the report can be viewed here.
SNL BA Assessment Report 2015
Part II: Report on This Year’s Assessment Project
The objective of this SNL’s undergraduate program annual assessment project was to assess student learning for the L1 competence, and to compare the effects on student learning of three course designs (v1.0, v1.5, and v2.0) of the Independent Learning Seminar. A rubric was created, tested, and used on a randomly selected sample of students from each version. Overall, findings indicate that a high percentage of students demonstrate L1 mastery, and the course designs do not have a significant effect.
In Summer 2013, as a result of undergraduate curriculum improvements, SNL revised its L1 competence to: “Can use independent learning skills and strategies to organize, initiate, and document prior, current, and future college-level learning.” A new course, Independent Learning Seminar, was developed and required of students’ to demonstrate their L1 competence mastery.
Three versions of the Independent Learning Seminar have been taught to date. The first (v1.0) and second (v1.5) designs of the course require students to complete an Independent Learning Pursuit (ILP) as their final deliverable in the course, whereas the third (v2.0) version of the course focuses on smaller assignments called knowledge synthesis pages (KSP) as deliverables. We examined the type of deliverable as a factor in demonstrating L1 competence, particularly students’ skills in documenting their competence-based learning.
Interpretation of Results
Students in all three sections scored fairly well across the four learning indicators, which were derived from the competence criteria. Overall, 79% were satisfactory or above. Although all criteria had slight variations, the criterion with most variation was surfacing prior learning, which was statistically significant. This may indicate that students in 1.5 were better able to recall prior learning experiences than students in 1.0 and 2.0, or that the assignments in 1.5 better helped students evoke prior learning experience. Instructor bias may play a role in these findings given that there is only one 1.5 instructor. Follow-up inquiry will include the repeated testing of the L1 competence across a variety of instructors.
SNL is pleased to find that students are achieving the L1 competence. More discussion is needed to determine whether an Independent Learning Pursuit is the desired deliverable of the ILS course.
Recommendations and Plans for Action
Based on our findings, we will propose to SNL Curriculum Committee the following recommendations:
- Examine techniques and structure from 1.5 for possible replication in other courses
- Require use of the rubric by ILS instructors to give a larger sample size than the 33 assessed
- Consider the pros/cons of ILP as a course deliverable and when the offering of the course is most beneficial for achieving L1 competence.
In deliberation with SNL Curriculum Committee, we will finalize the following actions:
- Identify course components from 1.5 that may be relevant in all ILS offerings (January 2016).
- Based on the ILP pass ratio, consider a competence revision. Deliberate with the Curriculum Committee to determine whether the ILP should be included as part of the competence statement (Fall 2015).
- Deliberate on the need for L1 competence revision and or rubric refinement Fall 2015-Winter 2016.
- L1 champion to distribute and require ILS instructors to use ILS rubric (Spring 2016).
The complete report and findings can be viewed here.