IPA2016

Kids Taking Action

Writing Great Rubrics

Rubrics are wonderful tools for defining quality for a process, product of behavior.  They are powerful tools for both teaching and learning as they help communicate what it takes to succeed

Rubric Design
A great rubric has row which list each criteria or essential skill that will be measured. This is not to list every possible skill that could be measured in this task, but the key and essential skills which need to be measured.  The columns should be the potential levels of performance ranging from ideal to not evident.

TIP: When writing the performance levels it is often easiest to start by describing ideal and work backwards.

Download Our Classroom Rubric Guide here:

http://www.kidsatthecore.com/Downloads/ClassroomRubrics.pdf

Understanding Assessment Quality: Validity, Reliability and More

I find it hard to imagine a system that is completely perfect and void of human influence or error.

However, there are ways to increase the confidence we can have in the system.

1) Start with valid assessments.  What is a valid assessment?  It is an assessment that accurately measures what it is intended to measure.  That means the questions are written so students get the correct/incorrect because of their knowledge of that content and skill NOT because of cultural bias, NOT because of excessive wordiness, etc.   For example, a question with a very difficult vocabulary word might become invalid because students are getting it wrong because they don’t know that word, not because they don’t understand the concept.  The data is measuring who knows the word: not what you intended to measure.   You can’t prove an assessment is valid until you have historical data, so it is important that we give the assessment to the students and examine the scores.  “…Validity is concerned with the confidence with which we may draw inferences about student learning from an assessment.  Furthermore, validity is not an either/or proposition, instead it is a matter of degree,” (Gareis & Grant, 2008 p.35)

Thus, increasing validity will be an ongoing district process.

2) Start with Reliable Assessments.  What is a reliable assessment?  It is an assessment that will yield repeatable results. If you take an assessment and give it to a group of kids one year, then give it to a similar group of students next year you get similar results.  Reliability in selected response (multiple choice) tests is easier to achieve.  However, if a teacher gives an assessment to period 2, realizes some trouble the students had and therefore gives different directions to period 4 than they are interfering with the reliability.  Thus, reliability requires consistency.

Rubrics are a viable and great tool for measuring growth over multiple data points.   It is important to recognize, however, that ensuring reliability (repeatability of results) can be a little more difficult with a rubric-graded open ended task than a multiple choice test.  Rubrics must be created and implemented so that the grader(s) have very specific understandings of what each level of the rubric means.  Inter-rater reliability, or consistency between multiple graders, is important so the whole team would agree on the same score for the same work.  Even more essential is that the individuals grading are consistent within themselves.  That means when you grade a batch of student work, and then another batch of student work several months or even a year in the future, your scoring methods are the same (Student work earning 3 looks the same as another sample earning a 3, as another sample earning a 3.  Every time).  Ensuring repeatable results with the same rubric is ESSENTIAL to getting reliable data as well as useful data for talking about students and how they are growing.

3) Assessments Should be High Quality : Teachers should be able to explain why the assessment set accurately measures student growth in the key areas of their curriculum and administrators should be able to understand common elements in quality growth assessment design.  Using a district rubric or checklist that looks at alignment, distractor and wrong answer use, growth design, cognitive demand, validity and reliability will help indicate the quality of the assessments.  “Gaming the system” with unaligned, easy, or schemed assessments should thus be highlighted and prevented.

4)  Make Data Collection Simple: Asking teachers to fill out complex spreadsheets opens us up for unintentional errors in data entry.  Systems of scantron tools and ways to consistently and automatically collect data will minimize errors both intentional and unintentional.

Multiple Teachers Grading:

The validity should not be affected by the fact that there is more than one grader.  In fact, the multiple minds at the table during the creation process should help increase the validity of the assessment and help increase the fact that the assessment truly measures what it intended to measure.

Reliability is another issue altogether.  When multiple teachers are grading assessments, we need to make sure their results are repeatable no matter who grades the test.  Start by increasing inter-rater reliability by having a “trade and grade” professional development event.  Teachers can learn about how the others would have graded the same questions.  Some districts have use two graders on the same assessment and use averaged the score.  Other districts don’t allow teachers to grade their own student’s work.  Ultimately, if multiple teachers using the same assessment have worked to ensure a high degree of comparability in the way they give scores, the data produced will be reliable.

 

Writing Assessment Questions

When writing quality questions to gauge student understanding and learning of a skill or topic, it is important that teachers are very intentional about the type of question they choose to use. Why? #1: Taking assessments takes time!  Every time you give a test, you give up instructional time.  Make that sacrifice worth it.  Make the assessment crafted carefully so you get good data about student learning back.  Think about what elements of student thinking you want to learn about, and craft questions carefully to get at that thinking.  #2:  Grading assessments takes time!  Every question you grade takes your time from other tasks (even taking a well deserved break!).  So, if you are going to write a free response question as opposed to a multiple choice question…and take the careful time to grade it well, make sure it is the right kind of question that will give you more information about student thinking.  For example, if you ask students to recall 3 facts from an article you will likely not gain much more information about student thinking by asking students in extended than a multiple choice question.  If that is the case, save your grading time and use the multiple choice format.  Use extended response questions for a topic that requires analysis, or critical thinking processes:  these tell you more about student misconceptions and what your instructional “next steps” should be.

There are essentially three ways we can assess student knowledge:  1)  Selected Response Questions 2) Constructed Response Questions and 3) Performance Based Questions.

Lets look at each of these in detail.

1) Selected Response Questions:  This type of question will ask students to select the correct answer from a provided set of answers. Examples include: True/False, Matching, Multiple Choice, and fill in the blank (if answer list is provided).

Considerations when using Selected Response questions:

Selected response questions will take more time to write because you have to write the wrong answers.  Writing the wrong answers can often be the most time consuming part of the process.  These questions, however, do take less time to grade.  So the time investment is very “up front.”  The data you get from a selected response question really hinges on the quality of your distractors.  If you have a correct answer and 3 “throw away” answers, teachers won’t get a good idea about student misconceptions from their wrong answer choices.

It is important to realize that selective response questions may not be appropriate for all ages or all content areas.  For example content areas that are performance based (ie: fine arts classes) might not be well assessed through a series of selected response questions.

2)  Constructed Response Questions:  This type of question asks students to construct their own answer to a question typically by writing a short response or working through a problem on paper.  Examples include: open ended math problems, essays, extended response items, and even graphic organizers. Considerations when using Constructed Response questions:

Constructed response questions are easier to write than a selected response question because the writer doesn’t have to create any wrong answers (distractors).  On the other hand, these questions take more time to grade.  They require a specific rubric to determine how points are awarded and to define “quality” and next steps in learning.  The rubrics used require inter-rater and intra-rater reliability:  consistency between multiple graders (when applicable) and consistency between the same grader.  It is important that a teacher be very consistent with the way the rubric is used to grade all students both within a prompt from the same date, as well as a prompt from dates in the future.  The rubrics need to be descriptive enough that less than full credit should point out misconceptions, and next steps.

It is important to realize that constructed response questions may not be appropriate for all ages or subjects.

3)  Performance Based Questions:  The performance based questions ask students to demonstrate understanding by performing a task or creating a product.  Examples Include:  Singing a piece of music, creating a model, demonstrating a food preparation technique.

Considerations when using Performance Based questions:

The performance questions may not take as much time to write as a selected response question as you don’t have to create possible wrong answers, however they will take more time to grade and administer.  These assessments require a specific rubric to determine how points are awarded and to define “quality” and next steps in learning.  The rubrics used require inter-rater and intra-rater reliability:  consistency between multiple graders (when applicable) and consistency between the same grader.  It is important that a teacher be very consistent with the way the rubric is used to grade all students both within a task from the same date, as well as a task from dates in the future.  The rubrics need to be descriptive enough that less than full credit should point out misconceptions, and next steps.

It is important to realize that constructed response questions may not be appropriate for all ages or subjects.

Using the Same Exact Test Multiple Times

Can I use the exact same exact assessment questions multiple times to measure student growth?  Yes and No.  It depends on the purpose and how you use it.  Lets go through a couple of examples.  

a)  I am looking to see growth in math facts (memorization)

Yes, use the same test.  Using the exact same questions on a test will get at memorization and recall.  A skill that is centered around memorization and recall (such as sight words, vocabulary, math facts) could very easily use the same question.  Think about what you teach related to that skill.  If everything you teach is in the memorization/recall band, you will want a memorization/recall test.  However, if you teach apply, analyze, create and other levels…perhaps you want to create an assessment that reflects the whole range of your teaching.

b) I am asking students to play the same type of music, and I will be listening for improvement (using a rubric)

Yes, using the same rubric is fine….and necessary for comparable data.  Any time you have a series of performance tasks, the task itself may be different but you can use the same rubric to see how students are growing in elements of their performance.

c)  I have a math chapter test and I want to give them the same test at the start of the quarter (pre-test & post-test are identical)

No, this is not a good idea for a couple of reasons.  First: Students will likely remember that assessment, even if you don’t go over the answers that will likely effect your data.  Second:  Pre-tests are GREAT teaching tools!!  When a student sees a question and they don’t know the answer, you have just accessed a great teachable moment.  It would be a shame to pass that up because you were using the exact same question later.