Beyond the Bubble: A New Generation of Historical Thinking Assessments

>>My name is Mike [inaudible]. I work with Kathy in the Education
Outreach at the Library of Congress. We are thrilled to welcome you to the second
annual online conference for educators. Our session today, Beyond the Bubble, a New Generation of Historical Thinking
Assessments, will be recorded, as Kathy said. Without further adieu, I’d like
to introduce our presenter. Joel Breakstone co-directs the Teaching with
Primary Sources program at Stanford University and directs the Stanford
History Education Group. Along with Mark Smith and Sam
Wineburg, he led the development of the group’s assessment
website, Beyond the Bubble. He received his PhD from Stanford
Graduate School of Education. Before coming to Stanford, he taught
high school history in Vermont. Welcome, Joel, and take it away.>>Thanks, Michael, I really appreciate it. Good evening, everybody. It’s a pleasure to be joining
you here virtually. I’m looking forward to the session. And what we hope to do, I hope to do, during
the course of this session is to introduce you to the materials that we’ve made
available on Beyond the Bubble, and also to talk about how you
might use them in your classrooms. My background is I was a
high school history teacher in the per block before I begin work is
different because the history teacher in Bradford, Vermont, before
I began work at Stanford. And when I was a history teacher in Vermont,
I had a lot of questions about how best to have my students engage
in historical inquiry. How could I have them understand that history
wasn’t a single narrative, but it was a series of competing accounts, and that students needed
to wrestle with different pieces of evidence and try and make sense of
what happened in the past. But I often found that that
was really hard to do. My students weren’t always
ready to read historical text. And it was often challenging for me as
a teacher to locate those materials. And those questions were ultimately what led me
to go to graduate school at Stanford and to work with people at the Stanford
History Education Group. And when I was arriving at the
Stanford History Education Group, the group had just released a new curriculum,
called the Reading Like a Historian Curriculum. This is a screen capture of our
old website. In the bottom right-hand corner
you can see the link to curriculum. And so in the fall of 2009, we released
the Reading Like a Historian curriculum, which at that time was composed of
75 US history lessons which focused on having students engage in historical inquiry. To ask them to read like a historian. To use the same historical
thinking skills that historians use when they engage with historical text. To have students consider
the source of the documents. Who wrote it, when, why, and for what purpose? And to think about the context
in which documents were created. And to corroborate across
multiple documents and to look for points of similarity and disagreement. And to read closely to look at, what were
the words that the author chose to use, what words were left out, and
what content was left out. And the curriculum proved to be
popular and we got a lot of emails. The most frequent request was — or question
was, where is the world’s history materials? Since then, we have worked
to address that shortcoming and now have world history materials as well. But the second most frequent question
was, where are the assessments? If I’m going to emphasize these
historical thinking skills in my classroom, if I’m going to ask my students to source
documents and to corroborate documents and to place them into context, how do I know
whether or not they can actually do that? And we didn’t have a really
good answer for those emailers. And at the same time, there was
a much broader conversation going on about a need for new types of assessments. In 2009, President Barack Obama said,
I’m calling on our nation’s governors and state education chiefs to
develop standards and assessments that don’t simply measure whether
students can fill in a bubble on a test, but whether they possess 21st-century skills
like problem-solving and critical thinking. And he said that and we said,
yes, we absolutely agree. Where are those assessments, especially
if you are a social studies teacher? Where do you find the types of assessments? And we didn’t have a clear answer. What we were interested in doing
though was to create new types of forms of assessments, in particular. We thought that there was a need for new types of assessments moving beyond multiple-choice
questions and also types of assessments that teachers could use for
formative assessments. And what I’d like to do is just
take a moment to talk briefly about what your thoughts are
about formative assessment. So when you hear the term “formative
assessment,” what comes to mind? So seeing some of these initial responses. So the notion of having quick informal checks on student understanding is certainly
a big piece of formative assessment. End process assessment, so trying to
gauge student learning along the way. During the course of instruction or unit,
the types of assessments you want to do in preparation for a summative assessment. And I think that that comment
is also really important. About that ultimately formative assessment is
used to inform our instruction as teachers. That we can gather data from
formative assessments to shape what we do next as teachers. And to inform students about what
they need to do, to do better, right. If we have a clear goal set out, how do we help
the students to understand what they need to do to ensure that they can meet
that end-product goal? And there is an enormous amount
of material out there teaching — talking about formative assessments. If you go on to Amazon, you can find no
shortage of books and instructional guides about making assessment happen
in your classrooms. But if you’re a history social studies
teacher, there aren’t a lot of materials that are providing actual
assessments to use formatively. And that’s what struck us. When we — and we weren’t the only
ones who were struck by the lack of strong formative assessment materials. Former Secretary of Education, Arne
Duncan, wrote, not enough is being done to use high-quality formative
assessments to inform instruction in the classroom on a daily basis. Too often teachers have been on their
own to pull these tools together. And we’ve seen in the data, the quality of
formative tools has been all over the place. So we were interested in creating assessments
that focused on historical thinking skills and could be used for formative purposes. So we were wanting to make
historical thinking assessments. And we wanted to make materials
that could be used formatively. At the same time, we joined the Library of
Congress’s Teaching with Primary Sources. We knew that we wanted the materials that would
be the basis for these assessments be drawn from the world’s largest
library, the Library of Congress. But what we wanted to do was to have
an idea of what were the assessments that were already available to
history social studies teachers. And we really saw them as existing
on a continuum of complexity. At one end of the spectrum
are multiple-choice questions. Which are easy to give and easy to score,
but they’re not the best tool necessarily for measuring all the types of ways of thinking
we’d want our students to be able to do. So at the other end of the spectrum
are document-based questions. Which are really rich tasks that asks
students to read a series of primary documents and to use them to construct an analytic
essay in the course of usually about an hour. Their complexity though can also
make them slightly imprecise. Sometimes it can be hard to know why
a student who struggled on a DBQ. In addition, they’re not the best tool for
formative assessment because they take a while for students to complete and even
more time for teachers to score. Similarly, multiple-choice questions
are not always the best formative tool because they don’t give us a lot
of feedback about student thinking. And so we wanted to try to fill this middle
range of trying to make something more complex than a multiple-choice question
but less time-consuming than a DBQ. Something that could be used more readily
in a history classroom on a daily basis to give teachers feedback, as we’ve describe,
that could inform instruction and could be used as a gauge of learning in process. And so rather than describing those in
more detail, I’d like to show you one of the assessments that we have made. This is an assessment that focuses very
specifically on the skill of sourcing, and particularly whether or not students will
pay attention to the date of the document. So this image, present students
with this painting, which the title of it is “The
First Thanksgiving, 1621.” It is from 1932. And the author is — or rather
the artist is JLG Ferris. And the question is, this painting helps
historians understand the relationship between the Wampanoag and
the Pilgrim settlers in 1621. Do you agree or disagree? Briefly support your answer. What do you think would be some common student
mistakes if they were given this assessment? What would be common students mistakes you
think your students might make in responding to this assessment that focuses on
the date of the document and whether or not students will attend to the dates? Yeah, so these responses are right on target. Point out what are some of the most common
types of responses that we would see. A lot of students will repeat back content that
they have learned previously and really focus, more than anything, on the content of the
document, rather than noticing the date at all. And so this is the most common
student response we’ve seen. We have now piloted this with
thousands of students and this type of response is the most frequent. A student wrote “true,” because you can see
how they’re interacting with each other. Without any picture, you couldn’t
really see how Wampanoag Indians and the Puritans interacted with each. Essentially, a picture is worth 1000 words. If you didn’t have this picture, how could
you know anything about what happened between the Pilgrims and the Wampanoag? In addition, we see students who have
a very different type of response. This student wrote, as soon as the
settlers arrived, there’s mass curiosity which turned into violence and hatred. There is ever such a party
between the two peoples. They couldn’t even understand each other. So clearly this student has a more critical
vision of American history and has a notion that there was conflict between
the Native Americans and Europeans. But really there are in many ways,
the two responses are very similar. Because in both cases the student
has ignored a crucial detail, which is the date of the document, and has not
acknowledged some of the limitations that exist with using a document that was created more than
three centuries after the event that it depicts. At this point, sometimes people ask, well,
why not just make a multiple-choice question? What is the value-added of having
students write short responses as they complete one of these task? And what we think is that the short
responses, even a couple of sentences, provide really useful information for us as
teachers to inform our next steps in teaching, and gives us really good detail —
detailed information about student thinking. So, for instance, this student,
they rejected the sources. So if this was a multiple-choice question,
this student would’ve gotten full credit, because they have said it
was not a useful source. But there are some issues in the reasoning. So the student wrote, painted
by a white person from Europe, of course they would be extremely biased in
order to bring to light a highly-positive image of the Europeans as to excite
more foreign expeditions, because Indians looks so friendly and thankful. So on the one hand, the student
has made some assumptions. Assuming that JLG Ferris must
be a white person from Europe. And also appears not to have paid attention to
the date of the document, given that they seem to think that in 1932, there are efforts
to “excite more foreign expeditions.” And so that’s problematic. However, on the other hand, the
student has done something really good, because they at least are paying attention
to the fact that the author of the document or the artist of this document
influences the content of the document. So if this is my student, I can say, I’m
really glad that you’re thinking about the way in which the author influences the
content of the document, however, we also need to think carefully about when the
document was created and we want to make sure that we don’t jump to conclusions about
authors of documents without more information. And so that way, it really could
serve a more formative purpose. We also see some really polished responses. This was a response actually
written by an eighth grader. And she wrote, this painting was drawn
311 years after the actual even happened. There’s no evidence of historical accuracy, as we do not know if the artist
did research before painting this, or if he just drew what is a
stereotypical Pilgrim and Indian painting. This is a really nice response, because
the student has done two things. Both identified the limitation of the — of using a document that’s created three
centuries later, but has also noted that if we knew more about what the
artist had done in preparing for the — to create this document, we might be better able to evaluate its usefulness
as a historical source. What’s striking is that it is written by an
eighth grader and it’s something we found over and over again is that there is not a clear
developmental trajectory for a student’s ability to engage in historical thinking. What we found, perhaps not surprisingly,
is students who have been taught to engage in historical thinking are able
to do so, and those who haven’t, don’t engage in historical thinking as often. We’ve seen fifth graders who’ve
constructed similar responses as this one, not quite as polished in terms of the prose, but accurately identifies the
issue associated with the date. In contrast, we’ve also worked with twelfth
graders who have taken AP US History who have not done quite as well, in part because that wasn’t the focus
of the class they were taking. But what students see is
that — what this shows is that students need this type
of direct instruction. This particular assessment focuses on sourcing,
other assessments focus on content knowledge. We have an assessment that is about the Civil
Rights Movement and it asks students to look at two different documents and
to reason about which came first. Others focus on contextualization. And others focus on corroboration. The assessments are all available
on In order to download PDFs of the assessments,
you need to create a free login for the website. The same login works for Beyond the
Bubble and also our other website, the Stanford History Education Group’s
main website,, to download Reading Like
a Historian lesson plans. And you can do that on the
lower half of the page. The assessments themselves are available
on the tab that says “assessments.” And when you click on that,
you’ll be taken to this page. The assessments are organized by type. So we have multiple versions of each assessment. So we have additional versions of this same
Thanksgiving assessment where there is a gap in time between the document
and the event that it depicts. And if you click on to the assessment,
you’re taken into the assessment page itself, where there is a description of the assessment,
there is a link to Common Core standards that are addressed by the particular
assessment for the social studies. And on the right-hand side
there is a navigation bar. There’s also a link to the PDF of the document,
and also a link to where the document lives in the digital archives of
the Library of Congress. For each of the assessments, there
is a short three-level rubric. Three is proficient, students have
absolutely gotten the answer correct. Two or emergent, is students have done
something good but it’s not all the way there. Or, one, basically they have missed the boat and
have engaged in a variety of common mistakes. And for each of those we have
links to sample student responses, the types of responses we see most frequently
when we have piloted those materials. And for those sample student responses,
there are these little blue thumbtacks, and if you roll over those thumbtacks, there are
short annotations about what we think is going on with that particular response. There is also a short streaming video that
goes along with each of the main assessments that talk about what we think is going on with this particular assessment and/or
how you might use it in your classrooms. Which — and then finally, there is also
a section where you can leave comments and you can also see what other teachers
have said about these particular tasks. The question though is about a
classroom implementation, or, what do I do with these things, these new tests. How might I go about using them in my classroom? And so it’s an excellent
question and it’s something that we have spent time thinking about. Because there are a variety of
ways in which it can be difficult to implement new types of
assessments in classrooms. One issue that we’ve seen come up is that,
as teachers, we often rush to rubrics. When we give an assessment to our students,
we think that we need to give a grade to it and we need to put a score into our grade book. And students often feel that as well. We spent a lot of time with teachers in
San Francisco Unified School District, that they began to use these materials. And students would ask if they
didn’t get a grade back, they’d say, well, what about that quiz yesterday? What was my grade on the quiz? And sometimes that gets in the way of what the
tasks actually tell us about student thinking or the information we could use to
inform our next steps as teachers. Similarly, they don’t always line
up with the curriculum demands of — that we are working under. And these types of assessments that
focus on more than just content, also historical thinking skills,
aren’t always perfectly aligned. And even if they are shorter than a DB2, if
you have 150 students in your course load, it still takes a substantial amount of time to
grade one or two sentences from 150 students. And so we’re looking for ways to use
these materials in a range of contexts. And so there are a few things that
we saw as we’ve worked with teachers where teachers have integrated these materials
in ways that we found particularly productive. So one is to introduce historical thinking. One teacher that we worked with gave
students the Thanksgiving assessment at the very beginning of the school year. And she was pretty confident that students would
ignore the date, and then she could use this as an introduction to the idea of sourcing. And so she gave this task in the first
week of school, and as she expected, almost everybody missed the date entirely. Instead of collecting those assessments and
grading them, she just had students hold onto them and asked students,
how many of you noticed the date? Very few students had. And she then said, look,
in this class, this year, we are going to really pay
attention to the dates of documents. We’re going to pay attention to
source information of documents. And that’s why it’s important. Because if you don’t, you might think that a document created 300 years
later was a really useful source. And that was an introduction
to the idea of sourcing with those students, not a grade in any fashion. But it also gave her some immediate information about what her students were
actually able to do. There was an immediate feedback about
their abilities to source documents. There’s also the possibility that we
can monitor student understanding. That we can get feedback on how well students
are using these — understanding these concepts. So, for instance, the same teacher who used that
Thanksgiving assessment during the first week of school, followed it up with this
assessment a couple of weeks later. And it is the same assessment, but it
uses this alternative image of the signing of the Mayflower Compact from 1859, more
than 230 years after the actual event. And it asks the same question. In this case, given that she had introduced
the idea and had gone over the response with students at the beginning of the
year, she then graded students responses on this particular task,
because she felt confident that at this point students should
have gained that understanding. And so it was a way to monitor that their
understanding of this topic going forward. And you are not limited by the
assessments that we have on our website. Our hope is that the teachers will create
their own versions of these assessments and that really they are shells that you
can fill in with other types of documents. The Thanksgiving assessment just requires
a document where there is a gap in time between the events — or the document
and the event that it depicts. Some of our favorite Library
of Congress websites to locate materials are the
Library’s homepage, The teachers’ page, it has really an
incredible array of materials as well that have been carefully curated for
teachers, really document sets, lesson plans. Some of the best materials that are available on the Library’s site have been
put together on the teachers’ page. And one of my favorite sites is, which had thousands of historic newspapers that are fully searchable
by keyword and by date and by location. And so you can then download
full PDFs of the newspaper page from that includes your search terms. A really fantastic resource to be able to
show students a whole range of documents from all across the United States. For world history teachers, the World Digital
Library, it’s an incredible resource that is in partnership with cultural institutions all
around the world with the Library of Congress. And so and it continues to
expand rather rapidly. So really great resources from all
across the world, and you can search for both time period and geographic location. And then a more recent addition
is, which is a partnership with WGBH in Boston. And it is an archive of public radio
and television materials, broadcasts. So really some outstanding
resources to locate there as well. And our hope is that teachers will be
able to use the assessments we’ve created as a jumping off place and create new
versions of these same types of assessments. And then the final way that we envision
these materials being used is as a way to approach broader historical topics. And we saw teachers do that with another one
of our assessments which focuses on this image. How many people are familiar with this image? And/or how many of you have a poster
of this image in your classroom? Lots of people, right. A really iconic image from the Great
Depression and from the Dust Bowl, this image Migrant Mother taken
by Dorothea Lange in 1936. And we wanted students to think about the
context in which the image was created. Because often we forget that there
is a photographer behind photographs. Often it becomes simply students see
photographs as windows into the past rather than as a product of a particular person. And Dorothea Lange was employed by the federal
government to build support for their programs. And we wanted students to think
about this image in that context. And so we presented students with
this image, the source information, the title Migrant Mother, 1936, by Lange, and
she took the photograph in Nipomo, California. And then we told students that the Resettlement
Administration was one of FDR’s agencies that helped people from the Dust Bowl. The agency hired photographer
Dorothea Lange to take pictures that would build support for its programs. This photograph was one of
several that Lange took of migrant farm worker Florence
Thompson and some of her seven children. Then we asked students a series of questions. First, why might the background
information above lead you to question the photograph’s reliability? Two, what is one feature of the photo that
might lead you to question its reliability? Explain your reasoning. And three, what is one thing that
you’d want to know about Dorothea Lange or how she took this photo to
better determine its reliability? This would help me determine the
reliability of the photo because: we provide students with this sentence starter. And so when we worked with teachers in San
Francisco, they found some immediate patterns in terms of feedback from
students based on their responses. Some student responses showed
some fundamental misunderstandings in terms of historical knowledge. This student said that the Dust Bowl
did not take place in California. The Dust Bowl was a Midwestern crisis
that destroyed countless acres of land and did not understand clearly
that many migrants ended up in California as a result of the Dust Bowl. Other students put together
more sophisticated responses. This student wrote, the photo was taken
in order to gain support for agencies that helped people in the Dust Bowl. This could mean that the photo is fake and only
made so that you feel sorry and want to help. This is like propaganda. A really nice response. Both indicating that there is a
possibility that the author’s — that the photographer’s motivation
influenced the content of the photograph, but also tempering the response so as not to
claim too much based on the evidence available. With the second question, we again see
some slightly problematic responses. This student wrote, Florence Thompson
shows her emotion in the photograph. She should have been happy about
moving away from the Dust Bowl. Again, a little problematic in
terms of misunderstanding the plight that faced many migrants once
they arrived in California. Other students had more sophisticated responses. One feature of the photo that lead me to question its reliability is the
way her hand is posed on her face and the way her children
are both hiding their faces. It looks posed and artificial. The photo doesn’t look candid
like a candid honest shot. Again, raising the possibility
that there might have been a way in which Lange had posed the photograph. What was striking though in using
the assessment in the classroom was that the teacher found many students were
ready to simply reject the assessment. They did not want to consider
possible limitations. They simply wanted to reject it entirely. Which the teacher found slightly troubling and
wanted students to think a little more carefully about the way in which, even if Lange might
have been paid to take this photograph, that it still provides useful evidence. And she complicated their
vision of the photograph by providing outtakes from the same session. So these are the additional images
that Lange took of Florence Thompson and her family in Nipomo, California. And asks them whether or not given this
additional information, that would shift the way that students thought about the images. A couple of students were willing to say
that this shifted their understanding. They were a little more willing to think
of the Migrant Mother as a useful source. But many students stuck to their guns
and said, nope, I think it’s all staged. Which the teacher was troubled by. And asked the students, how many of
you have ever posed for a photograph? And when they were reluctant to raise
their hand, she said, well, you know, all of you took school photographs, so
clearly you have posed for photographs. Is that not a useful piece of evidence about you at a particular time and
place, even if it is posed? And as the conversation went on, the students
and the teacher had engaged in a conversation about what makes a piece of evidence
useful and what makes a photograph useful or not in terms of historical evidence. And although I’m not sure that the teacher
was aware, but in the process, the teacher and the students had entered into a
conversation that is a historical debate. A historian, James Curtis, has
written about this image of Lange, and he has argued that Lange had
indeed staged this photograph. In contrast, Errol Morris, the documentary
filmmaker, has argued against it, and has challenged this notion that
indeed Lange had staged the photographs. And Errol Morris wrote, is the argument
that Lange should not have intended to produce any effect, none whatsoever, isn’t
the issue a deeper, far more problematic issue? Mainly, did the photographer
intent to deceive the viewer? Was he or she using photography
as sleight-of-hand? Is the problem intentions? But if so, how could one do
away with them altogether? We read correctly or incorrectly
a photographer’s intentions into every photograph we see. We also imagine the intentions
of the people in a photograph. We see intentions everywhere. A very deep historical question
being raised here. And in the process of this short assessment,
the teacher had brought his students into this much larger frame of wondering
about how to use photographs as evidence. So a different way of thinking
about these assessments. They could also be used on a unit test,
or as a exit ticket at the end of class. Our hope is that they are flexible and that
they could be used in a variety of contexts. And that as new types of assessments, students
are going to be practiced in using them and opportunities to understand what are these
types of questions and what do they look like and what our expectations are for the
way that they are going to answer them. Please note down our contact,
my contact information. We spend a great deal of our time sending
materials out into the wilds of the web, and we’d love to hear how
teachers are using these materials. And if you have any questions or comments about
them, please feel free to reach out to us. If you’re interested to know when new
materials are going live on Beyond the Bubble, the best place to find out is on
our Twitter or our Facebook pages. Sorry the @ symbol is blocked off
there on the sheg underscore Stanford. We don’t send out emails, but we
do post on Twitter and Facebook when we have new assessments go live. We have quite a few assessments that will
be going online in the coming months, so if you want to know when those
go on, check out those posts.

Leave a Reply

Your email address will not be published. Required fields are marked *