The Worked Example: Invitational Scholarship in Service of an Emerging Field

I'm live-blogging again from the Digital Media & Learning Conference: Diversifying Participation at UC San Diego. Here goes!

James Paul Gee is chairing this panel (with Sasha Barab and Valerie Shute), and will talk about a relatively new idea, using paradigmatic examples of how you think about what you do, and how you think this emerging field should work. So, the Worked Example is a true work in progress, or maybe better phrased as an invitation to create networks and clouds of people who discover they share some of their assumptions or adapt their assumptions. Check below for definition of "worked example" in bold, and visit Sasha's example here: http://ijlm.net/knowinganddoing/10.1162/ijlm.2009.0023

Partiicapants: Sasha Barab, Valerie Shute, James Paul Gee

Valerie Shute: My language is assessment and measurement, so I'm learning some of the nuances and language of digital media and learning. Our worked example works on a powerful approach, valid assessments that support learning. I'll be talking about Quest to Learn (Q2L), but it is applicable to DML projects that want to support learning, especially 21st century learning skills. Evidence-Centered Design (ECD)

What is: a worked example? What is our problem? What is our claim? What is evidence-centered design? What are our worked examples?

A worked example is a public display of methods of valuding and thinking about a specific problem. Proposed as an example of good work to promote debate about what such work might look like and future shapes of the area.The goal would not be for the proposed approach to become the accepted one, but for it to become fodder for new work and collaboration.

What is the problem, generally? A lot of recent research has richly described what kids are doing with digital media. How can we go beyond describing to actually assessing? We need to develop valid assessment so we can inform these constructs. The problem specifically: Quest to Learn is new. How can we effectively caputre the values and goals of Q2L's stakeholders so that we can develop an objectives/values model, and how can we accurately assess important competencies which Q2L claims it's teaching kids?

Two Q2L Research Strands

"The strength of the pack is the wolf, and the strength of the wolf is the pack." Rudyard Kipling. We are identifying and modeling the important stakeholders. What attributes of the students are important for success in the 21st century? Systems thinking, collaborating, resource-management skills.

ECD should be used for assessing Q2L (school and student levels) because it allows for the colleciton of qualitative and quantitative data.

ECD is one approach. Many constructs: systems thinking, culture of kindness, resource management. Systems thinking model includes a flow chart of dynamic thinking, closed-loop thinking, test the model, transfer the model, demonstrate agency, understand system elements and interrelations, understand basic system concepts, determine hypotheses, understand quality of relations, etc....

What is Evidence Centered Design? (Mislevy, Steinberg, & Almond, 2003). Assessment Design: Competency/Objectives Model (what do you want to say about the person/school?) Evidence Model: (what observations would provide best evidence for what you want to say?) Task/Action Model: (What kinds of tasks/situations let you make the necessary observations?)

Design & Diagnosis: Assessment Models & Metrics, and Monitor and Diagnose Success. Image shows flow of Comp/Objectives to evidence accumulation to evidence to evidence identificaiton to task/action. At any point in time, you would have information the kid at any point in time.

Our worked example: Q2L needs to be evaluated across various levels-- from the whole school to the individual student. The general aim of this research is to conduct a preliminary multi-level assessment of Q2L, setting up evidence-based models to assess the school, the data, the students.

Towards School Assessment: identify/model critical goals/values. This will provide a clear target toward which the school aims. Currently different sources of info and beliefs on goals/values but no consensual model. This iterative design effort involves stakeholder interviews, surveys, focus groups, observations.

School data: Interviews with Q2L stakeholders (teachers, game designers, curriculum designers, admins, N=16). Asked what is of value to Q2L (500+ pages of transcripts.) Focus groups (follow-up). After themes were induced from interviews, and survey for additional comments sent out/returned, focus groups further fleshed out the Q2L model. Observations: Bi-weekly classroom (etc.) observations, with 2 independent observers, because "actions <often> speak louder than words."

From interviews, we found six different values or objectives that emerged: culture (container in which all else resides, to support 21st century learning), systems thinking, design thinking, collaboration, resource management, pedagogy.

Culture: culture of innovation, culture of kindness (was talked about a lot), culture of meaning, culture of community. Design process, design aesthetics, respect, compassion, identity, reflexivity, safety, privacy, etc.

Systems thinking: thinking about the world that is very different than the norm, which is linear thinking.

Design thinking: tinkering, the idea of supporting the culture, and the ability to iterate, to pause and reflect, what did I get right, what did I not get right, persistence.

Collaboration: work together effectively with other people, teachers/teachers, teachers/designers, students/students, students/teachers.

Resource Mgt: what do I need to know, and how do I get that information? Collecting, evaluating, filtering information.

Pedagogy: instantiating a cohesive model of 21st century learning, that it is child-centered, interest-driven, everything is interactive, challenge-based, situated, etc.

Student assessment: 18 mos, 4 assessment periods, first completed (N=72).

Student variables: systems thinking: dyanmic thinking, closed0loop thinking, transfer model. Collaboration, cooperating, negotiating, influence. Resource mgt: making plans, etc.

RQ: does the incoming cohort of 6th grade students improve performance over 18 months for systems thinking, collab, and resource mgt? Do students at Q2L perform comparably to matched students (in other NYC public schools).

When we're done,  will have a model of important goals/values of the school (eg the culture it intends to foster, the kinds of new thinking and skills it wants its students to develop).

Summary: DML environments are intended to improve the lives of young people but good assessments are needed. ECD lets us create and use good (valid/reliable) assessments to syntheize info from disparate sources into a common framework for both school and student level analyses. (There was more, but I missed it)

Sasha Barab (Indiana University, Learning Sciences)  Worked Example: Invitational Scholarship in the Serivce of an Emerging Field.

To contextualize why worked example is relevant, learning scientist looks at theory and impact. What is, what could be. Observation, design. Understand, change. Verifiability, plausibility. Tension of wanting to observe the work, but also this desire to change it at the same time. A lot of what we do is design-based research, where there is some sort of problem, then we have a theory, then we design, and evolve a theory. If we find something happening or not happening, we revise. We are trying to make sense, because our theoretical models don't allow us to look at this in a lab.

Transformational Play as an example: a theory we are start to developing in our work. Playing transformationally involves taking on the role of a protagonists who must employ conceptual understandings to understand and, utlimately, make choices that have the potential to transform a problem-based fictional context. In contrast to what most curriculums look like. Consequentiality, intentionality, legitimacy, context, person, content. When I try to illuminate that to other people, I can put that in articles. Or I can have people to go to our website so that they can log-in to our 3D world, you engage as an avatar, look at the experience, have an experience, then ask us questions. I can lose you in a paper, when my persuasive argument doesn't engage you. But when you experience a worked example, you experience it and the persuasive argument is no longer something that occurs simply in text. If I try to immerse you in the world, it will affect what happens to you, but I want you to understand this theory of transformational play. Watch video, see what kids are doing -- it's a much more complicated story to use a worked example, and I can also invite you in to refute, ask questions, helped me evolve the theory. Take ideas that we have, that might be emergent, and have it sit somewhere between scholarly publication (or instead of?) and having you spend hundreds of hours to get the same experience. Put worked example between those two pillars.

Check out Sasha's worked example here: http://ijlm.net/knowinganddoing/10.1162/ijlm.2009.0023 or http://tinyurl.com/barabwe

He shows how Jim Gee has commented on the worked example, to give Sasha feedback, but also to share with greater field and network. Then shows how Henry Jenkins has commented, as well as others.

(Really compelling -- I have to admit I didn't fully understand what this meant until I saw Sasha's examples. It goes back to the idea of storytelling to describe what is happening with digital media and learning.)

Worked Example: Goal. Make happenings visible (not simply describe them), advance generalizable insights (not simply show example), share evolving manifestations (not simply ready-made products), reveal reasoning (not simply outcome findings), promote ongoing discussion (not simply reporting out), distribute meaning making (not simply printed manuscript), make plausibility arguments (not simply verifiability), increase accessibility (not simply insider community).

Worked Example: Features: underlying theoretical claim, assemblage of media, user data, author's meta interpretations, mechanism description, design iterations, contextual description.

Conclusions: the presentation both serves as a contextual instantiation of a theoretical conjecture and, by scaffolding discussion among peers around the example, invites verification or refutation of the conjecture.

* to advance plausible conjectures about contemporary society and culture, we must not simply present the context of justification but relate the context of discovery

* to present these contexts in an effective and compelling way, we must employ various media in the sharing of our work

*to invite the audience of scientific publication to participate in the context of discovery, we must employ not only expository modes of composition but also chronological exhibitions like narrative and process anlysis

* to engage a shared dialogue that advances scientific knowledge, we must select instances that may elicit or illuminate a class of phenomena

* to critically engage and evolve core issues, we must not only advance the verification of our claims but invite the critical involvement and dialogue of others.

-----------------------------------

James Paul Gee: this is not about winning, it's about mapping the field. The approach to assessment in Valerie's talk is where the military has "put their flag," then you can decide if that works for you. Sasha has now modeled one of the best models of digital play, but we want another set, we want a movement. We need more of these so we can see what else is tenable.

Discussion with audience! Too much too fast, moving over to twitter #dml2010.