Grading 2.0: Evaluation in the Digital Age
In this HASTAC forum, three Scholars invite you to consider evaluation and assessment in the face of new forms of digital media, new kinds of skills and technologies, and the ever-changing landscape of education and academia.
Grading 2.0: Evaluation in the Digital Age
As the educational and cultural climate changes in response to new technologies for creating and sharing information, educators have begun to ask if the current framework for assessing student work, standardized testing, and grading is incompatible with the way these students should be learning and the skills they need to acquire to compete in the information age. Many would agree that its time to expand the current notion of assessment and create new metrics, rubrics, and methods of measurement in order to ensure that all elements of the learning process are keeping pace with the ever-evolving world in which we live. This new framework for assessment might build off of currently accepted strategies and pedagogy, but also take into account new ideas about what learners should know to be successful and confident in all of their endeavors. 
How do we better align grading and assessment techniques so that they are more in line with how students learn today? The traditional 'teach to the test' evaluation paradigm continues to produce a classroom experience that focuses on specifically 'testable' results. That testing paradigm is also disconnected from all of the creative, production, remixing, and networking skills that students are developing through their everyday engagement with new media. Another issue is that the traditional assessment system tends to measure students individually and via multiple-choice and written-response questions. As teaching practices evolve to include more team-based projects that involve the use of smart tools to solve problems or communicate ideas, it will become increasingly difficult to assess students in the traditional ways. Furthermore, current widely-used tests are not designed to gauge how well students apply their knowledge to new situations.
In addition, how can digital media be used to develop new grading and assessment strategies? For example, last year, Cathy Davidson posted a blog entry here on HASTAC, called How to Crowdsource Grading. It outlined her course and the development of a new grading rubric, and student assessment techniques, to better work with her students in the classroom and in their course blog. The course was built around the idea of self-motivation, peer-to-peer review and a very clearly outlined contract for students on what each grade represented. This post attracted lots of commentary -- prompting her to write a response, called Crowdsourcing Authority in the Classroom. There are hundreds of other experiments such as these around the country, in elementary schools, high schools, college classrooms, corporate environments, and even feedback systems for nonprofits and municipalities. And there are more and more researchers thinking through these issues. There is clearly a great amount of interest in developing new technologies, and new forms of pedagogy, to better reflect grading, peer interaction and learning in the digital age -- help us think through these questions, experiments and strategies!
How to grade, assess, teach, learn and structure the learning experience for students in the digital age?
Many interesting projects are working on this question, and we invite you to share others with us below. For example:
- The Learning Record, a portfolio-based evaluation system designed to emphasize student learning, not product-based outcomes
- Nils Peterson and his colleagues at the Center for Teaching, Learning, & Technology (at Washington State University) have been working on developing new assessment strategies and forms of classroom engagement
- Pecha Kucha in the classroom - reframing the presentation from the unstructured long-form speech to the conversation-starting breakdown
- Digital Youth Research was a 3 year project to investigate how kids use technology and media in their everyday learning. They have reports available on their site, and the group recently published a book, Hanging Out, Messing Around, and Geeking Out.
- Re-mediating assessment, a blog considering participatory assessment models in education, authored by Daniel T. Hickey, Michelle Honeyford, and Jenna McWilliams (Indiana University).
- The DML Research Hub, funded by a MacArthur grant, is supporting two projects. One, lead by Mimi Ito, is called Distributed Learning Research Network, and works on distributed learning that happens in social environments. The other, lead by Joseph Kahne, is called Youth, New Media, and Public Participation Research Network, and investigates the ways that youth, through social and political participation in online communities, affects their capacity and motivation to engage in social and political issues.
- Cathy Davidson and David Theo Goldberg's report, The Future of Learning Institutions in a Digital Age (also available as a free PDF). The report found that students are learning in deeply collective and innovative ways, and that learning institutions - schools - have to keep up or risk obsolescence. They offer ten principles for redesigning learning institutions and pedagogical systems to better reflect the way students learn today. The book-length version of the project, The Future of Thinking: Learning Institutions in a Digital Age will be coming out in 2010.
We invite you to join us as we discuss:
1. Technology & Assessment:
- How can educators leverage the affordances of digital media to create more time-efficient, intelligent, and effective assessment models?
- How can we use new technologies and new understandings of mind and cognition to help us build better metrics, rubrics, etc.?
- What can emerging technologies teach us about evaluation methods? What would the perfect evaluation tool for user-generated content look like?
2. Assignments & Pedagogy: 
- How can we develop assignments, projects, classroom experiences, and syllabi that reflect these changes in technology and skills?
- What does it mean to design a course that takes seriously the idea that learning can happen through these digital technologies, new models for grading & evaluation, and new media skills?
- How can we prepare our students for the kind of social, global, collaborative work in many of todays professional work environments?
- Of course, developing and implementing these strategies takes a great deal of time and effort on the teacher's part. With ever-increasing stresses on teachers, professors and departments, how can we support innovators who are already crunched for time and energy?
3. Can everything be graded?
- How important is creativity, and how do we deal with subjective concepts in an objective way, in evaluation?
4. Assessing the assessment strategies:
- How do we evaluate the new assessment models that we create?
- What is consistent about all of these forms of evaluation? What are the constants in evaluation and grading?
- The huge problem of unequal access to technology and digital literacy needs to be considered - how do we account for these differences within our classrooms, schools, and countries?
What are your strategies and experiments?
We're thrilled that HASTAC and the Scholars program are supporting these types of peer-to-peer conversations and are providing the online space to collectively share our strategies, ideas and experiences so that we might learn from each other.
A special welcome to the members of the Office of Assessment and Innovation at WSU who will be joining us in the comments!
HASTAC Scholar Discussion Hosts:
- John Jones, Ph.D. student, University of Texas - Austin
- Dixie Ching, Ph.D. student, New York University
- Matt Straus, Math/Statistics major, Duke University
About HASTAC Scholars program
Find Scholars
Upcoming Group Events
-
February 3, 2012 - 7:00pm - 9:00pm
-
February 15, 2012 - 11:00am - 1:00pm
-
June 25, 2012 (All day) - June 26, 2012 (All day)
Assignments and Pedagogy
One of the concerns I have as a learning designer is how to prompt authentic work rather than pseudo-authentic work. The potential is there to invite students to find authentic work in internet communities but it seems that too often educators default to something that they can be confident in assessing themselves. We all do this. We ask the questions that we already know the answers to or at least the general type of answer we would like to see. And so when we start asking students to do authentic work in real communities online, often I think they can sense the falseness of this, the "creepy treehouse" quality if we're not willing to let go and let them investigate things we don’t fully understand ourselves. Suddenly, we can start to worry, what use am I as the teacher? How can I be confident in my new role as guide? Because even if we're willing to move away from sage on the stage, I sense there is reluctance to no longer being the one who holds together the center and direction of the course. I've seen students become completely revved up when they get their hands on an authentic question but I've also seen their suspicion when they realize that the instructor is looking for a given answer.
"Creepy Treehouse" -- what a great way to put it!
Hi Jayme,
Sorry for the late reply... just now catching up on all the great conversations that went on today. Loved the "creepy treehouse" concept! Everyone needs to read the original post, but the gist is this:
<snip>
creepy treehouse
n. A place, physical or virtual (e.g. online), built by adults with the intention of luring in kids.
Example: “Kids … can see a [creepy treehouse] a mile away and generally do a good job in avoiding them.” John Krutsch in Are You Building a Creepy Treehouse?”
n. Any institutionally-created, operated, or controlled environment in which participants are lured in either by mimicking pre-existing open or naturally formed environments, or by force, through a system of punishments or rewards.
Such institutional environments are often seen as more artificial in their construction and usage, and typically compete with pre-existing systems, environments, or applications. creepy treehouses also have an aspect of closed-ness, where activity within is hidden from the outside world, and may not be easily transferred from the environment by the participants.
n. Any system or environment that repulses a target user due to it’s closeness to or representation of an oppressive or overbearing institution.
n. A situation in which an authority figure or an institutional power forces those below him/her into social or quasi-social situations.
With respect to education, Utah Valley University student Tyrel Kelsey describes, “creepy treehouse is what a professor can create by requiring his students to interact with him on a medium other than the class room tools. [E.g.] requiring students to follow him/her on peer networking sites such as Twitter or Facebook.”
adj. Repulsiveness arising from institutional mimicry or emulation of pre-existing community-driven environments or systems.
Example: “Blackboard Sync is soooo creepy treehouse.” Marc Hugentobler
<snip>
-So, closed-system knock-offs are clearly not the answer. But it seems as if the well-meaning professor who tries to "go along with what the cool kids are doing" can fare just as poorly (unless you're KSU's Michael Wesch). I suppose it comes down to whether you really understand the medium (and the message!)
Anyway, thanks for your insightful post. What a great way to kick off the forum! :)
creepy treehouse v. sandbox v. exposure to the world
I am totally not going to defend the creepy treehouse. But I closed the online component of my course to the world (still allowing guests to log in w/ a guest username and pw) this semester amid concerns about FERPA and some student's objection to writing in public. Some students seemed to want to construct the same artificial boundaries in the online space that they were used to in more traditional classrooms. In other words, they wanted a sandbox, not a live site. As one student put it "I don't want people to google me and find dorky things like rhetoric." Has anyone else experienced resistance to using open Web 2.0 tools?
I'm torn about what to do next semester. Either I keep it closed, or I make students sign something that says they must be willing to write online, or I let them build their own treehouses. I like the latter idea best, but I'm wondering, then, how to foster any sense of collective classroom cohesion. If they are all playing in their own treehouses, doesn't that create a whole other set of problems?
Strangely, students also decided, as a class, that they wanted to do a facebook fan page for their collaborative class project.
It all depends I guess!
Great point Will. I feel like you're doing the absolutely right thing by negotiating this issue *with your students*. As you can see, creepy treehouse is very much a perception thing and there are no hard and fast rules in terms of what makes sense to students, gives them a "safe space" to experiment with their writing and thinking. Sometime a walled garden is best, sometimes they want to broadcast to the world using Facebook. Anyway, I guess I'm not really offering any answers here; please let us know what happens next semester!
Hybrids?
Yeah. Next semester, I will let the whole HASTAC know. As I suggested in my other comment (the one about teaching FAIL), I am learning a lot in those moments where I am forced to stammer "Uh...good question...I'm not sure of the answer." (Those moments are frequent this semester.)
I am leaning toward a hybrid approach for next semester: make students create their own online workspace and then use the course site as a place where we come together. I'm not sure about permissions...if the students want to only invite classmates and the instructor (required) but not the world (ok) then it could get complicated with a score of different blogs/wikis/CMSs. Has anyone tried the hybrid approach? Is there a way to keep it simple?
Hub and Spoke Model
Will,
We have used the term Hub and Spoke for the model you just labled "hybrid." The thought is that students should be working in their own space(s) as appropriate to their community of practice. The course hub is a place that the students all pull together, and post a link to where they work, have meta-discussion of their work, seek peer assessment, etc. The work takes place outside. The access to the Hub would be controlled by the University, the access to the spokes would be controlled by the student (and is often public). The post explores the rationale of using SharePoint as a platform rather than Blackboard. Today I might have less focus on SharePoint than when it was written.
Now the challenge. If students are working in spokes all over the Internet, how does the assessment get done and brought back to the university. Hence our invention of the Harvesting Gradebook concept.
Lots of love for the hub and spoke model
Thanks for the hub-and-spoke link. That is a super-helpful way to think about it. Glad to know such a model has been vetted.
I'll probably use drupal for the hub--that's what we've been using for everything this semester--and point them to a handful of tools for the spokes. Do y'all train them in the spoke tools? Or just let them use/figure out whatever tools they want to use?
I suspect for assesment, the Learning Record that has been mentioned around here will be really useful. But I'm interested in the Harversting Gradebook, too.
From the Creepy Tree House
Reading your post, Will, made think about the creepy treehouse from another perspective, and from that perspective how much the classroom is an eerie fort.
But yes, we've also discovered a similar phenomon in our work where we have invited professionals to give feedback to students (in a professional program). We have come to understand that many of students have not been adequately prepared to appreciate that kind of opportunity (even when the outcome of the activity might turn into a job interview). That experience helped us understand how the challenges an individual instructor faces might be ameliorated by working with colleagues to scaffold a more coherent curriculum in which such opportunities are not unusual. As long as you're the only one in your program moving outside of the classroom fort, I would guess you will have students who continue to believe that's the only "dorky" place where rhetoric matters.
Unique is the new normal
Yeah, Gary, when I read "working with colleagues to scaffold a more coherent curriculum in which such opportunities are not unusual" I was like "That's what I'm talking about, people!" And that's what we are trying to do in the DWRL. When my students ask "Why aren't we just using Blackboard?" (i.e. why do we have to learn another set of interfaces) I tell them we are trying to learn, as a group, what other opportunities exist.
FERPA SCHMERPA
Read FERPA. It prohibits the University from disclosing certain information about students. It does not preclude a theater class that does a public show. Or Marching Band class. Or service learning in a class.
Being forced to act in a public creepy treehouse could be objectionable to students, but that should also reflect poorly on the instructor and institution.
FERPA Schmerpa indeed
Yeah. I think my FERPA fears were largely unfounded and uninformed. They came out of a THATCamp discussion we had a while back. You know, I am teaching with all these Web 2.0 tools for the first time (although I've been using them for a decade) and someone threw up a red flag and another person threw up another red flag and i was just like "fine, I'll close the door like a real classroom." Plus, my students are, in some cases, using copyrighted content. So there's the whole fair use thread we could open up too. Personally, I just had to narrow the number of variables I was dealing with. But I'm really excited about the hub-and-spoke system for next semester. I think that's going to solve a ton of problems.
Has HASTAC done forums on the legal and legislative issues of Web 2.0? (Sorry, I'm also a HASTAC n00b this semester.) It seems like that would be useful. Off the top of my head, i can imagine threads on FERPA, Fair Use, Copyright/Copyleft, Campaign Finance Reform, etc.
Are you saying students' objections should reflect poorly on the instructor? I'm not sure I understand.
Creepy Treehouse is Creepy
I was trying to say that having students working in public in a creepy treehouse is creepy and reflects poorly on instructor and institution. Having students working on real authentic stuff in public reflects well.
Students may object to working in public, but the challenge is to help them appreciate that real performance before a real audience is what its really about. Imagine the member of the Marching Band class that didn't want to march at the Homecoming football game. Students get that Marching Band is about performing in public. In your class it should be as obvious why they are working in pubic.
The Pen name
Not exactly a solution to your situation, Will; however, I've found that asking students to adopt pen names (and explain their choices) is a productive excercise for multiauthored blogging throughout the quarter/semester. It gives them a space to work through the notion of writing for a public and how, of course, identify formation is involved in that process. That, and it's fun to create a pen name! I generally use WordPress as the platform (one class blog with many authors).
(BTW: We should chat at some point about your work on audio recording. I'm doing similar stuff in my dissertation, which is a cultural history of magnetic recording, especially audio recording.)
legal/ethical issues of web 2.0 -- some resources
Hi Will,
Concerning "legal and legislative issues of Web 2.0," a few resources come to mind:
"The new digital media are a frontier rich with opportunities and risks, particularly for young people. Through digital technologies, young people are participating in a range of activities, including social networking, blogging, vlogging, gaming, instant messaging, downloading music and other content, uploading and sharing their own creations, and collaborating with others in various ways. In this paper, we explore the ethical fault lines that are raised by such digital pursuits. We argue that five key issues are at stake in the new media, including identity, privacy, ownership and authorship, credibility, and participation. Drawing on evidence from informant interviews, emerging scholarship on new media, and theoretical insights from psychology, sociology, political science, and cultural studies, we explore the ways in which youth may be redefining identity, privacy, ownership, credibility, & participation as they engage with the new digital media. For each issue, we describe and compare offline and online understandings and then explore the particular ethical promises and perils that surface online."
Evaluation Ethics
Though the book is about program evaluation, I recommend Evaluation Ethics for Best Practice: Cases and Commentaries, edited by Michael Morris. The principles pertain.
What I am doing is having a
What I am doing is having a public weblog where I can do normal blog posts (notes to class, interesting links, etc) and copies of handouts/schedule/syllabus/etc. and then using Blackboard (ugh) for more private writing.
Can everything be graded?
I'm a little worried about the use of the term "creativity" since it covers so much and tends to get lumped in primarily with the arts as a soft and fuzzy skill that's hard to quantify but should, you know, be promoted in the "hard disciplines" so that people can be well rounded. My background is in fine arts and I do believe that creativity can be disciplined and, yes, assessed. But the question here seems to be, "Do we REALLY have to assess everything?" And as a supposed assessment specialist just let me say, "God, I hope not -- just shoot me now." Recently I've become very interested in ideas about play and have read Stuart Brown’s book on Play. I find I need to play myself in order to keep my sanity (evidence) but even here people offer their unsolicited assessment of my work and its artistic merits. Sometimes I just want to say, "This isn’t serious; there needs to be a space left where a person can play without worrying about being evaluated." It’s critical to remaining innovative. But how do we promote that space without ruining it?
Feeding the beast
The word "assessor" historically meant someone who assisted the judge, which reminds me to think about who is the judge? If it is the students and we are assisting them in making a judgement about their work, then the assessment is "assessment FOR learning" (as opposed to "assessment OF learning," which I call "feeding the beast"). While in the FOR mode, it would be good to have educational use of a timeline replay button, as in the new google WAVE; and also to enlist increasingly powerful automated text analyses, and advanced visualizations as seen in the National Center for Supercomputing Applications Automated Learning Group.
Feeding the beast with "assessment OF learning" means shaking out a grade, putting a summary mark of some kind that aggregates all the process and product into a normative and comparative symbol. The beast, in various cases, is the certification review board, state board of education, college, department, school, major professor, teacher...any other entity outside of the learner. I think of those outside audiences in two clusters: trusted others such as mentors and advisors selected by and honored by the learner, and all others, including the institutional "community of practice" of a profession, culture, etc. These beasts must be fed by someone, usually not the learner except in the case of the immediate teacher, but sometimes on their behalf. Unfortunately, many times feeding the beast causes the one doing the feeding to ignore or forget the learner (perhaps unintentionally) through aggregation and summarization. Learners like rich details in feedback about singular items; beasts like summaries and large "n" sizes. We should try to avoid calling both processes by one name, since they are so very different in purposes, appropriate methods, and aesthetics.
One concern I have about "assessing process" is its impossibility in many important contexts. For example, the process that unfolds a piece of improvizational music has to be directly experienced by both the creator and listener (could be both roles in one person or different people) in order to be taken in and understood, and no amount of translation or summary can then provide other people with that same experience, if they missed the performance. "Assessing" the work, in this case, seems to be essentially expressing one's opinion about the totality of the lived experience at some later time, after short term memory begins to fade and the details are a shadowy reverberation. Thinking about both the FOR and OF learning stances, I'm not sure either one can effectively apply. Perhaps this dilemma also arises in "self-assessment" which is another story altogether (e.g. can one assess one's self FOR as well as OF learning? Are these two different metacognitive skills?).
Art Crits are the Toughest Evaluation Ever!
Hi, Jayme, Just FB'ing with a friend in the arts and we were exchanging stories about devastating Art Crits we'd witnessed, where the art "mentor" goes from painting to painting (or whatever the medium) and delivers a public, sometimes devastating evaluation. Thought I would just get this in here to break up any implicit binary between creative/non-quantitative/soft/touchfeely/arts on the one hand and quantitative/hard/tough/science-technology on the other. To face the art world, art reviewers, and those Crits, you have to be tough as nails. I know that's not what you are referring to but I just wanted to make it clear that non-standardized does not necessarily equal "easy" or "freewheeling."
Teachers v. Guides
Jayme, I totally agree: as instructors we have to allow students autonomy to follow interesting questions wherever they go. In my experience, I've found that one way to facilitate this kind of freedom is to let students know that it's ok to fail. That is, if as an instructor I'm going to encourage my students to be adventurous, I have to let them feel free to follow a train of thought or project even if it ends up not working out. Otherwise, as you point out, the whole process seems like I'm forcing them to a pre-determined conclusion.
Failing
Hi John. I made the mistake of not writing directly in the comments box so I've been busy correcting all the dropped punctuation in my posts. Lesson learned.
I agree about telling students it's ok to fail but I find sometimes that students don't really believe that the instructor is serious about only evaluating them on process. They are understandably suspicious. Sometimes it can take the better part of the semester for them to stop looking for the instructor's approval of their products.
"Fail like you mean it" (or, some wise words from Dean Kamen)
Hey John,
Your comment made me go find this Dean Kamen video I watched as part of preparing/thinking about the forum prompt. Wise words, no? :)
Pay attention to your mistakes (Teaching FAIL)
I love this thread. I was thinking about suggesting a whole forum called "Teaching FAIL." I've been compiling some material to that end. Here's an excerpt from the audio version of Jonah Lehrer's book How We Decide. In it, he mentions the work of Carol Dweck. Dweck's work in developmental psychology focuses on <gross reduction> what happens when you tell kids they are smart versus what happens when you tell them they tried hard </gross reduction>. Paraphrasing Niels Bohr, Lehrer suggests that "expertise is the wisdom that emerges from cellular error." I initially wanted to bracket out the "cellular" part. But that might be the whole point. Error is felt in/by the body, which is what makes it so powerful. In some ways, it brings pathos back into a longtime logocentric discussion.
"expertise is the wisdom that
"expertise is the wisdom that emerges from cellular error"
I like that.
Thanks for mentioning Dweck
Hi again Will,
Yes, love that quote as well! Also, thanks for mentioning the importance of Dweck's work as it relates to the whole enterprise of learning. I'm paraphrasing as well here, but her research has shown the importance of adopting an “incrementalist” view of intelligence, in which aptitude is viewed as something malleable (as opposed to “fixed” or a “born trait) and dependent on variables like time-on-task. Students who are shown to hold an incrementalist viewpoint tend to put greater effort towards learning because they believe their efforts can change their intellectual abilities. So as you imply above, it's more beneficial for students if they are praised for persistence and effort, an evaluation paradigm that allows for plenty of "cellular error."
WillBurdette likes this
Not to flood the place with the greatest hits of pop writers, but the time-on-task thing is like Malcolm Gladwell's 10,000 hours to expertise idea, right? (I think I've almost racked up the 10,000 hours needed for the dilettante merit badge.) Like Tiger Woods isn't Tiger Woods just because he was born athletic. He's Tiger Woods because his dad made him practice all the time and would do things like drop his golf bad right when Tiger was in the middle of his swing. Every incremental stroke is part of a larger career that is shaped by those individual strokes.
funny coincidence
What a funny coincidence. I write a quick post about Dweck's work and the cut-and-dry obvious superiority of supporting/inculcating an incrementalist theory of intelligence in students, then go to a brown bag at NYU given by a post-doc in my lab named Paul O'Keefe, where Paul proceeds to convince us how it's really not that simple. Possessing an "entity theory of intelligence" (the belief that intelligence is fixed) may actually be more appropriate depending on the activity goal... well, at least that's his early inklings. I'll try to get him to post something... maybe as a separate blog post, since we're getting a bit off topic... stay tuned!
Hmmmmmm
I wonder what "depending on the activity goal" means...do tell if you have more information.
On Error
Good stuff, Will. I've also been thinking through ways to embrace error (as an opportunity for new learning) in humanities courses. Have you tried this at all? Like, say, through some particular exercises or outcomes? My courses tend to be process-obsessed; however, I haven't found an effective way to have everyone involved isolate what was at one point an "error" and articulate how that error became an opportunity. For instance, rhetorics of "revision" tend to focus on the fix, not the opportunity.
Regardless, thanks for all of your comments here!
Thanks Dixie for posting the
Thanks Dixie for posting the link to Kamen's video.
It brings together for me some of the central issues on this forum: creativity, learning, and assessment. Kamen discusses how often people begin without a map, make many mistakes, but then when they finally get a route, they just pass along the map. The process (the recap of mistakes) are either erased or else presented as watered down anecdotes along the way. Many of us in writing studies see this same phenomenon. Teachers want their students to write well (not unreasonable) but often this want turns into a focus on what the student considers to be his or her final map. There’s no discussion of the process along the way and little value in taking chances or failing.
Another video clip on creativity is this one by Ken Robinson on TED. He makes the argument that schools kill creativity.
Also, he talks about how the unpredictability of the future makes it extremely difficult to figure out what to teach let alone what to access. While he stresses that being wrong is of course not the same as being creative, Robinson is convinced that fear of failure prevents creativity. I tend to agree.
On Fear of Failure
Thanks, Patrick. I like your emphasis (via Robinson) on the fear of failure. To add to this discussion, what role do you think (for a lack of a better term) "ownership" of the course material plays in a fear of failure? While I can't generalize, I would say that one thing I like very much about project-based curricula is how students tend to become the experts in their project development moreso than the instructor, especially when the projects mobilize competences that the students already have (likely unbeknownst to the instructor).
Along these lines and re: assessment, projects lend themselves to persuading someone's instructor, peers, and target audience(s), rather than being accountable for content produced by authorities on the topic (e.g., a scholar of computers and composition).
In other words, I'm wondering if it's a fear of failure that prevents (or curbs) creativity, or if it's the assumptions about expertise and authority that subtend content-based curricula. Could be both, perhaps. What do you think? I ask because I'm looking for models that are less focused on the student and more focused on the institutionalization of learning and the culture of instruction in education.
Appreciate your time!
Failure, Expertise, Ownership, and Creativity
Interesting question Jentery . . . I think fear of failure and assumptions about expertise and authority do play a role in curbing creativity. I really think both do.
You got me thinking about the relationships between failure and ownership and failure and expertise. Can one exist without the other? And in what ways can both curb creativity? And perhaps more pointedly, are their productive ways where these notions can work together?
It strikes me that notions of failure and ownership are closely connected, maybe even inseparable. On the one hand, I can think about a (bad) model of learning where ownership is exclusively in the hand of the teacher (or the institution). In such cases we might see the classic case of the disengaged student attempting to give the teacher what he or she wants—a reproduction of the teacher's experise. I realize that this is a crude oversimplification. On the other hand, I really like the model when ownership is shared, something like what Cathy Davidson describes in How to Crowdsource Grading. In this model, students do indeed become experts (and maybe owners) of the curriculum.
But what role should teachers' expertise play in the enactment of a course? Or in the "institutionalization" of content knowledge? I'm considering trying Davidson's idea of crowdsource grading next semester when I teach a class on writing technologies. When I look at her wonderful inquiry-based model, I find myself asking where is her expertise, her “ownership,” if we want to use that term? My sense is that both are there in, for example, the course design and the suggested readings. But it is also open to revision, radical revision in fact.
So, as I plan my future course, I wonder why am I still a little nervous about adding a line like this one from Davidson's syllabus?
"all students will receive the grade of A if they do all the work and their peers certify that they have done so in a satisfactory fashion. If you choose not to do some of the assignments and receive a lower grade…”
Maybe the answer is that many of us have been socialized to display our ownership and expertise through the distribution of grades...and maybe this model is what limits creativity.
Sorry for the ramble. Let me know if you have any thoughts here.
Billy Collins on Grading
Billy Collins is reading his poetry at the National Writing Project conference in Philadelphia. Here are a few lines from "School," a poem in which the teacher imagines all of the students he every taught coming together in a small town. Collins speaks to the ways in which grades can become self-defining (or a least the ways in which educators can imagine their students):
Their grades are sewn into their clothes
like references to Hawthorne.
The A's stroll along with other A's.
The D's honk whenever they pass another D.
You can see the full poem here.
portfolio-based evaluation the only way to go
While the new media side of this question is interesting, I think the biggest change we need to work toward is a shift away from assesment on outcomes to something which primarly focuses on process. There is a growing body of literature in Educational Psychology that suggests that at all stages, but particularly at the basic stages of developing profficiencies, that a focus on process is far more important than outcomes.
In the big picture, I think we need to be focusing on fostering the kinds of habits which these portfolio assesments support. Namely, metacognition and self-regulation.
Process--because learning is recursive
I don't know that outcomes and process need to be understood as being at odds, but I completely agree that we do little in currciula and grading to put the appropriate emphasis on learning processes. As Jayme notes, promoting self-assessment and student agency is essential. But the other reason is simply because learning takes time, which an ePortfolio imports. To that end and as prelude to our harvesting work, and to underpin some of Trevor's points and pointing to research on learning, I point here:
Portfolio-based evaluation
Trevor, I agree that the main benefit of portfolio-based assessment is metacognition and self-assessment (as well as self-regulation.) One of the things that we would like to see happen with our "Harvesting Gradebook" project is that while institutional assessment might get embedded next to the students' work, wherever it resides, that students would be encouraged to craft their own embedded feedback systems so that they could say, "I was trying to do this with this particular project. Is that the reading you're getting? Have I succeeded? How or how not?" That allows the students some autonomy about the kind of feedback they are soliciting, for them to become more intentional about what it is they are trying to achieve and savvy about how to ask for the feedback they need.
Implementation of student requested feedback
Jayme could have pointed back to this piece from a year ago where a design was proposed for a system to allow learners to request specific feedback in a more general campus-supported harvesting feedback tool. All told, the technology for these ideas is readily available, but as Jon Mott notes in this post, its not really a technology problem. Mott is reflecting on a John Seely Brown talk that explored the role of play and the different kinds of ourselves that we are educating.
Spare me the technolgy vs good pedagogy sermon!
Is anybody else but me weary of this line of argument? Yes, Nils, I could have pointed back to the blog post you mention but I would also like to point out that those are pictures, PICTURES of a dashboard. I know because I created those pictures. You say "All told, the technology for these ideas is readily available, but as Jon Mott notes in this post, its not really a technology problem." Actually the technology isn't readily available and that is part of the problem. We have had a lot of great ideas about how to streamline this process; how to make it easy to create and embed feedback mechanisms; how to visually output the data in ways that capture process - things that grades can't even begin to do; but the truth is that we have very little humanpower to put behind making it a reality. The technology does matter when the implementation for really great ideas - such as crowdsourcing formative feedback in meaningful and scalable ways - are being blocked for lack of human resources to focus on the implementation.
I guess my post is really about asking if there is anyone in the HASTAC community that is interested in collaboratively working on some of these tools to make them a reality? Sure, great tools don't make great teachers but they can make a difference in doing things that were never possible before - to get people thinking about assessment as truly formative without increasing the time commitment.
great tools for great teachers
Hi Jayme,
I would definitely be interested in working on assessment tools with you -- and I'm sure many others will want to join before the forum's over! One straightforward way to think of improving the "effectiveness" of formative assessment in the classroom is "simply" by designing a tool that gets the information from the "reviewer" (teacher or student) to the "reviewee" in a more timely manner. One can easily dream up ways that technology can make that a reality, today.
Also, I totally agree with you that great tools don't necessarily make great teachers, but I did want to mention the possibility of designing assessment tools that can (almost as a welcome "side effect") illuminate or broaden the user's perspectives on assessment. In a way, one of the many neat things harvesting gradebook does is make the student more aware of this "business of assessment": what tools & metrics are commonly involved, how to construct a rubric, how to select an appropriate scoring method, etc. Another example is Herb Ginsburg/Wireless Gen's mClass:math pda product, which employs a diagnostic interview protocol based on Prof Ginsburg's research on how young children come to understand (or not understand) mathematical thinking. It's conceivable that a teacher who uses this diagnostic tool enough times will gain a better understanding of children's developmental trajectory towards math proficiency.
some tools are already available, if used well
In my Interdisciplinary Computing and the Arts classes, I've been using a wiki, which could be said to be a very simple tool, but its been interesting to see how its been used.
For example, in my Intro to Electronics for Art class (wiki here: http://bang.calit2.net/wiki/VIS147A), I asked the students to post their first assignment in the wiki, but I didn't specify a format. The first poster created their own page, included multiple photos and a text description of their project and their process. (the projects page is here: http://bang.calit2.net/wiki/147a_Throwies_Projects). What happened next was that all the other projects followed that format, which was a format I was very happy with. The student actually told me later that they wanted to post first to encourage the other students to do better, setting a high bar. I felt that by simply making the project documentation something public, it improved the quality of the projects. Students also know that the projects are on the web so that I can share them easily with other instructors, which also may motivate them.
I'm very much an advocate of using open souce tools, and I think that a wiki is an easy way to allow students to share their work online, and can also be used in a wide variety of ways. I've also invited students to add relevant examples to the sections for each week, but they haven't gone that far yet! I did have to take some class time to explain how to edit a wiki, though, even in an upper division art and technology class! So don't assume that students all know how to edit wikipedia just because they use it daily.
Initially, I was motivated to use a wiki because it related to my own pedagogical approach, trying to learn from Paolo Friere and bell hooks how to encourage a horizontal learning environment where every student is empowered as an educator and a learner. It also is great because it allows me to easily compile examples and links to readings for class, which can then be used for lecture notes since they're already online. I usually use my wiki instead of a slideshow now, because there doesn't seem to be much benefit to slides.
It needs to be Blogger simple
Jayme, I hear you, the Harvesting Gradebook is documented, pilot tested, and we are using it for WSU accreditation, but we have not gotten the last mile -- its not 1-2-3 simple the way it needs to be (no matter how hard I imagine what it could be). So, in a way, it is a technology problem, we can't have the learning and assessment conversations that are needed, because the tool's clumsy aspects get in the way.
[ For those who have not figured it out, Jayme and I are colleagues. We blog here, where we have agreed to be able to disagree in public. We agreed this because we recognize a community of practice, in Lave & Wenger's view, is not a "warmly persuasive term for an existing set of relations." (Williams, 1977) ]
And I understand your post as a version of Margo Tamez' "Urgent call for help" a post aimed at rallying help to work on a problem. Tamez has created an interesting portfolio of her work in a community of practice. Her work has been very formative to shaping my understanding of ePortfolios and collaboration in community.
So, to add my voice to yours, we need to get the last mile. Recently Theron found a teacher using a Google Docs spreadsheet to make an online rubric for grading -- another attempt at ideas parallel to his demonstration of mashing up a portfolio with a rubric in Google Docs.
Each of these demonstrations are tantalizing, but each has a few too many wrinkles to meet the criteria of "Blogger simple." Strapped for resources to forge ahead as fast as our imaginations run, I agree we could use some help getting the last mile.
I'd be interested in
I'd be interested in collaborative work on this problem. I developed an "eFolio" also called the "Personal Learning Plan" (Gibson Personal Learning Plan on Google) and it has seen a lot of use over the years. I recently put it into a LAMP framework and I'm working now to bring that new version more into a WEB 2.0 style, but as someone mentioned here, its tough to make the problem of assessment easy and straightforward. I think the problem of increasing the time commitment has to be addressed by saying that we actually need many more time-based measures, which I think is the only way to capture the dynamics of change over time. So this leads me to believe that we HAVE to invent more automated feedback mechanisms of very high quality, which can provide useful feedback FOR learning. (For the assessment OF learning, we can stay with the broken, few measures per unit of time, system that we have now, which is not going anywhere soon anyway).
Games and simulations show the way to do this in many respects, because the software stays in immediate, real-time connection with the actions of the user and gives rapid, ample hints for improved performance. A layer that can be developed much further is the post hoc analysis of the user path of a particular performance and a comparison with the crowd-sourced paths of other problem-solvers who utilized the same digital space. Ron Stevens' work at UCLA is a great model for some of this.
While working on an online case-based reasoning application with Sara Dexter, now of the University of Virginia, I was part of a team that built an eassy-scoring tool using a simple Bayes network that got trained from well-scored pieces. The engine could, almost out-of-the-box, do better than unmoderated humans using a rubric to score student work. I think this area holds some promise for a 2.0-type performance assessment system created and owned by some CoP.
Finally, for now, Robert Mislevy and his colleagues and collaborators have laid out the theoretical framework for this new kind of performance assessment system. So the outlines and hints of how to move forward are out on the table. There is a need for a collaborative research project to pull these strands together in new ways and make additional advances that might potentially address the subject of this great dialog at HASTAC.
Portfolio grading technologies
I use the Learning Record portfolio system for evaluation in my courses, and one of my colleagues, Will Martin, has created an excellent online tool for storing and commenting on student work. You can view his walkthrough of the system (pdf), but if you're interested in trying it out, it is currently live at https://lro.cwrl.utexas.edu/ for anyone to use.
The LR doesn't currently support the multi-layered feedback that Jayme has described as part of the Harvesting Gradebook, but Will created the LRO as an open source project, which I believe he will soon be releasing so that others can build on the framework he has created.
The Learning Record Workshop
Also, the conceptual framework of the learning record, which the DWRL will be addressing this Friday in the LRO workshop, is adaptable to just about any platform. It can be delivered in Word, Drupal, or even (gasp) on paper.
We'll live tweet it. Follow DWRL (http://twitter.com/dwrl) if you are interested. #LRO
Adaptability, Play, and Assessment
Thank you, John, Dixie, and Matt, for organizing this wonderful forum. Great topic!
Per the comments above about outcomes, as an instructor who relies heavily on project-based curricula, I've found that asking students to compose their own outcomes is an incredibly productive exercise, especially when those outcomes go through a peer review process and are mobilized in the final evaluation of a student's project. Of course, these student-written outcomes can function in tandem with outcomes written by the instructor, too. That way, you find a balance between institution- and community-based learning.
Echoing Jayme's comment about metacognition and self-assessment, I would add that curricula with flexible and modifiable infrastructures that privilege modeling critical thought (over, say, being responsible for specific content or canons) really lend themselves to the crucial moment when a student asks a peer, "Have I succeeded?" If students are responsible not for the content of a certain text, but rather for articulating why, how, and for whom they made particular moves in a project, then they can also ultimately address how to replicate those moves, revise them, or teach them to someone else. The project, as a model, has a work-life beyond a course. And again, outcomes are crucial to this kind of process-oriented learning climate.
My only concern is naturalizing the word "play" as the antithesis of science and such. After all, play is serious and culturally embedded, and it has consequences. I concur with the impulse to encourage "low stakes" (read: will not be evaluated) exercises in a given class; however, what are the effects of rendering "low stakes" synonymous with "play," particularly in the context of the classroom or the course? True, no one's going to quibble with the notion that students should be given opportunities to foster their imaginations and take risks, yet what I'm wondering is how to foster adaptability and self-awareness. Both can be quite playful (in the sense of taking a chance and embracing error), indeed. But, as Jayme points out, perhaps the primary issue here is how to sustain a space where chance and error are not just ok; they are exactly how you learn and document learning (sort of like a change log in code). Right now, the only thing I found that works consistently is resisting the urge to grade students on their work until the quarter's/semester's end and relying primarily on peer review and workshops for feedback.
What works for others? What chancy spaces for abductive reasoning, modeling, or the like have you found productive?
play, games and theater of the oppressed
Perhaps this is because I'm in a visual arts department, but almost every quarter I find a way to tie in some physical theater exercises from Theater of the Oppressed. Augusto Boal has a wonderful book entitled "Games for Actors and Non-Actors", some of which are more verbal than physical, but all allow a space of play and experimentation with ideas. What I love about these games is not just their ability to spur conversation and engagement, but the way that students every quarter tie the material into the games in novel ways. Some ones that I stick to are Colombian Hypnosis, the Sculpture/Image game and the Mirror/Image game. Clearly these work best in a performance art class, but even in my interdisciplinary technology and arts classes, they serve as ways to allow students to think totally differently about interactivity, hierarchy and learning, outside of just a discussion sitting in a classroom. You'd be surprised how easy it is to find an empty, unused outdoor space inbetween classes on a university campus. Herre are two pictures of a game called Grandmother's Footsteps, an excellent exercise in rule bending, hierarchy and social inclusion/exclusion 1, 2. Often, students will relate the class discussion back to embodied experiences they had in the game.
Another technique i try to use to make my classes more horizontal (and neither of these two ideas really have to do with new media but i wanted to repond to your question), is to make sure that students have time for small group discussion, groups of 4 or 5 at the most, to discuss readings before we talk about them as a group and then i add my personal analysis later in case something was missed. That way, they can emote to their peers their real thoughts and get the thinking going, rather than just sit silently in a large group not feeling like they could say what they really think about the reading. It usually helps getting more people talking, but I still end up with the problem of the same few people raising their hands to speak even after those small groups, so I usually have to intervene a bit to hear from more people. I think the important point here is that students need spaces to talk to their peers without us watching or being present, apart from the comment I made earlier about wikis, and they don't nevessarily have the time to do this on their own. Electronic forums can facilitate this, but I think things like webct are usually open for instructors to see, and everyone's busy on facebook talking about silly photos and people's outfits, not the latest reading about the philosophy of electronics.
Assess the prompt for this discussion
Washington State University has been exploring assessment in a community as an alternative way to think about grading.
To experience a simple method for harvesting feedback from work anywhere on the Internet, as well as create more fodder for this discussion, please assess Fiona's opening prompt for this discussion (above) and then click HERE to open the feedback tool. After some data has been collected we'll post a link to view the results.
Interesting First Response
So far we have just one response to the survey above. It is generating conversation here at WSU, which I'll update here when it settles. This reviewer gave an incomplete set of numeric scores using the rubric, but has very interesting comments which are below.
Identification of the Problem:
All of these things are "true"; none of them is mutually exclusive; many of them are difficult to do; many of them need contextualizing knowledge such as student group members should first understand group self-assessment and group processes); many of these have been discussed in relation to the very first instructional technologies (e.g., See Roxanne Hiltz's work on virtual classrooms and asynchronous learning networks, where she emphasizes how to use systems and software to promote participation and collaboration); each sentence could be a discussion thread.
Contextual Factors:
No one, without clear prompting, training, and encouragement from other group members, is likely to engage in the higher levels of contextual factors in blog posts. So, using these kinds of ratings will end up seemingly very critical and I in no way want to be critical of these intentionally very preliminary thoughts. Using these criteria might be a good goal of a very rigorous discussion session, once major positions and issues and background are made available. That is, these kinds of criteria are likely to be met only in quite sophisticated venues such as school debating clubs or courses on logic and argumentation.
Own Perspective:
For many such questions, I prefer:
"Clearly presents and justifies own view or hypothesis while qualifying or integrating contrary views or interpretations."
Forcing the implied critique of "May remain within “safe” or predictable parameters," depending on the issue, could seem condescending.
Data:
No sources except an occasional link to another blog post are presented throughout the discussions -- for good reason, and because these kinds of assessments were not explicitly made part of the goals of the discussion. So, post-hoc application of such criteria I think could be quite harmful.
Feedback on Criteria:
Trying to accomplish even the middle of all of these dimensions would require quite a long time, multiple revisions, and much longer text than most any kind of blog implies or could tolerate.
Now, if these dimensions were the point of a course, and separate components of a final paper were each developed to work on each one of these dimensions separately, with good feedback, leading up to a final integrated argument and project, with group evaluation (following good group process understanding), that would be a great course and learning process.
Respond to these two perspectives on assessment:
1. Only faculty are qualified to create and ratify assessment criteria.
2. Expert consensus from the community of practice should validate the assessment instrument and criteria.
I think all potential stakeholders should be involved in validating assessment criteria.
Tensions
Thanks to the organizer of the forum questions -- these are definitely concerns and ideas to gnash thoroughly.
I'd like to offer a few tensions that I'm seeing from both the forum prompt and from my own experiences (teaching my own classes and mentoring other instructors):
1) What do we do with the desire for "effectiveness" and "time-efficiency" (which is a problematic and very real constraint) and the desire to be good teachers, mentors, and learners? Unfortunately, these things are often at odds. And is technology the best practice to find ways to be effective and time-efficient?
2) I completely agree with Jayme and John's points about making sure that students do have the opportunity to risk and fail, to discover what it means to be "creative" and to be "productive" without the threat of "assessment." But I also think that we can tease out the differences between "evaluation" (which can be different than assigning grades or numbers) and "assessment" (which, given the language of testing, is about grades and numbers). So, how might a paper or a project or a presentation be evaluated (e.g. workshop or peer review feedback) but not assessed?
Moreover, how might we need to better negotiate the two poles? After all, as much as we want to be student-centered and to have student-generative learning, there is still a time and place for direction, framing, and lecture. I agree that process is important, and things like self-evaluation and metacognition are important -- but we still have to teach students and ourselves what that looks like, what that means, and how to evaluate/assess those things. Given that one student's metacognitive process is not the same as another's, the danger here is ceding too much to the student. And how do we balance one student's desire to learn the "old-fashioned way" against another student's desire to "mix it up?" It might be also helpful to think about how to work within prescribed/proscribed frameworks (given that teachers are often constrained by larger structures, e.g. institutional demands) through oppositional, autoethnographic, and resistant practices. (Obviously, we're all doing this to some degree already.)
3) Finally, I have a specific concern about collaborative projects -- mainly because I find them a pedagogical minefield -- given that asymmetries in power, in interest, in willingness, in ability, and in personality (even personal politics and values) often produce challenges in assessment and evaluation. Having survived a number of group projects at almost every level of my education and having tried out multiple configurations of collaboration, I sometimes find the drive toward collaboration for collaboration's sake to be counterproductive. I do recognize the value in group work, in group assessment, and in the nitty-gritty of just trying to figure out how to work, live, and socialize in groups. it's just a thorny process.
Simultaneously, I also find collaborative work to be used by some instructors as "time-savers" and as "time-fillers." How might we metacognitively approach our own desires for collaboration? Again, what happens when our best intentions and best practices simply do not yield what we want out of these projects, processes, and technologies? Do we simply keep trying to apply the collaborative (or technological) fix?
Collaborative projects
Thanks for pointing out these tensions. I agree that collaboration for the sake of collaboration is troublesome. It's one of the reasons we were drawn to Hastac from Cathy Davidson's interview about participatory learning. Certainly, putting students into arbitrary groups feels phony -- though there are many people who would argue that it happens all the time in the working world when we are forced to be on committees that hold no interest for us -- but I think that others of us would agree that "surviving" these kind of artificial group setups is not really the most productive learning experience. What really drew us to the participatory learning article was this notion that people could "join" at all levels. That there could be lurkers. That people could come and go as they became interested and saw ways that they could contribute. So much of what we have been thinking about at WSU is how to provide these participatory spaces and make them conducive to contributing.
But this seems to require a really different setup for assessment. I agree with Trevor that the way out would seem to be some sort of portfolio assessment where the student could make sense of his or her contributions and interactions in various participatory spaces but it quickly becomes so different from the traditional classroom situation with a grade at the end that we'd be kidding ourselves if we didn't admit it's a little scary. Suddenly we're not only asking about the value of classroom assessment but the value of the university as a credentialing body. But I agree wholeheartedly that there is a critical need for instructors to act as guides. And universities are unique in the kind of diverse intellectual power that they house in one place. It really seems like it's the assessment piece that is the missing part of the puzzle.