Why Robots Mustn't "Read"

Couple days ago, Fabulous Fred sent us to the NYTimes to learn about the Educational Testing Service's essay-reading computer program. Here is my response:

The last paragraph of the article includes this:

"Writing is a social practice, and it's the job of teachers to orchestrate rituals and relationships that support effective and meaningful writing," she said. "To do that, teachers need appropriate class sizes and professional development far more than they need a particular technology."

Why is this point made at the end of the article, instead of at the beginning? It's where the real trouble lies. With every explosion in class size, we've been moving ever nearer automated, quantified assessment of student writing. It had to happen. And it's impossible. Inherently meaningless.

Proof:

Scene: 35 or 40 writing instructors in a room, in groups of five. Huge piles of bluebooks containing exam essays on every table. A prepared selection of four or five, xeroxed in a pamphlet, in front of each reader, names marked out, identified by letter.

Group leader: OK, everybody, open your pamphlets, read the essay marked "P," and give it a score from one to five. Five minutes for this, let's get going, people!

Rustling as everyone reads and scores. By Minute Four, everybody's done.

The group leader has drawn a scoresheet on the board.

GL: Rosemary?

Rosemary: Five.

GL puts a tic in 5 column: Tom?

Tom: Six.

GL tics 6 column: Annie?

Annie (looking perplexed): Three.

GL tics 3 column.

And so on. Gradually, as all the marks appear, so does a pattern--85% think P deserves to pass; some think it's a strong pass. A few disagree.

GL: Would someone who gave P a 3 care to speak to why it's a 3?

Annie says something like: Well, this person is obviously not in control of transitions. The essay lurches from paragraph 2 to paragraph 3, we don't know what connection the writer's making between the points.

Tom: You don't see the connection? It's implicit! The reader's got to be willing to do some work, here.

Someone else who gave it a 3: And the writer is obviously a stranger to semicolons--look at the mess in paragraph 3! Fragments everywhere!

Someone else: That's parallel construction! You can do that with semicolons!

1st someone else: No, you can't! Each element has to be able to stand alone!

GL: People! People!! Can we move along, here? The head readers have decided that P is a 5 because [blahblahblah]. Please turn to Q now. Read and score. Go.

________________________________________

This is "norming," and every big reading session begins with this kind of tuneup, to remind readers of the holistic standards, which are entirely internal and subjective, but oddly enough, functionally uniform! It works every time--and I've done this dozens of times, with thousands of essays--most readers will agree, most of the time.

The trouble comes when anybody tries to articulate these standards, to quantify them. Watersheds abound. Readers fall out over just about any imaginable element. So whose standards are they going to program into the auto-reader: the reader who thinks semicolons can only join independent clauses, or the one who allows freedom of choice in sentence elements, so long as they're grammatically parallel? There's no real answer to this question; it's all a matter of opinion; there's no FACT here. And machines deal in facts.

What I'm looking forward to: the student who gets ahold of the program, finds out how many "because"es and how many semicolons are required to pass, then writes total gibberish that nevertheless fits the prescribed pattern, and passes. It WILL happen.

Comments

[Da Goddess]

Beautifully said.

[fredf]

You go, girl.