Fooling the ETS e-Rater Grader

Submitted by Joanna Fu on Thu, 04/26/2012 - 3:02pm.
Joanna Fu's picture

In light of the several talks that we've had recently about scoring essays and other free text response sections on tests, I came across this example of a poorly written essay that could get awarded a top grade from the E.T.S. e-Rater system.

The accompanying article notes the following flaws in the technology:

1. The e-Rater cannot decipher truth, so a student will not be penalized for writing that the War of 1812 started in 1945.

2. Longer essays (even if they are filled with nonsensical sentences) are awarded higher scores than shorter essays. The e-Rater also prefers long sentences to short.

3. When in doubt, using a big word in place of a smaller word increases one's score.

It's fascinating to see how far technology has evolved so that we can now use an algorithm to measure the quality and development of an argument -- and it's also interesting to see how much further there is to go before a computer scorer can fully replace a human grader.

 

EdLab Review: Learn21

Submitted by Joanna Fu on Wed, 04/18/2012 - 3:23pm.
Joanna Fu's picture

In 2010, the West Virginia Department of Education began Learn21, a website that offers interactive online games for students from pre-kindergarten to twelfth grade in the following subject areas: math, science, social studies, language arts, wellness, art, music, dance, theatre, agriculture, finance, and engineering.

Pros:
Teachers can use the games in class, and students can access the website when not in school if they would like to play some more or review a subject. Learn21 offers a wide variety of subject areas for study -- the humanities are prominently featured in addition to math and science -- and the entire site is free. Some of the games are quite innovative and intriguing, even for adults: for example, "The Great Flu" in Grade 9-12 Wellness attempts to educate players about the dangers and difficulties of containing flu pandemics.

Cons:
Such a large database of games and tools requires ongoing maintenance -- I noticed that several of the links to games (e.g., "Multiple Percents" in Grade 9-12 math resources) no longer work. The games linked on Learn21 are on third party sites, and the West Virginia Department of Education claims no affiliation or association with these sites -- which means that some of these games are very well developed and supported while others are not. In addition, some of the instructional tools and videos are slow to load and do not offer assistance when it would be helpful to the user.

 

Infographic: Where and How People Use Tablets

Submitted by Joanna Fu on Wed, 04/11/2012 - 5:33pm.
Joanna Fu's picture

I've found that whenever I meet someone who owns a tablet, I want to know what usage habits are, and when they see a need for a tablet over a smartphone. The infographics below help to summarize things:

data-how-mobile-01-2012

data-how-mobile-02-2012

via: Adweek

 

The Importance of Habit Testing

Submitted by Joanna Fu on Mon, 04/09/2012 - 1:11pm.
Joanna Fu's picture

As we’ve been working to revamp a couple of existing EdLab projects (Survey Sidekick, Pocket Knowledge), I’ve been thinking a lot about how people will be using these products and how we should design these tools so that people will prefer to use them over anything else. For example, with Survey Sidekick, I know that I've always preferred to begin the survey brainstorming process in a plain text editor before moving it to the actual creator. What features should Survey Sidekick therefore have to possess to compel me to use it through all stages of survey development? How do we redesign Pocket Knowledge so people think to continually return to it as a resource for research?

This TechCrunch article discusses “habit testing” -- measuring and collecting data about user habits in order to understand who your users are, what aspects of the product is habit forming, and why those qualities are habit forming.

 

The "Stupidity" of Games (and an Upcoming Talk)

Submitted by Joanna Fu on Thu, 04/05/2012 - 2:27pm.
Joanna Fu's picture

I’m fascinated by games, though I sometimes feel as though I come to them from an outsider perspective – I barely played any while growing up (I didn't encounter a single console until the ripe old age of 25!). I’ve generally resisted playing any games on my iPhone, and I've tried but failed to see the addictiveness of popular games like Angry Birds, Words with Friends, and Fruit Ninja. But games still hold a special allure, and I love trying to understand why and how people play.

In the New York Times article "Just One More Game..." Sam Anderson discusses the recent proliferation of mobile games, as well as the addictiveness of these "stupid games" and their impact on productivity. In one memorable line he notes, "Plants vs. Zombies, ate up, I’m going to guess, a full 'Anna Karenina' of my leisure time."

I do feel that he fails to address one key point: that not all games are “stupid games.” One could make a point that all forms of entertainment produce enjoyable but unchallenging media. But a movie like “Wrath of the Titans” can’t (and shouldn't) be compared to “Mulholland Drive,” nor should “Twilight” be judged by the same standards as “Madame Bovary.” I would argue that “smart games” develop a strong affinity for characterization and storytelling in players and can hone puzzle-solving skills. I'd like to see how educational games can harness the qualities that these "smart games" possess.

 

Most desirable life skills for 2020?

Submitted by Joanna Fu on Wed, 02/29/2012 - 12:48pm.
Joanna Fu's picture

According to a new report from the Pew Research Center’s Internet & American Life Project and the Imagining the Internet Center at Elon University, Millennials (those born between 1981-2000) will both “benefit and suffer due to their hyperconnected lives.” While a slight majority of respondents felt that the future for the hyperconnected will generally be positive, there are clear worries that this connection will also lead to a thirst for instant gratification and shallow decision-making abilities.

I can certainly empathize with study participants about the difficulties of living at a time where there is so much new content being created -- but what I found to be most interesting in the study (and also relevant to the EdLab) were the predictions about the most desired life skills for young people in 2020. They included:

- public problem-solving through cooperative work (crowd-sourcing solutions)

 

New videogame website archive at the British Library

Submitted by Joanna Fu on Fri, 02/17/2012 - 3:38pm.
Joanna Fu's picture

Curators at the British Library are in the process of creating an archive to preserve computer game websites and gaming culture. They are specifically interested in collecting resources that "discuss the cultural and societal impact of computer games, for example research on the impact of games on children’s development."

They are looking for gamers and designers to suggest material they consider worthy of preservation -- what are some seminal games that come to mind?

One note: while the British Library's collection will be U.K.-centric, there is a similar archive, the "How They Got Game" project at Stanford University.

 

Failing well?

Submitted by Joanna Fu on Fri, 02/10/2012 - 10:38am.
Joanna Fu's picture

A top high school in London is holding a “failure week” featuring discussions, activities, and workshops discussing coping with failing and the negative effects of never experiencing failure -- in an effort to show that “it is completely acceptable and completely normal not to succeed at times in life.”

This brought to my mind our recent EdLab Seminar on SuperFutures – a career planning platform to help teenagers determine a job path and teach real-world skills from real-world industry experts. I’m thinking two things:

1. Should failure (and the skills of resilience and risk embracing) be some of these real-world skills that are taught to young adults?

2. Is it unrealistic for students to know precisely what their best fit career is while still in high school, and are we trying to provide too much of a safety net (and thus hindering development)? I know that many of my peers changed majors multiple times while in college, and others have changed career paths several times as well… but is the road half taken completely wasted? Or does that exploration help provide us with the space we need to truly grow and learn?

 

The Academic Journals Edition of Clash of the Titans?

Submitted by Joanna Fu on Thu, 02/02/2012 - 12:17am.
Joanna Fu's picture

In response to a post written by noted British mathematician Tim Gowers, nearly twenty members of MIT’s faculty have signed a pledge to boycott Reed Elsevier (the world’s largest scientific journal publisher) and are “refusing to publish, referee, or do editorial work unless they [Elsevier] radically change how they operate.” In addition, an online pledge against Elsevier currently has 2,892 signees from universities around the world, from both junior and senior scholars across a wide array of disciplines.

The frustrations from these scientists and academics boil down to three key points: 1) exorbitantly high pricing practices (running in the tens of thousands), 2) the practice of bundling subscriptions of less desirable journals to more valuable ones, forcing libraries to purchase journals that they don’t want and won’t use, and 3) Elsevier’s support of the Research Works Act which would prevent the free flow of information.

 

Live, from the 2012 Youth Media Awards!

Submitted by Joanna Fu on Mon, 01/23/2012 - 12:01pm.
Joanna Fu's picture

I’ve been in Dallas, TX for the last few days to serve on the Stonewall Book Awards Committee at the ALA Midwinter conference. In addition to nonfiction and fiction book prizes, the committee selects a winner for the Mike Morgan and Larry Roman Children’s & Young Adult Literature Award -- given annually to "English-language children’s and young adult books of exceptional merit relating to the gay, lesbian, bisexual and transgender experience." I'm so happy that after months of confidential reading and discussion, I can finally reveal this year's winner: “Putting Makeup on the Fat Boy,” written by Bil Wright and published by Simon & Schuster BFYR, an imprint of Simon & Schuster Children’s Publishing Division.

The awards were presented at the ALA Youth Media Awards, which includes The Printz, Caldecott, and Newbery Awards (often dubbed "the Oscars" of the youth book world). Also presented was the winner for the Odyssey Award (best audiobook for children and/or young adults) and the Andrew Carnegie Medal (for excellence in children's video).

 
XML feed