'Mischievous responders' confound research on teens

Teenagers face some serious issues: drugs, bullying, sexual violence, depression, gangs. They don't always like to talk about these things with adults.

One way that researchers and educators can get around that is to give teens a survey — a simple, anonymous questionnaire they can fill out by themselves without any grown-ups hovering over them. Hundreds of thousands of students take such surveys every year. School districts use them to gather data; so do the federal government, states and independent researchers.

But a new research paper points out one huge potential flaw in all this research: kids who skew the results by making stuff up for a giggle. "Mischievous Responders," they're called.

They may say they're 7 feet tall, or weigh 400 pounds, or have three children. They may exaggerate their sexual experiences, or lie about their supposed criminal activities. In other words, kids will be kids, especially when you ask them about sensitive issues.

Jackson Terry, 14, says he answered honestly when he took one of these surveys last year, but he knows kids who didn't.

"They handed out the sheet, I believe it was in language class," says Terry, who's from Granville, Ohio. "The teacher was in the room. It was anonymous. I think they asked us about bullying, do you feel safe in school, some questions about drugs, the learning environment."

Some kids "would joke through the entire thing and have a cocky attitude about it," Terry says. "Afterwards some would say, yeah, No. 5, that's totally not true; I just made something up."

Joseph P. Robinson-Cimpian, the author of the new paper, is an assistant professor of educational psychology at the University of Illinois, Urbana-Champaign. He says he first noticed this phenomenon, and coined the term "mischievous responder," in a paper he co-authored in 2011 with Dorothy L. Espelage.

"We were interested in the disparities between LGBT and non-LGBT youth in suicidal ideation, feelings of belonging, text-message bullying," he recalls. "One of our reviewers asked us, 'How do you know these kids are actually gay?' And giving some thought to it, we said, 'Let's figure out who those kids are.' "

Robinson-Cimpian and his co-author came up with a clever test.

They chose a set of answers on the survey — questions with responses that adolescents were likely to find funny, but ones that were statistically unlikely to be related to being lesbian, gay, bisexual or transgender. Their height, for example.

The more of these way-off-base answers that someone gave on these questions, the researchers surmised, the more likely they were to be lying about being LGBT as well.

And they did find a correlation. For example, 41 percent of the students who claimed they were transgender also claimed to be extremely tall or short, and the same percentage also claimed they were in a gang.

This is important because researchers are often the most interested in minority groups, and so the undetected presence of a small number of jokesters can seriously mess up results.

In a 2003 study, 19 percent of teens who claimed to be adopted actually weren't, according to follow-up interviews with their parents. When you excluded these kids (who also gave extreme responses on other items), the study no longer found a significant difference between adopted children and those who weren't on behaviors like drug use, drinking and skipping school. The paper had to be retracted. In yet another survey, fully 99 percent of 253 students who claimed to use an artificial limb were just kidding.

"Part of you laughs about it, and the researcher side is terrified," says Robinson-Cimpian. "We have to do something about this. We can't base research and policy and beliefs about these kids on faulty data."

Doing follow-up interviews isn't always practical or safe. For example, you can't very well ask a teenager's family if he or she is indeed gay. Robinson-Cimpian instead suggests designing surveys with the mischievous responder in mind. For example, one survey included a fake drug on a long list of real drugs. Another simply asked, as the last question, whether the respondent had told the truth; 12 percent fessed up.

Jackson Terry says that from a teenager's point of view, when he fills out a survey, he wants to know: "Who's going to know this? Who am I telling? This is kind of personal stuff."

And so he has another suggestion for researchers: Be upfront about what you're asking and why. Speak their language. "Be more realistic, and I guess more friendly." Copyright 2019 NPR. To see more, visit https://www.npr.org.