May. 31st, 2014

koganbot: (Default)
Reposting this from Tumblr, where Tom Ewing links a piece at NPR by Amy Kamenetz. Here are Tom's comments (though I'm not sure whom he's quoting at the start) and underneath them my own speculation:

If kids report that they’re transgender and have one leg and belong to a gang and have several children … take it with a grain of salt.
This is a good article on, basically, kids trolling surveys for a laugh. It happens a lot. If I was a kid in the age of online surveys, I’d do it too. Especially if it was one of those surveys where the sole purpose is a hand-wringing clickbait headline about how kids these days think the Earth is flat or would marry their iPhone or whatever. If you’re offered a ludicrous answer in a survey designed to confirm someone’s view of how crappy the modern world is or how dumb everybody is - well, it’s hardly surprising some people take the hint.

But as one look at the article summary tells you, this is also a real problem. What the industry euphemistically calls “hard-to-reach” populations - small minority populations, basically - are actually harmed by this kind of stuff. The article has a good example - a study that reported negative impacts of adoption turned out to show nothing of the sort when troll answers got taken out.

I am not part of any population that suffers from prejudice or bias - name a privilege and I benefit from it. But I am a researcher, so I see at reasonably close hand what happens to data. And it seems to me that data and representation have a treacherous relationship. Inevitably, since people find in data what they are looking for.

On the one hand, data can offer stark evidence of inequalities, different needs and priorities, and different experiences: numbers that can be vital in making a case for change. On the other, data can be the comfort blanket that tells decision makers that change isn’t important. Research can erase minorities by reducing them to the status of a statistical insignificance, or it can ignore the diversity of their experiences in favour of a data-enforced average. There is every reason for people to mistrust data and research.

And cases like the adoption study one introduce yet another such reason - the possibility that careless research will end up magnifying the voices of the mischievous (or, let’s face it, malicious) and endorse stigmatizing myths instead of revealing anything useful. The remedies outlined - dummy questions in particular - are ingenious, and this kind of internal check should be routine in any important survey. But the uneasy relationship between research and representation - at the analysis stage as well as the collection stage - is harder to solve.
Article doesn't mention gender, but I would wager that most mischievous responders are male. I'd also bet - not quite as confidently - that because the surveys were done in a classroom, and despite their being anonymous, they got a higher rate of mischief than if the responders hadn't been in the same room together in a teen-specific setting.

I'm relying on my imperfect memory here, but I recall an article in Billboard in the '80s that stated that teen girls had an outsize effect on what was played on Top 40 radio because researchers simply didn't trust what teen boys would tell them and therefore discounted what the boys said or wouldn't even survey boys - if I remember right, the article didn't cite boys' tendency towards mischief but rather said that boys were hostile and defensive (and I assume underlying this, frightened; and I assume the mischief is somewhat fear-based itself). Whereas you could trust a girl's response much more, that she listened to what she said she did, that she bought the advertised products she said she did. (But from reading your posts over the years I wouldn't be surprised if you were to tell me that the reliability of girls' responses is only relatively better, that there are all sorts of reasons that even sincere responses can't be trusted, ranging from the respondents' not understanding the question to their not knowing their own mind, etc.)

Profile

koganbot: (Default)
Frank Kogan

July 2025

S M T W T F S
  12345
6789 101112
13141516171819
20212223242526
2728293031  

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Aug. 18th, 2025 12:44 am
Powered by Dreamwidth Studios