How should we treat list questions?

Question

Questions have been voted down, put on hold, or closed for being "list questions" without an objective answer. I think this is a problem, for a few reasons:

1. The rule is inconsistently applied. This question ("What non-programming book is vital for learning the CS mindset?") was shut down, while this question (Advances in CS appropriate for CS1 and CS2 made by female computer scientists) was unchallenged (except by me). It has been suggested in chat that questions some should be held to a different standard from other questions, something I disagree strongly with. (If I misunderstood the chat comments, I hope someone will correct me.)
2. List questions are useful. The answers to both of the above questions are interesting and valuable. Resource request questions are fundamentally list questions. If you look at the help guide, it says that subjective questions (which includes many list questions) are all right if they:
   • inspire answers that explain “why” and “how”
   • tend to have long, not short, answers
   • have a constructive, fair, and impartial tone
   • invite sharing experiences over opinions
   • insist that opinion be backed up with facts and references
   • are more than just mindless social fun

I propose clarifying the rules about list questions and applying them consistently.

Answer 1

Why list questions get a bad rap

I'm pretty sure we can agree on this—it's not fun to spend time trying to write a question, and just get answers like this:

You should look at [this book]; it's really great and relevant here. [Author] did a really great job at explaining it, so you should definitely read it.

Is this secretly spam? Just someone's favourite book that they wanted to share? List questions tend to attract answers that offer something which does, technically answer the question, but don't give anything really useful on their own. In that example, is there actually anything you can learn from in that post? Not without reading the book, which isn't really what we expect from answers here—there should be at least something that you can gain by reading the answer which doesn't send you somewhere else.

This is particularly important for links, which can quickly go dead, rendering the answer completely devoid of useful information. For the rationale of this, see Your answer is in another castle: when is an answer not an answer?

As users look through the answers of these questions, even with link-only or brief explanations, they upvote their favourites. Brief, link-only answers can often float to the top of the list, sending a signal that we think these answers are the best on our site. But we can do much better than that—the most insightful answers I've seen on this site haven't necessarily been the most popular or widely held opinions, but many list questions favour popular, well-known answers over truly thoughtful answers.

"But these are bad answers, not bad questions!"

Perhaps so, but when so many bad answers appear on list questions, people begin to associate list questions with low quality content, rightly or wrongly.

Following the principles outlined in Good Subjective, Bad Subjective can help, but this has to actually be applied strictly to stop bad answers from popping up.

At the moment, I feel that, if we're prepared to take on the additional moderation load, accepting more subjective answers might work. If we're prepared to ensure that every answer meets this minimum standard of explaining what they suggest, then the lists can become more manageable and useful.

List questions—the wrong question

Often, you don't really want a huge list of answers from which you can pick out useful solutions to you. Generally, explaining your situation can help to narrow down the solution set enough that you don't end up with a huge list of every vaguely relevant answer. If, despite giving your situation, there are still many answers, your question might still fall on the 'Good Subjective' side if you demand good explanations to your answers, and don't accept vague suggestions.

But it's always a judgement call; it's generally difficult to explain why a question is too subjective in words, but after spending a while in the Stack Exchange system, I know it when I see it.

Answer 2

First and most important thought: I want to reiterate to everyone that, as the OP of the question about women, I have recused myself from all moderating questions regarding the question itself. (I did do some moderation work on the answers, however). Therefore, when I make my case for the question here, I do so purely as a community member, not as a diamond. To do otherwise would be unfair. If the others in the community feel that it should be closed (or protected), then so be it. Please take the rest of my commentary in that light. I am not a moderator in this answer.

So within that light, I argue that my question about females has real criteria. Early on, when we were discussing whether cs1 and cs0 should be tags, we spent a fair amount of time discussing this study.

On the top of page 4, it gives a wonderful graph of topics. The graph on the right, I believe, is relevant here. The further out a topic is from the center, the more agreement among instructors that that topic had at least some place within a CS1 course. "Types", "Control", and "Style" had the strongest results; everyone agreed that they had some place. By contrast, "Lists", "Sorting", "Recursion", and "Abstraction" created very little agreement. There are other topics on the list as well.

There are actual numbers that go into making that graph; I don't have access to the direct numbers, but they clearly exist.

Now, when asking about which women, there would be basically two judgements to make regarding how well a particular answer fit with my question. The first would be, how significant was the person's contribution to the particular subfield, and the second would be how important is the particular subfield to eary cs studies.

The first question is subjective by its nature. The second question, however, is quantifiable (and some such quantifying work has been done in that study.) I would go so far as to suggest that there would exist a particular woman who would become a best fit, though (and this is very important) I don't have to be right about that for the question to have sufficiently met the criteria to be an allowable list question.

A good list question simply provides criteria by which the answers can be judged on a linear scale. There does not have to be perfect agreement by everyone on what should top that scale, though a well-formed question should net results that are not entirely surprising. I don't believe that the top answers to my question would shock anyone.

However, there was definitely messiness there, and I don't want to duck that. My question had a straightforwardly bad title. I won't defend that title! During the time that the question was a HNQ, most of the answers that we got ignored the question body, and just answered the title. I think this had something to do with the fact of the HNQ itself; people were attracted from outside of our community by the title, and that is what they responded to.

But ultimately, through the work of many people (especially, especially, especially thesecretmaster and Heather -- you two are amazing), it got cleaned up. If there were not strong enough criteria for a list question, there would have been no meaningful basis for that cleanup. The question itself provided the information that was needed to guide the cleanup efforts. This is as strong an indicator as any I could possibly hope for that the question provides meaningful criteria. If it did not, that cleanup would have simply been impossible.