What criteria should be used to judge questions as they are asked

Question

When a question is asked, the mods and folks in the classroom and guidance office make judgements about it. Is this a good question? Some criteria are used to give advice to OPs and even to close questions. People want the criteria to be objective and one of the "supposedly objective" criteria is "does this question have an objectively best answer. I think that in the recent discussions, the word objectively is being misused: That word doesn't mean what you think it means. This post (Advances in CS appropriate for CS1 and CS2 made by female computer scientists) for example asks for "Who, then, are scientists that are, or were, prominent in the field of computer science and whose important work is appropriate to incorporate into the early years of a CS degree?". There is no objectively best answer to that question. I can't say that strongly enough. There are tons of great answers and some are better than others, but none can be best unless you are willing to consider "all possible women in cs" and "all statements made about them" and rank them.

However, after a lot of answers come in, you can choose (pretty subjectively) the best among the answers you got, or you can apply the "most votes" criteria which sounds objective, but is really an aggregation of subjective votes by members. But, you say it's all we got.

Well maybe so, but by that criteria EVERY possible question has a "best" answer. Just wait for the votes, count them up. Done. Best. But then a better answer comes in.

---

Instead of looking for a criteria that can be applied at the beginning of the process that expects an objectively best answer, I suggest the following be used instead.

There should be some criteria, perhaps implicit and perhaps explicit in the question, by which answers can be judged on a somewhat linear scale. This is better because "Grace Hopper is more important than, say Barbara Liskov" or "this answer is more complete than that answer." Not a linear scale, of course, but you can compare, but I defy you to pick a best. Barbara Liskov's substitution principle, for example is likely about as important in CS1 as any other, including the work of Hopper on, say, Cobol.

You say Hopper, I say Liskov. Not objective. Neither is "best". Each is important, And moreover each is "more important" or "better" than the other depending on your criteria.

So, I pray you. Don't expect or look for best. Comparable is as high a standard as you will be able to reach.

---

Moreover, what is needed by the mods and others is judgement. Look at it, consider its ups and downs, use criteria not as absolute rules but as guides to a wise decision. It will be inherently subjective. Live with it.

The problem is that the criteria for selecting answers must be able to be applied before any answers have been written. The criteria therefore can NOT be entirely dependent on the answers that happen to be given. Unless you have a time machine.

Answer 1

Questions which have no correct (or objectively best) answer can still be on topic on our site. Certainly in our field and in the fields of other stack exchange sites, questions can exist which do not have answers that can be judged to be objectively correct or incorrect. For example, from a quick glance at the Hot Network Questions sidebar, I see the parenting question "Being told I'm the father to a son I don't want". This question clearly has no correct answer, because the OP has a number of equally viable options. A question like this one is a good question because it establishes criteria by which answers can be evaluated. In the case the criterion is the answer that would be the "best" advice. There is no objectively best way to handle the situation, and so no objectively best advice or most correct advice, but there are better pieces of advice and worse pieces of advice. Voting should allow the better advice to rise to the top can the worse advice to fall to the bottom.

On the other hand, for something like this (now closed as "Too Broad" and deleted) question "What language is used for CS1 (Introductory Programming) at your institution?" (2k+ users can follow the link), there is no way to see the difference between better language+institution pairs and worse language+institution pairs. That question simply asks for a list of things, and establishes no criteria on which the answers can be evaluated on. A question like "Which language do you find most effective for teaching object oriented programming?" would be a good question because it establishes criteria on which answers can be measured, in this case the criteria is the languages effectiveness for teaching OO, although there is still no way to select an objectively "best" answer.

When we say that there much be criteria on which an answer can be evaluated on, we don't mean evaluated objectively. The evaluation can also be subjective, but it has to be based on some criteria, not a question whose answers will simply be a list of different possibilities which cannot be compared with each other because there is no metric or criterion established to compare them.

---

Here is a relevant selection from the asking pages in the help center (emphasis mine):

You should only ask practical, answerable questions based on actual problems that you face. Chatty, open-ended questions diminish the usefulness of our site and push other questions off the front page.

If your motivation for asking the question is “I would like to participate in a discussion about ______”, then you should not be asking here. However, if your motivation is “I would like others to explain ______ to me”, then you are probably OK.

avoid asking subjective questions where …

• every answer is equally valid: “What’s your favorite ______?”
• your answer is provided along with the question, and you expect more answers: “I use ______ for ______, what do you use?”
• there is no actual problem to be solved: “I’m curious if other people feel like I do.”
• you are asking an open-ended, hypothetical question: “What if ______ happened?”

All subjective questions are expected to be constructive. What does that mean? Constructive subjective questions:

• inspire answers that explain “why” and “how”
• tend to have long, not short, answers
• have a constructive, fair, and impartial tone
• invite sharing experiences over opinions
• insist that opinion be backed up with facts and references

---

The answer to your question may not always be the one you wanted, but that doesn’t mean it is wrong. A conclusive answer isn’t always possible. When in doubt, ask people to cite their sources, or to explain how/where they learned something. Even if we don’t agree with you, or tell you exactly what you wanted to hear, remember: we’re just trying to help.

Answer 2

Why Education sites on SE are Different

The rules mentioned here: https://cseducators.meta.stackexchange.com/a/225/1293 seem fine to me and allow appropriate judgement.

However, I'd like to say a few words about why we are different from the more "technical" sites such as SO.SE.

For technical questions, SE works because the "community" contains experts who can provide "best" answers. However, it isn't much of an exaggeration to suggest that the hardest questions on SO are easier than the easiest questions at csEducators.SE, or any similar site (mathEducators...).

To justify such a statement remember that our "product" in education is people, not programs, and people are infinitely variable. To educate effectively you need to help students "change the wiring in their brains". I don't say that metaphorically. It is literally - biologically - true. Synapses need to be rearranged with connections established and strengthened. The implication of that is that educators deal with a much larger problem space than programmers in which context and nuance is everything. A program may need to work in, say, tens of different environments, but educational practice must apply to millions.

For this reason, "best" answers are, in principle, largely unavailable for many questions of importance. Earlier I posed a question about Reaching Every Student, a very difficult (maybe impossible) problem to solve completely. I've found it impossible to tag any answer as "best". However all contribute to an understanding of the issue (which is why I asked it), and any of them might give an individual teacher some insight into how to improve "reach" in her/his own class. Not all will apply (nuance and context), but some are likely to.

Moreover, in posing the question here, where teachers gather, lets people think about their own practice and what they already do and so contribute to the solution.

Of course, some questions just seek information from others, posed by those with little experience in some facet of teaching computing. Those questions are much easier to deal with, but ultimately have less impact. "How can I effectively introduce MVC in a first course" is such a question. But even there, context matters. Who are your students? What do they know? What are their/your expectations? ....

But my bottom line is that questions should be "useful" in improving practice, even when they don't lead to "perfect" solutions.

An example might help to clarify this. I also posed the question on Pointers. I have my own views on this, but didn't express them in the question. User nocomprende and I seem to have different philosophies on the question as to whether teaching should be top down or bottom up: concrete -> abstract or abstract -> concrete. Each of us, I'm sure has constructed valid methodologies consistent with our philosophy. We would therefore, probably want to give different answers to the Pointer question. But saying that one of our answers is "better" than the other is nearly impossible to measure. That is because the teaching takes place within that philosophy and also within a particular teaching environment. I've surrounded that question with other, possibly elaborate, other techniques/methods and so has nocomprende. My students are fine. Nocomprende's students are fine. It doesn't depend on this one question, but on the entire framework. At the end of the day (or curriculum) our students wind up knowing approximately the same things and can be equally effective, even if in different ways.

As a consequence of this BTW, I seldom down vote answers to my own questions unless the answer is clearly harmful. I wouldn't tolerate "beating the answer into them," for example.

Context - Nuance

Answer 3

I think that snap decisions are given too often. People read superficially, then choose a broadsword and say: "Too Subjective", "List Question", "Answer is in the question", "Too broad", "Too narrow"...

You can surmise this from a question I asked and in minutes was told both that it would require a book to answer and I had pre-framed the answer in the question. Maybe so, but maybe also, go slow. Give yourself a few minutes to read, forget, re-read and then make comments. An hour is not going to change anything for the worse.

Also, sometimes substantial edits are made which I am at a loss to know how to judge. I referenced a source in one question and included a couple quotes that framed my question, and these were removed. In another case, someone suggested that I self-answer, which is a huge offense on other SE sites I have used. Good practice on other sites seems to be bad practice here?

I would like to see things given more space to develop. I guess that can be hard to achieve when questions become popular quickly, or when extensive edits will go unnoticed by people who already read. Not sure how to address all these concerns, but "pause to think" is the best I can offer.