"I mean, the sense in which it also applies to the people, is that your report should summarize the 'likelihood' that your results were generated by a 10% propensity for all-10s to guess within 30 minutes, not the 'likelihood' that your results were generated by a less than 50% propensity for all-10s to guess within 30 minutes.  Because to do the latter thing you have to make a bunch of weird assumptions, and your math is just going to get more and more needlessly complicated as you dig yourself in further."

"And by way of showing how much further into complicated trouble you'd end up digging yourself:"

"Again, let's say we were going by bucketed hypotheses.  One hypothesis, the meta-coin hypothesis, says that there's a 1/3 chance we live in a world where all-10s have a 10% propensity to solve 2-4-6 in 30 minutes, 1/3 chance it's 20% propensity, 1/3 40%.  The other hypothesis, the fair-coin hypothesis, says we live in a world where all-10s have a 50% propensity to solve in 30."

"We don't actually need to consider the probability of these two hypotheses relative to each other, because our experimental report is just going to summarize the 0.2 'likelihood' of the data assuming the meta-coin hypothesis bucket, and the 0.3 likelihood of the data assuming the fair-coin hypothesis."

"So we publish our report."

"Along come some replicators.  They test 5 more people.  They get YES NO NO NO YES, so also two subjects who guessed and three who didn't."

"Now what?  What does the combined evidence say?  Anybody want to give the obvious wrong answer?"

"I mean, the sense in which it also applies to the people, is that your report should summarize the 'likelihood' that your results were generated by a 10% propensity for all-10s to guess within 30 minutes, not the 'likelihood' that your results were generated by a less than 50% propensity for all-10s to guess within 30 minutes.  Because to do the latter thing you have to make a bunch of weird assumptions, and your math is just going to get more and more needlessly complicated as you dig yourself in further."

"And by way of showing how much further into complicated trouble you'd end up digging yourself:"

"Again, let's say we were going by bucketed hypotheses.  One hypothesis, the meta-coin hypothesis, says that there's a 1/3 chance we live in a world where all-10s have a 10% propensity to solve 2-4-6 in 30 minutes, 1/3 chance it's 20% propensity, 1/3 40%.  The other hypothesis, the fair-coin hypothesis, says we live in a world where all-10s have a 50% propensity to solve in 30."

"We don't actually need to consider the probability of these two hypotheses relative to each other.  Let's say we test five all-10 subjects and get NO YES YES NO NO, meaning two subjects guessed within the time limit, three didn't.  Our experimental report is just going to summarize the 0.2 'likelihood' of the data assuming the meta-coin hypothesis bucket, and the 0.3 'likelihood' of the data assuming the fair-coin hypothesis.  That's true regardless of the 'relative prior-odds' of the two hypotheses relative to each other."

"So we publish our report.  0.2 likelihood for the less-than-50% bucket, 0.3 likelihood for the 50%-propensity hypothesis.  There's a questionable assumption that 10%, 20%, and 40% were all 1/3 likely assuming the propensity was under 50%, but fine, whatever, we've got to assume something to report on that whole bucket all at once, right."

"Along come some replicators.  They test 5 more people.  They get YES NO NO NO YES, so also two subjects who guessed and three who didn't."

"Now what?  What does the combined evidence say?  Anybody want to give the obvious wrong answer?"

"I mean, the sense in which it also applies to the people, is that your report should summarize the 'likelihood' that your results were generated by a 10% propensity for all-10s to guess within 30 minutes, not the 'likelihood' that your results were generated by a less than 50% propensity for all-10s to guess within 30 minutes.  Because to do the latter thing you have to make a bunch of weird assumptions, and your math is just going to get more and more needlessly complicated as you dig yourself in further."

"And by way of showing how much further into complicated trouble you'd end up digging yourself:"

"Again, let's say we were going by bucketed hypotheses.  One hypothesis, the meta-coin hypothesis, says that there's a 1/3 chance we live in a world where all-10s have a 10% propensity to solve 2-4-6 in 30 minutes, 1/3 chance it's 20% propensity, 1/3 40%.  The other hypothesis, the fair-coin hypothesis, says we live in a world where all-10s have a 50% propensity to solve in 30."

"We don't actually need to consider the probability of these two hypotheses relative to each other.  Let's say we test five all-10 subjects and get NO YES YES NO NO, meaning two subjects guessed within the time limit, three didn't.  Our experimental report is just going to summarize the 0.2 'likelihood' of the data assuming the meta-coin hypothesis bucket, and the 0.3 'likelihood' of the data assuming the fair-coin hypothesis.  That's true regardless of the 'relative prior-odds' of the two hypotheses relative to each other."

"So we publish our report.  0.2 likelihood for the less-than-50% bucket, 0.3 likelihood for the 50%-propensity hypothesis.  There's a questionable assumption that 10%, 20%, and 40% were all 1/3 likely assuming the propensity was under 50%, but fine, whatever, we've got to assume some 'prior distribution' to report a combined likelihood on that whole bucket all at once, yo."

"Along come some replicators.  They test 5 more people.  They get YES NO NO NO YES, so also two subjects who guessed and three who didn't."

"Now what?  What does the combined evidence say?  Anybody want to give the obvious wrong answer?"