"Likewise if you started thinking that 'this coin isn't random' or 'this coin is biased to favor Queen' was a good way to describe the hypothesis you were considering. If two experiments show that the same coin is probably biased to Queen by notably different amounts, they're pointing at incompatible ways the world can be, and something is wrong, some condition has changed between experiments, at least one group is screwing up."
"You definitely wouldn't say, 'Well, our hypothesis was that this coin was biased to favor Queen, and group one spun it a bunch of times and found that it came up Queen 900 times out of 1000, and group two spun it a bunch of times and it came up Queen 520 times out of 1000, and both of those results are instances of 'the coin came up Queen more often than it came up Text', so both confirm the hypothesis that 'the coin is biased Queen', and the experiment has 'reproduced'. You are actually less confident after two apparent confirmations of your original statement than you were after one confirmation, because in the details of the particular worlds, it's clear that something was wrong with at least one experimental setup."
"But that apparent paradox is just an artifact of bucketing together different ways the world could be, that yield very different likelihoods on the exact data observed, into one metahypothesis of 'this coin is biased to favor Queen'. If you said instead 'the coin is 90% Queen' or 'the coin is 52% Queen', there would be no illusion of the experimental result having been 'reproduced' by the 'replication', it would be clear that the data from one experiment was compatible with that exact hypothesis and the data from the other experiment was not."
"Likewise if you started thinking that 'this coin isn't random' or 'this coin is biased to favor Queen' was a good way to describe the hypothesis you were considering. If two experiments show that the same coin is probably biased to Queen by notably different amounts, they're pointing at incompatible ways the world can be, and something is wrong, some condition has changed between experiments, at least one group is screwing up."
"You definitely wouldn't say, 'Well, our hypothesis was that this coin was biased to favor Queen, and group one spun it a bunch of times and found that it came up Queen 900 times out of 1000, and group two spun it a bunch of times and it came up Queen 520 times out of 1000, and both of those results are instances of 'the coin came up Queen more often than it came up Text', so both confirm the hypothesis that 'the coin is biased Queen', and the experiment has 'reproduced'. You are actually less confident after two apparent confirmations of your original statement than you were after one confirmation, because in the details of the particular worlds, it's clear that something was wrong with at least one experimental setup."
"But that apparent paradox is just an artifact of bucketing together different ways the world could be, that yield very different likelihoods on the exact data observed, into one metahypothesis of 'this coin is biased to favor Queen'. If you said instead 'the coin is 90% Queen' or 'the coin is 52% Queen', there would be no illusion of the experimental result having been 'reproduced' by the 'replication'. Fix a single hypothesis, a single effect-size, that makes the data have independent likelihoods between one experiment and another, that fully specifies the likelihood of the data as a matter of logic and doesn't change when we read other experimental reports. Summarize the likelihood for that, and it would be clear that the data from one experiment was compatible with that exact hypothesis, and the data from the other experiment was not."
"Which, uh, yeah, the moral is, there's this - certain exact way to do SCIENCE! correctly and the details out of dath ilan actually matter a lot for making your whole Civilization's SCIENCE! output fit together and have the whole thing make any sense. Even for a small project like ours, it's still probably best to do it that way, if you want things to make any sense."
"Likewise if you started thinking that 'this coin isn't random' or 'this coin is biased to favor Queen' was a good way to describe the hypothesis you were considering. If two experiments show that the same coin is probably biased to Queen by notably different amounts, they're pointing at incompatible ways the world can be, and something is wrong, some condition has changed between experiments, at least one group is screwing up."
"You definitely wouldn't say, 'Well, our hypothesis was that this coin was biased to favor Queen, and group one spun it a bunch of times and found that it came up Queen 900 times out of 1000, and group two spun it a bunch of times and it came up Queen 520 times out of 1000, and both of those results are instances of 'the coin came up Queen more often than it came up Text', so both confirm the hypothesis that 'the coin is biased Queen', and the experiment has 'reproduced'. You are actually less confident after two apparent confirmations of your original statement than you were after one confirmation, because in the details of the particular worlds, it's clear that something was wrong with at least one experimental setup."
"But that apparent paradox is just an artifact of bucketing together different ways the world could be, that yield very different likelihoods on the exact data observed, into one metahypothesis of 'this coin is biased to favor Queen'. If you said instead 'the coin is 90% Queen' or 'the coin is 52% Queen', there would be no illusion of the experimental result having been 'reproduced' by the 'replication'. Fix a single hypothesis, a single effect-size, that makes the data have independent likelihoods between one experiment and another, that fully specifies the likelihood of the data within a world as a matter of logic, and doesn't change when we read other experimental reports. Summarize the likelihoods for hypotheses like that, and it would be clear that the data from one experiment was compatible with an exact hypothesis, and the data from the other experiment was not."
"Which, uh, yeah, the lesson is, there's this certain exact way to do SCIENCE! correctly, and those details actually matter a lot for making your whole Civilization's SCIENCE! output fit together and have the whole thing make any sense. Even for a small project like ours, it's still probably best to do it that way, if we want things to make any sense."
"Likewise if you started thinking that 'this coin isn't random' or 'this coin is biased to favor Queen' was a good way to describe the hypothesis you were considering. If two experiments show that the same coin is probably biased to Queen by notably different amounts, they're pointing at incompatible ways the world can be, and something is wrong, some condition has changed between experiments, at least one group is screwing up."
"You definitely wouldn't say, 'Well, our hypothesis was that this coin was biased to favor Queen, and group one spun it a bunch of times and found that it came up Queen 900 times out of 1000, and group two spun it a bunch of times and it came up Queen 520 times out of 1000, and both of those results are instances of 'the coin came up Queen more often than it came up Text', so both confirm the hypothesis that 'the coin is biased Queen', and the experiment has 'reproduced'. You are actually less confident after two apparent confirmations of your original statement than you were after one confirmation, because in the details of the particular worlds, it's clear that something was wrong with at least one experimental setup."
"But that apparent paradox is just an artifact of bucketing together different ways the world could be, that yield very different likelihoods on the exact data observed, into one metahypothesis of 'this coin is biased to favor Queen'. If you said instead 'the coin is 90% Queen' or 'the coin is 52% Queen', there would be no illusion of the experimental result having been 'reproduced' by the 'replication'. Fix a single hypothesis, a single effect-size, that makes the data have independent likelihoods between one experiment and another, that fully specifies the likelihood of the data within a world as a matter of logic, and doesn't change when we read other experimental reports. Summarize the likelihoods for hypotheses like that, and it would be clear that the data from one experiment was compatible with an exact hypothesis, and the data from the other experiment was not."
"Which, uh, yeah, the lesson is, there are these careful precise details about how to do SCIENCE! correctly, and those details actually matter a lot for making your whole Civilization's SCIENCE! output fit together and have the whole thing make any sense. Even for a small project like ours, it's still probably best to do it that way, if we want things to make any sense."
"Likewise if you started thinking that 'this coin isn't random' or 'this coin is biased to favor Queen' was a good way to describe the hypothesis you were considering. If two experiments show that the same coin is probably biased to Queen by notably different amounts, they're pointing at incompatible ways the world can be, and something is wrong, some condition has changed between experiments, at least one group is screwing up."
"You definitely wouldn't say, 'Well, our hypothesis was that this coin was biased to favor Queen, and group one spun it a bunch of times and found that it came up Queen 900 times out of 1000, and group two spun it a bunch of times and it came up Queen 520 times out of 1000, and both of those results are instances of 'the coin came up Queen more often than it came up Text', so both confirm the hypothesis that 'the coin is biased Queen', and the experiment has 'reproduced'. You are actually less confident after two apparent confirmations of your original statement than you were after one confirmation, because in the details of the particular worlds, it's clear that something was wrong with at least one experimental setup."
"But that apparent paradox is just an artifact of bucketing together different ways the world could be, that yield very different likelihoods on the exact data observed, into one metahypothesis of 'this coin is biased to favor Queen'. If you said instead 'the coin yields 90% Queens' or 'the coin yields 52% Queens', there would be no illusion of the first experimental result agreeing with the second result, there would be no illusion that the result had 'reproduced'. Fix a single hypothesis, a single effect-size, that makes the data have independent likelihoods between one experiment and another, that fully specifies the likelihood of the data within a world as a matter of logic, and doesn't change when we read other experimental reports. Summarize the likelihoods for hypotheses like that, and it would be clear that the data from one experiment was compatible with an exact hypothesis, and the data from the other experiment was not."
"Which, uh, yeah, the lesson is, there are these careful precise details about how to do SCIENCE! correctly, and those details actually matter a lot for making your whole Civilization's SCIENCE! output fit together and have the whole thing make any sense. Even for a small project like ours, it's still probably best to do it that way, if we want things to make any sense."
"Likewise if you started thinking that 'this coin isn't random' or 'this coin is biased to favor Queen' was a good way to describe the hypothesis you were considering. If two experiments show that the same coin is probably biased to Queen by notably different amounts, they're pointing at incompatible ways the world can be, and something is wrong, some condition has changed between experiments, at least one group is screwing up."
"You definitely wouldn't say, 'Well, our hypothesis was that this coin was biased to favor Queen, and group one spun it a bunch of times and found that it came up Queen 900 times out of 1000, and group two spun it a bunch of times and it came up Queen 520 times out of 1000, and both of those results are instances of 'the coin came up Queen more often than it came up Text', so both confirm the hypothesis that 'the coin is biased Queen', and the experiment has 'reproduced'. You are actually less confident after two apparent confirmations of your original statement than you were after one confirmation, because in the details of the particular worlds, it's clear that something was wrong with at least one experimental setup."
"But that apparent paradox is just an artifact of bucketing together different ways the world could be, that yield very different likelihoods on the exact data observed, into one metahypothesis of 'this coin is biased to favor Queen'. If you said instead 'the coin yields 90% Queens' or 'the coin yields 52% Queens', there would be no illusion of the first experimental result agreeing with the second result, there would be no illusion that the result had 'reproduced'. Fix a local hypothesis, a single effect-size in this case, that makes the data have independent likelihoods between one experiment and another, that fully specifies the likelihood of the data as a matter of logic, and doesn't change when we read other experimental reports. Summarize the likelihoods for hypotheses like that, and it would be clear that the data from one experiment was compatible with an exact hypothesis, and the data from the other experiment was not."
"Which, uh, yeah, the lesson is, there are these careful precise details about how to do SCIENCE! correctly, and those details actually matter a lot for making your whole Civilization's SCIENCE! output fit together and have the whole thing make any sense. Even for a small project like ours, it's still probably best to do it that way, if we want things to make any sense."
