#3 - If every obvious hypothesis has unexpectedly low 'likelihood' over all the combined data, it means the true theory wasn't in your starting set, often that different experiments had different hidden conditions

Let's say you're testing (only) the hypotheses for 20% propensity-to-solve and 60% propensity-to-solve.

Suppose you test 300 subjects, and 100 of them solve it.

At this point, obviously, they're not going to be multiplying things out by hand any more, and are going to be working with logarithms...

log2 (0.2) = -2.322
log2 (0.4) = -1.322
log2 (0.6) = -.737
log2 (0.8) = -.322

(bonus points, Keltham declares, for anybody who already noticed that 2 entries in this table are blatantly redundant, but he's writing them out anyways for ease of reference)

Loss to 20% propensity:
-2.32 * 100  +  -.32 * 200   = -296

Loss to 60% propensity:
-.74 * 100   +  -1.3 * 200   = -334

Now, if you are a naive six-year-old being led astray by an older child trolling you, you might look at this, and proclaim that the 20%-propensity hypothesis did much better than the 60%-propensity hypothesis, by a likelihood factor of over a hundred billion.

But really what this data is yelling at you is "The true hypothesis was not in your explicit hypothesis set!"

Why?  Consider about how well the two theories say they ought to do, loss-of-2s-wise, in the worlds where they are actually true:

Expected loss given 20% propensity:
-2.32 * 60  +  -.32 * 240   ~  -2&1/3 * 60 + -1/3 * 240  = -140 + -80  = -220

Expected loss given 60% propensity:
-.74 * 180  +  -1.32 * 120  ~  -3/4 * 180 + -1&1/3 * 120 = -135 + -160 = -295

...which is to say that both hypotheses lost way more than they expected to lose.

This is a sort of hint that tells you to look for a new hypothesis, like "1/3 propensity", say.  It holds even if you get the subtler hint that some hypothesis is doing way better than it expected to do.

There's obviously ways to think about all the possible propensities at once, but Keltham doesn't think he can get to those this morning, given that they don't have calculus yet.

#3 - If every obvious hypothesis has unexpectedly low 'likelihood' over all the combined data, it means the true theory wasn't in your starting set, often that different experiments had different hidden conditions

Let's say you're testing (only) the hypotheses for 20% propensity-to-solve and 60% propensity-to-solve.

Suppose you test 300 subjects, and 100 of them solve it.

At this point, obviously, they're not going to be multiplying things out by hand any more, and are going to be working with logarithms...

log2 (0.2) = -2.322
log2 (0.4) = -1.322
log2 (0.6) = -.737
log2 (0.8) = -.322

(bonus points, Keltham declares, for anybody who already noticed that 2 entries in this table are blatantly redundant, but he's writing them out anyways for ease of reference)

Loss to 20% propensity:
-2.32 * 100  +  -.32 * 200   = -296

Loss to 60% propensity:
-.74 * 100   +  -1.3 * 200   = -334

Now, if you are a naive six-year-old being led astray by an older child trolling you, you might look at this, and proclaim that the 20%-propensity hypothesis did much better than the 60%-propensity hypothesis, by a likelihood factor of over a hundred billion.

But really what this data is yelling at you is "The true hypothesis was not in your explicit hypothesis set!"

Why?  Consider about how well the two theories say they ought to do, loss-of-2s-wise, in the worlds where they are actually true:

Expected loss given 20% propensity:
-2.32 * 60  +  -.32 * 240   ~  -2&1/3 * 60 + -1/3 * 240  = -140 + -80  = -220

Expected loss given 60% propensity:
-.74 * 180  +  -1.32 * 120  ~  -3/4 * 180 + -1&1/3 * 120 = -135 + -160 = -295

...which is to say that both hypotheses lost way more than they expected to lose.

This is a sort of hint that tells you to look for a new hypothesis, like "1/3 propensity", say.  It holds even if you get the subtler hint that some hypothesis is doing way better than it expected to do, losing fewer twos than it said it should lose.

There's obviously ways to think about all the possible propensities at once, but Keltham doesn't think he can get to those this morning, given that they don't have calculus yet.

#3 - If every obvious hypothesis has unexpectedly low 'likelihood' over all the combined data, it means the true theory wasn't in your starting set, often that different experiments had different hidden conditions

Let's say you're testing (only) the hypotheses for 20% propensity-to-solve and 60% propensity-to-solve.

Suppose you test 300 subjects, and 100 of them solve it.

At this point, obviously, they're not going to be multiplying things out by hand any more, and are going to be working with logarithms...

log2 (0.2) = -2.322
log2 (0.4) = -1.322
log2 (0.6) = -.737
log2 (0.8) = -.322

(bonus points, Keltham declares, for anybody who already noticed that 2 entries in this table are blatantly redundant, but he's writing them out anyways for ease of reference)

2s lost, 20% propensity:
-2.32 * 100  +  -.32 * 200   = -296

2s lost, 60% propensity:
-.74 * 100   +  -1.3 * 200   = -334

Now, if you are a naive six-year-old being led astray by an older child trolling you, you might look at this, and proclaim that the 20%-propensity hypothesis did much better than the 60%-propensity hypothesis, by a likelihood factor of over a hundred billion.

But really what this data is yelling at you is "The true hypothesis was not in your explicit hypothesis set!"

Why?  Consider about how well the two theories say they ought to do, loss-of-2s-wise, in the worlds where they are actually true:

Expected 2s lost given 20% propensity:
-2.32 * 60  +  -.32 * 240   ~  -2&1/3 * 60 + -1/3 * 240  = -140 + -80  = -220

Expected 2s lost given 60% propensity:
-.74 * 180  +  -1.32 * 120  ~  -3/4 * 180 + -1&1/3 * 120 = -135 + -160 = -295

...which is to say that both hypotheses lost way more than they expected to lose.

This is a sort of hint that tells you to look for a new hypothesis, like "1/3 propensity", say.  It holds even if you get the subtler hint that some hypothesis is doing way better than it expected to do, losing fewer twos than it said it should lose.

There's obviously ways to think about all the possible propensities at once, but Keltham doesn't think he can get to those this morning, given that they don't have calculus yet.

#3 - If every obvious hypothesis has unexpectedly low 'likelihood' over all the combined data, it means the true theory wasn't in your starting set, often that different experiments had different hidden conditions

Let's say you're testing (only) the hypotheses for 20% propensity-to-solve and 60% propensity-to-solve.

Suppose you test 300 subjects, and 100 of them solve it.

At this point, obviously, they're not going to be multiplying things out by hand any more, and are going to be working with logarithms...

log2 (0.2) = -2.322
log2 (0.4) = -1.322
log2 (0.6) = -.737
log2 (0.8) = -.322

(bonus points, Keltham declares, for anybody who already noticed that 2 entries in this table are blatantly redundant, but he's writing them out anyways for ease of reference)

2s lost, 20% propensity:
-2.32 * 100  +   -.32 * 200   = -296

2s lost, 60% propensity:
-.74 * 100   +  -1.33 * 200   = -338

Now, if you are a naive six-year-old being led astray by an older child trolling you, you might look at this, and proclaim that the 20%-propensity hypothesis did much better than the 60%-propensity hypothesis, by a likelihood factor of over a hundred billion.

But really what this data is yelling at you is "The true hypothesis was not in your explicit hypothesis set!"

Why?  Consider about how well the two theories say they ought to do, loss-of-2s-wise, in the worlds where they are actually true:

Expected 2s lost given 20% propensity:
-2.32 * 60  +  -.32 * 240   ~  -2&1/3 * 60 + -1/3 * 240  = -140 + -80  = -220

Expected 2s lost given 60% propensity:
-.74 * 180  +  -1.32 * 120  ~  -3/4 * 180 + -1&1/3 * 120 = -135 + -160 = -295

...which is to say that both hypotheses lost way more than they expected to lose.

This is a sort of hint that tells you to look for a new hypothesis, like "1/3 propensity", say.  It holds even if you get the subtler hint that some hypothesis is doing way better than it expected to do, losing fewer twos than it said it should lose.

There's obviously ways to think about all the possible propensities at once, but Keltham doesn't think he can get to those this morning, given that they don't have calculus yet.

#3 - If every obvious hypothesis has unexpectedly low 'likelihood' over all the combined data, it means the true theory wasn't in your starting set, often that different experiments had different hidden conditions

Let's say you're testing (only) the hypotheses for 20% propensity-to-solve and 60% propensity-to-solve.

Suppose you test 300 subjects, and 100 of them solve it.

At this point, obviously, they're not going to be multiplying things out by hand any more, and are going to be working with logarithms...

log2 (0.2) = -2.322
log2 (0.4) = -1.322
log2 (0.6) = -.737
log2 (0.8) = -.322

(bonus points, Keltham declares, for anybody who already noticed that 2 entries in this table are blatantly redundant, but he's writing them out anyways for ease of reference)

2s lost, 20% propensity:
-2.32 * 100  +   -.32 * 200   = -296

2s lost, 60% propensity:
-.74 * 100   +  -1.33 * 200   = -338

Now, if you are a naive six-year-old being led astray by an older child trolling you, you might look at this, and proclaim that the 20%-propensity hypothesis did much better than the 60%-propensity hypothesis, by a likelihood factor of over a trillion.

But really what this data is yelling at you is "The true hypothesis was not in your explicit hypothesis set!"

Why?  Consider about how well the two theories say they ought to do, loss-of-2s-wise, in the worlds where they are actually true:

Expected 2s lost given 20% propensity:
-2.32 * 60  +  -.32 * 240   ~  -2&1/3 * 60 + -1/3 * 240  = -140 + -80  = -220

Expected 2s lost given 60% propensity:
-.74 * 180  +  -1.32 * 120  ~  -3/4 * 180 + -1&1/3 * 120 = -135 + -160 = -295

...which is to say that both hypotheses lost way more than they expected to lose.

This is a sort of hint that tells you to look for a new hypothesis, like "1/3 propensity", say.  It holds even if you get the subtler hint that some hypothesis is doing way better than it expected to do, losing fewer twos than it said it should lose.

There's obviously ways to think about all the possible propensities at once, but Keltham doesn't think he can get to those this morning, given that they don't have calculus yet.

#3 - If every obvious hypothesis has unexpectedly low 'likelihood' over all the combined data, it means the true theory wasn't in your starting set, often that different experiments had different hidden conditions

Let's say you're testing (only) the hypotheses for 20% propensity-to-solve and 60% propensity-to-solve.

Suppose you test 300 subjects, and 100 of them solve it.

At this point, obviously, they're not going to be multiplying things out by hand any more, and are going to be working with logarithms...

log2 (0.2) = -2.322
log2 (0.4) = -1.322
log2 (0.6) =  -.737
log2 (0.8) =  -.322

(bonus points, Keltham declares, for anybody who already noticed that 2 entries in this table are blatantly redundant, but he's writing them out anyways for ease of reference)

2s lost, 20% propensity:
-2.32 * 100  +   -.32 * 200   = -296

2s lost, 60% propensity:
-.74 * 100   +  -1.33 * 200   = -338

Now, if you are a naive six-year-old being led astray by an older child trolling you, you might look at this, and proclaim that the 20%-propensity hypothesis did much better than the 60%-propensity hypothesis, by a likelihood factor of over a trillion.

But really what this data is yelling at you is "The true hypothesis was not in your explicit hypothesis set!"

Why?  Consider about how well the two theories say they ought to do, loss-of-2s-wise, in the worlds where they are actually true:

Expected 2s lost given 20% propensity:
-2.32 * 60  +  -.32 * 240   ~  -2&1/3 * 60 + -1/3 * 240  = -140 + -80  = -220

Expected 2s lost given 60% propensity:
-.74 * 180  +  -1.32 * 120  ~  -3/4 * 180 + -1&1/3 * 120 = -135 + -160 = -295

...which is to say that both hypotheses lost way more than they expected to lose.

This is a sort of hint that tells you to look for a new hypothesis, like "1/3 propensity", say.  It holds even if you get the subtler hint that some hypothesis is doing way better than it expected to do, losing fewer twos than it said it should lose.

There's obviously ways to think about all the possible propensities at once, but Keltham doesn't think he can get to those this morning, given that they don't have calculus yet.