"To be clear, there is a fundamental vulnerability of this sort, but you can't exploit it just by being rich.  The Hypothetical Corrupted Keepers could prevent Civilization from doing anything they didn't want Civilization to do, by bidding in the prediction markets, openly as the Keepers, to say that dreadful bad results will happen if Civilization does that.  Relatively few people would want to bet against the Keepers - not because it's illegal, to be clear, but because you'd probably lose all your money unless you're Literally Nemamel."

"Then Civilization doesn't do that thing, and the market on that policy never resolves, and the Keepers don't lose any money for lying that way."

"This is a central problem in all of decision theory - that we never get to observe the results of the actions we don't take.  We never get to see our expected utility estimates tested for everything we thought had less than optimal utility.  There's a lot of places where Civilization solves that problem by randomizing slightly, if doing slightly the wrong thing won't be catastrophic.  But the really important predictions - you don't do that there.  And that does open you up to assault by the Hypothetical Corrupted Keepers, or other actors inside Civilization who can afford to put up a lot of money and come up with persuasive arguments and have enough of a reputation to make other traders nervous."

"But it requires acting openly, to make the other traders nervous to bet against you; there's a limit to how much you can corrupt Civilization that way, because if you do it a lot, people will be suspicious.  And not just anyone can do it, either."

"To be clear, there is a fundamental vulnerability of this sort, but you can't exploit it just by being rich.  The Hypothetical Corrupted Keepers could prevent Civilization from doing anything they didn't want Civilization to do, by bidding in the prediction markets, openly as the Keepers, to say that dreadful bad results will happen if Civilization does that.  Relatively few people would want to bet against the Keepers - not because it's illegal, to be clear, but because you'd probably lose all your money unless you're Literally Nemamel."

"Then Civilization doesn't do that thing, and the market on that policy never resolves, and the Keepers don't lose any money for lying that way."

"This is a central problem in all of decision theory - that we never get to observe the results of the actions we don't take.  We never get to see our expected utility estimates tested for everything we thought had less than optimal utility.  There's a lot of places where Civilization solves that problem by randomizing slightly, if doing slightly the wrong thing won't be catastrophic.  But the really important predictions - you don't do that there.  And that does open you up to assault by the Hypothetical Corrupted Keepers, or other actors inside Civilization who can afford to put up a lot of money and come up with persuasive arguments and have enough of a reputation to make other traders nervous."

"But it requires acting openly, to make the other traders nervous to bet against your reputation.  There's a limit to how much you can corrupt Civilization's decisions and markets that way, because if you do it a lot, people will be suspicious.  And not just anyone can do it, either."

"...that said, yes, the Keepers have bid against things in inscrutable ways, now and then.  It wouldn't be an especially prosocial act to compile a public list of all the times they've done that, but they've done that even in markets I've been tracking.  To this day I have absolutely no idea why the Keepers think long-term-damaging unspecified 'people will vote that was a bad idea later' consequences will happen, if Civilization makes a harder push on teaching average kids more 'computer-science' once my generation's kids are slightly smarter.  I mean, it's very credible that 'computer-science' reshapes some people's thoughts in some internally-damaging direction, which the Keepers would rather not point out explicitly for obvious reasons.  It doesn't obviously fit into any plan of corrupt world domination.  But... yeah, what the Keepers bid against, largely doesn't get done, and if they were Hypothetically Corrupted, they could in fact be steering Civilization that way."

"To be clear, there is a fundamental vulnerability of this sort, but you can't exploit it just by being rich.  The Hypothetical Corrupted Keepers could prevent Civilization from doing anything they didn't want Civilization to do, by bidding in the prediction markets, openly as the Keepers, to say that dreadful bad results will happen if Civilization does that.  Relatively few people would want to bet against the Keepers - not because it's illegal, to be clear, but because you'd probably lose all your money unless you're Literally Nemamel."

"Then Civilization doesn't do that thing, and the market on that policy never resolves, and the Keepers don't lose any money for lying that way."

"This is a central problem in all of decision theory - that we never get to observe the results of the actions we don't take.  We never get to see our expected utility estimates tested for everything we thought had less than optimal utility.  There's a lot of places where Civilization solves that problem by randomizing slightly, if doing slightly the wrong thing won't be catastrophic.  But the really important predictions - you don't do that there.  And that does open you up to assault by the Hypothetical Corrupted Keepers, or other actors inside Civilization who can afford to put up a lot of money and come up with persuasive arguments and have enough of a reputation to make other traders nervous."

"But it requires acting openly, to make the other traders nervous to bet against your reputation.  There's a limit to how much you can corrupt Civilization's decisions and markets that way, because if you do it a lot, people will be suspicious.  And not just anyone can do it, either."

"...that said, yes, the Keepers have bid against things in inscrutable ways, now and then.  It wouldn't be an especially helpful act to compile a public list of all the times they've done that, but they've done that even in markets I've been tracking.  To this day I have absolutely no idea why the Keepers think unspecified long-term 'people will later vote that was a bad idea' consequences will happen, if Civilization makes a harder push on teaching average kids more 'computer-science' once my generation's kids are slightly smarter.  I mean, it's very credible that 'computer-science' reshapes some people's thoughts in some internally-damaging direction, which the Keepers would rather not point out explicitly for obvious reasons.  It doesn't obviously fit into any plan of corrupt world domination.  But... yeah, what the Keepers bid against, largely doesn't get done, and if they were Hypothetically Corrupted, they could in fact be steering Civilization that way."

"To be clear, there is a fundamental vulnerability of this sort, but you can't exploit it just by being rich.  The Hypothetical Corrupted Keepers could prevent Civilization from doing anything they didn't want Civilization to do, by bidding in the prediction markets, openly as the Keepers, to say that dreadful bad results will happen if Civilization does that.  Relatively few people would want to bet against the Keepers - not because it's illegal, to be clear, but because you'd probably lose all your money unless you're Literally Nemamel."

"Then Civilization doesn't do that thing, and the market on that policy never resolves, and the Keepers don't lose any money for lying that way."

"This is a central problem in all of decision theory - that we never get to observe the results of the actions we don't take.  We never get to see our expected utility estimates tested for everything we thought had less than optimal utility.  There's a lot of places where Civilization solves that problem by randomizing slightly, if doing slightly the wrong thing won't be catastrophic.  But the really important predictions - you don't do that there.  And that does open you up to assault by the Hypothetical Corrupted Keepers, or other actors inside Civilization who can afford to put up a lot of money and come up with persuasive arguments and have enough of a reputation to make other traders nervous."

"But it requires acting openly, to make the other traders nervous to bet against your reputation.  There's a limit to how much you can corrupt Civilization's decisions and markets that way, because if you do it a lot, people will be suspicious.  And not just anyone can do it, either."

"...that said, yes, the Keepers have bid against things in inscrutable ways, now and then.  It wouldn't be an especially helpful act to compile a public list of all the times they've done that, but they've done that even in markets I've been tracking.  To this day I have absolutely no idea why the Keepers fear long-term consequences specified to the rest of us only as 'people will later vote that was a bad idea', if Civilization makes a harder push on teaching average kids more 'computer-science' once my generation's kids are slightly smarter.  I mean, it's very credible that 'computer-science' reshapes some people's thoughts in some internally-damaging direction, which the Keepers would rather not point out explicitly for obvious reasons.  It doesn't obviously fit into any plan of corrupt world domination.  But... yeah, what the Keepers bid against, largely doesn't get done, and if they were Hypothetically Corrupted, they could in fact be steering Civilization that way."