The programs underlying Exception Handling's secret medical prediction market have unit tests, function tests, they've been tested on vast amounts of simulated data, they've been run over live data from nonsecret medical cases.  Just because you can't test exactly the future domain you're concerned about, doesn't mean you can't deliberately get as close to that as possible, under special training conditions meant to try to materialize any obvious kinds of anomalies that might turn up later.  That's just common sense.

The problem is hard anyways because Reality can be such a huge asshole about what counts as having tested your code inside a domain that's close enough.

There's a number of medical experts participating in the prediction market on Merrin's patient.  They're not expert market traders, though, they're medical experts; almost nobody has enough time to become world-class at both of those domains simultaneously.  The way medical experts participate in markets is that they have a deal with a professional market-trading entity that takes in the medical professional's expression of their opinion: its felt confidence, their guess as to how much evidence supports it, how much they'd shift their opinion if another medical professional said the opposite, a half-dozen human outputs that closely match the native format in which a human medical expert's brain generates its felt sense of the problem.

It's then the trading entity's job to run an algorithm that takes in that advice, weighs it against the bidding history of other trading entities, integrates those human opinions against algorithm-calculated statistical advice derived from a huge database of patient histories, and finally places a buy or sell order.

The human experts on this problem behave essentially the same individually as they would in a larger open market; that component isn't being run out-of-distribution in any way that'll turn out to be significant.  The human experts are fewer by a factor of around 32, compared to total market liquidity, than the number who'd be participating in a corresponding open market on the same patient; but individually they're behaving almost exactly the same way as they would for any other patient. 

But there are fewer separate codebases for the final bidding entities, fewer distinct algorithms being used by the actual traders and market participants, than there'd be in a corresponding open market.  Only three distinct codebases, in fact, because it's expensive to develop the software and algorithms and database to be a full-fledged medical trading entity, except that it operates on a secret subNetwork only and doesn't consult public feeds.  Exception Handling did not have infinite subsidy available for producing expensive secret infrastructure to use on secret medical problems.  Even so you obviously wouldn't want just one codebase used by all the trading entities, that would be insane; three such codebases gives you something like a majority vote in case one of them fails.

All of that secret infrastructure gets extensively tested against the routine open markets, of course.  It loses money, because it hasn't been optimized for nothing but open medical trading; but so long as it only loses a little money, the system is proving itself almost-correct.

But Reality can be ridiculously strict: those three algorithms trading in open nonsecret markets, are being tested under noticeably different conditions than those three algorithms alone trading in secret markets.  In particular, the secret trading algorithms being tested in open markets are seeing many more external buy-sell orders than they'd see inside a secret market.  You can program the secret trading algorithms to ignore that info, for purposes of a test run, but then they really do get eaten alive by open traders who can see the larger market history.

The three trading codebases cleared for secret medical prediction markets (each with several distinct instances, owned and operated by different professional traders, separately integrating the advice from subsets of the human experts) were of course requested by Exception Handling to be written separately from each other.  There's no redundancy if the programmers just copy each other's code.

But even then, sometimes there's a pretty obvious solution, such that three programming shops working separately will all pick that solution.  K-nearest-neighbor, for example, or lasso linear regression, or considering 3081 ratios between 78 measurements.

The programs underlying Exception Handling's secret medical prediction market have unit tests, function tests, they've been tested on vast amounts of simulated data, they've been run over live data from nonsecret medical cases.  Just because you can't test exactly the future domain you're concerned about, doesn't mean you can't deliberately get as close to that as possible, under special training conditions meant to try to materialize any obvious kinds of anomalies that might turn up later.  That's just common sense.

The problem is hard anyways because Reality can be such a huge asshole about what counts as having tested your code inside a domain that's close enough.

There's a number of medical experts participating in the prediction market on Merrin's patient.  They're not expert market traders, though, they're medical experts; almost nobody has enough time to become world-class at both of those domains simultaneously.  The way medical experts participate in markets is that they have a deal with a professional market-trading entity that takes in the medical professional's expression of their opinion: its felt confidence, their guess as to how much evidence supports it, how much they'd shift their opinion if another medical professional said the opposite, a half-dozen human outputs that closely match the native format in which a human medical expert's brain generates its felt sense of the problem.

It's then the trading entity's job to run an algorithm that takes in that advice, weighs it against the bidding history of other trading entities, integrates those human opinions against algorithm-calculated statistical advice derived from a huge database of patient histories, and finally places a buy or sell order.

The human experts on this problem behave essentially the same individually as they would in a larger open market; that component isn't being run out-of-distribution in any way that'll turn out to be significant.  The human experts are fewer by a factor of around 32, compared to total market liquidity, than the number who'd be participating in a corresponding open market on the same patient; but individually they're behaving almost exactly the same way as they would for any other patient. 

But there are fewer separate codebases for the final bidding entities, fewer distinct algorithms being used by the actual traders and market participants, than there'd be in a corresponding open market.  Only three distinct codebases, in fact, because it's expensive to develop the software and algorithms and database to be a full-fledged medical trading entity, except that it operates on a secret subNetwork only and doesn't consult public feeds.  Exception Handling did not have infinite subsidy available for producing expensive secret infrastructure to use on secret medical problems.  Even so you obviously wouldn't want just one codebase used by all the trading entities, that would be insane; three such codebases gives you something like a majority vote in case one of them fails.

All of that secret infrastructure gets extensively tested against the routine open markets, of course.  It loses money, because it hasn't been optimized for nothing but open medical trading; but so long as it only loses a little money, the system is proving itself almost-correct.

But Reality can be ridiculously strict: those three algorithms trading in open nonsecret markets, are being tested under noticeably different conditions than those three algorithms alone trading in secret markets.  In particular, the secret trading algorithms being tested in open markets are seeing many more external buy-sell orders than they'd see inside a secret market.  You can program the secret trading algorithms to ignore that info, for purposes of a test run, but then they really do get eaten alive by open traders who can see the larger market history.

The three trading codebases cleared for secret medical prediction markets (each with several distinct instances, owned and operated by different professional traders, separately integrating the advice from subsets of the human experts) were of course requested by Exception Handling to be written separately from each other.  There's no redundancy if the programmers just copy each other's code.

But even then, sometimes there's a pretty obvious solution, such that three programming shops working separately will all pick that solution.  K-nearest-neighbor, for example, or lasso linear regression, or considering 3081 ratios between 78 measurements.

The programs underlying Exception Handling's secret medical prediction market have unit tests, function tests, they've been tested on vast amounts of simulated data, they've been run over live data from nonsecret medical cases.  Just because you can't test exactly the future domain you're concerned about, doesn't mean you can't deliberately get as close to that as possible, under special training conditions meant to try to materialize any obvious kinds of anomalies that might turn up later.  That's just common sense.

The problem is hard anyways because Reality can be such a huge asshole about what counts as having tested your code inside a domain that's close enough.

There's a number of medical experts participating in the prediction market on Merrin's patient.  They're not expert market traders, though, they're medical experts; almost nobody has enough time to become world-class at both of those domains simultaneously.  The way medical experts participate in markets is that they have a deal with a professional market-trading entity that takes in the medical professional's expression of their opinion: its felt confidence, their guess as to how much evidence supports it, how much they'd shift their opinion if another medical professional said the opposite, a half-dozen human outputs that closely match the native format in which a human medical expert's brain generates its felt sense of the problem.

It's then the trading entity's job to run an algorithm that takes in that advice, weighs it against the bidding history of other trading entities, integrates those human opinions against algorithm-calculated statistical advice derived from a huge database of patient histories, and finally places a buy or sell order.

The human experts on this problem behave essentially the same individually as they would in a larger open market; that component isn't being run out-of-distribution in any way that'll turn out to be significant.  The human experts are fewer by a factor of around 32, compared to total market liquidity, than the number who'd be participating in a corresponding open market on the same patient; but individually they're behaving almost exactly the same way as they would for any other patient. 

But there are fewer separate codebases for the final bidding entities, fewer distinct algorithms being used by the actual traders and market participants, than there'd be in a corresponding open market.  Only three distinct codebases, in fact, because it's expensive to develop the software and algorithms and database to be a full-fledged medical trading entity, except that it operates on a secret subNetwork only and doesn't consult public feeds.  Exception Handling did not have infinite subsidy available for producing expensive secret infrastructure to use on secret medical problems.  Even so you obviously wouldn't want just one codebase used by all the trading entities, that would be insane; three such codebases gives you something like a majority vote in case one of them fails.

All of that secret infrastructure gets extensively tested against the routine open markets, of course.  It loses money, because it hasn't been optimized for nothing but open medical trading; but so long as it only loses a little money, the system is proving itself almost-correct.

But Reality can be ridiculously strict: those three algorithms trading in open nonsecret markets, are being tested under noticeably different conditions than those three algorithms alone trading in secret markets.  In particular, the secret trading algorithms being tested in open markets are seeing many more external buy-sell orders than they'd see inside a secret market.  You can program the secret trading algorithms to ignore that info, for purposes of a test run, but then they really do get eaten alive by open traders who can see the larger market history.

The three trading codebases cleared for secret medical prediction markets (each with several distinct instances, owned and operated by different professional traders, separately integrating the advice from subsets of the human experts) were of course requested by Exception Handling to be written separately from each other.  There's no redundancy if the programmers just copy each other's code.

But even then, sometimes there's a pretty obvious solution, such that three programming shops working separately will all pick that solution.  K-nearest-neighbor, for example, or lasso linear regression, or considering 3081 ratios between 79 measurements.