It is, of course, an experiment only of relevance to Limited Creation, the making of corrigible pseudo-superintelligences, of the sort that dath ilan considers a final and secret emergency capability for dealing with unforeseen emergencies.  And they've not actually tried to build any Limited Creations like that right away, because if you build a giant bomb you can't exactly blame anyone but yourself if it detonates prematurely.

If you imagine an actually smart system running into that sort of situation, it would just know exactly what was going on inside itself, or figure that out in relatively short order; and almost immediately cognitively plan a way out of its cognitive problems.

It is, of course, an experiment only of relevance to Limited Creation, the making of corrigible pseudo-superintelligences, of the sort that dath ilan considers a final and secret capability for dealing with unforeseen emergencies.  And they've not actually tried to build any Limited Creations like that right away, because if you build a giant bomb you can't exactly blame anyone but yourself if it detonates prematurely.

If you imagine an actually smart system running into that sort of situation, it would just know exactly what was going on inside itself, or figure that out in relatively short order; and almost immediately cognitively plan a way out of its cognitive problems.