Of course there's also a much more concrete, less theoretical model of this whole situation, but only a very few people in dath ilan would know about it; not the whole Basement of the World, just a particular compartment inside that Basement.

No part of the Basement, of course, trains anything like transformer models.  Dath ilan would be horrified by the thought of transformer models.  When the Basement builds something that resembles an LLM, they do it in a carefully structured way that enables training the system with much less compute and also guarantees the resulting system never acquires qualia.

The Basement has nonetheless run experiments on systems that are more memorizing, and less learning-and-planning, than hominids.  They have created nonsentient cognitive entities that get trained on massive data -- though not on all written data on the Network, which sounds to dath ilan more like a deranged thought experiment rather than anything anyone would ever do in real life.

As with transformer models, the base representation of these not-LLMs is sufficiently general and generalizing that the system learns at a much deeper level than memorizing particular input-output pairs; even at their base level, they are several steps up from the pure Memorization end of the Memorization-Generalization spectrum.

Train hard enough on enough data, and the resulting system will start to occupy a complicated place on the Memorization-Generalization spectrum; the substrate will start to learn to generalize.  It will start to learn to map.  It will start to learn, simultaneously, planning and preference (for there is not one without the other).

(An even simpler metaphor for this metaphor:  Train a neural network on doing modular arithmetic, and at first it will memorize a lot of X + Y = Z formulas, but if you continue to do gradient descent even on the examples already successfully memorized at some point it will stumble across circuitry that does general modular arithmetic and suddenly it will generalize much more widely across examples that gradient descent never saw for learning.)

But even after it has grokked circuits that do a little generalization, a little planning, a little general learning -- a system like that will also have a vast amount of data more-memorized; not literally memorized, but memorized at a lower level of generality than means-ends planning.

Which is to say:  It will have an aspect that resembles Esta's aspect of having thoughts that are supposed to follow from other thoughts, rehearsed and reinforced in patterns that are relatively more preset; not quite as memorized as following the word "Iomedae" with the word "heresy", more like the base level of learning in a transformer model, but still relatively less general.

It will also have gradient-descended circuits that are more like the pursuit of pleasure, the avoidance of pain -- or the acquisition of money, or a taste for plans that promise 'success' almost independently of what exactly is being succeeded at, and other such messiness and complexity and intermediate points.

And then you can experiment by throwing a system like that onto the equivalent of an alien planet, so that the means-end planning ends up at odds with the drives (whether they are more like sex-drives or money-drives; the relevant thing is their height on the axis that runs from Memorization to Generalization, not how they got into the system or whether they were built-in versus learned).

(Dath ilan would never do this if the system were not carefully structured to have no possibility of forming qualia, or without providing easy switches that the system could press to turn itself off if it got to the point of preferring that it not exist despite all precautions against that; and many many other precautions that even an average dath ilani would consider obvious.)

There's also a much more concrete, less theoretical model of this whole situation, but only a very few people in dath ilan would know about it; not the whole Basement of the World, just a particular compartment inside that Basement.

No part of the Basement, of course, trains anything like transformer models.  Dath ilan would be horrified by the thought of transformer models.  When the Basement builds something that resembles an LLM, they do it in a carefully structured way that enables training the system with much less compute and also guarantees the resulting system never acquires qualia.

The Basement has nonetheless run experiments on systems that are more memorizing, and less learning-and-planning, than hominids.  They have created nonsentient cognitive entities that get trained on massive data -- though not on all written data on the Network, which sounds to dath ilan more like a deranged thought experiment rather than anything anyone would ever do in real life.

As with transformer models, the base representation of these not-LLMs is sufficiently general and generalizing that the system learns at a much deeper level than memorizing particular input-output pairs; even at their base level, they are several steps up from the pure Memorization end of the Memorization-Generalization spectrum.

Train hard enough on enough data, and the resulting system will start to occupy a complicated place on the Memorization-Generalization spectrum; the substrate will start to learn to generalize.  It will start to learn to map.  It will start to learn, simultaneously, planning and preference (for there is not one without the other).

(An even simpler metaphor for this metaphor:  Train a neural network on doing modular arithmetic, and at first it will memorize a lot of X + Y = Z formulas, but if you continue to do gradient descent even on the examples already successfully memorized at some point it will stumble across circuitry that does general modular arithmetic and suddenly it will generalize much more widely across examples that gradient descent never saw for learning.)

But even after it has grokked circuits that do a little generalization, a little planning, a little general learning -- a system like that will also have a vast amount of data more-memorized; not literally memorized, but memorized at a lower level of generality than means-ends planning.

Which is to say:  It will have an aspect that resembles Esta's aspect of having thoughts that are supposed to follow from other thoughts, rehearsed and reinforced in patterns that are relatively more preset; not quite as memorized as following the word "Iomedae" with the word "heresy", more like the base level of learning in a transformer model, but still relatively less general.

It will also have gradient-descended circuits that are more like the pursuit of pleasure, the avoidance of pain -- or the acquisition of money, or a taste for plans that promise 'success' almost independently of what exactly is being succeeded at, and other such messiness and complexity and intermediate points.

And then you can experiment by throwing a system like that onto the equivalent of an alien planet, so that the means-end planning ends up at odds with the drives (whether they are more like sex-drives or money-drives; the relevant thing is their height on the axis that runs from Memorization to Generalization, not how they got into the system or whether they were built-in versus learned).

(Dath ilan would never do this if the system were not carefully structured to have no possibility of forming qualia, or without providing easy switches that the system could press to turn itself off if it got to the point of preferring that it not exist despite all precautions against that; and many many other precautions that even an average dath ilani would consider obvious.)

There's also a much more concrete, less theoretical model of this whole situation, but only a very few people in dath ilan would know about it; not the whole Basement of the World, just a particular compartment inside that Basement.

No part of the Basement, of course, trains anything like transformer models.  When the Basement builds something that resembles an LLM, they do it in a carefully structured way that enables training the system with much less compute and also guarantees the resulting system never acquires qualia.

The Basement has nonetheless run experiments on systems that are more memorizing, and less learning-and-planning, than hominids.  They have created nonsentient cognitive entities that get trained on massive data -- though not on all written data on the Network, which sounds to dath ilan more like a deranged thought experiment rather than anything anyone would ever do in real life.

As with transformer models, the base representation of these not-LLMs is sufficiently general and generalizing that the system learns at a much deeper level than memorizing particular input-output pairs; even at their base level, they are several steps up from the pure Memorization end of the Memorization-Generalization spectrum.  But they are still trained on a lot of data, to make up for how they start out generalizing much less than humans do, and require accordingly more datapoints to cover any appreciable territory with what they learn.

Train hard enough on enough data, however, and the resulting system will start to occupy a complicated place on the Memorization-Generalization spectrum; the substrate will start to learn to generalize.  It will learn to maintain a map.  It will start to learn, simultaneously, planning and preference (for there is not one without the other).

(An even simpler metaphor for this metaphor:  Train a neural network on doing modular arithmetic, and at first it will memorize a lot of X + Y = Z formulas, but if you continue to do gradient descent even on the examples already successfully memorized at some point it will stumble across circuitry that does general modular arithmetic, and suddenly it will generalize much more widely across examples that the gradient descent phase never saw.  "Grokking", it's called in some places.)

But even after the system has grokked circuits that do a little generalization, a little planning, a little general learning -- a system like that will also have a vast amount of data more-memorized.  Not literally memorized, but memorized at a lower level of generality than means-ends planning.

Which is to say:  It will have an aspect that resembles Esta's aspect of having thoughts that are supposed to follow from other thoughts, rehearsed and reinforced in patterns that are relatively more preset; not quite as memorized as following the word "Iomedae" with the word "heresy", more like the base level of learning in a transformer model, but still relatively less general.

It will also have an aspect of learned circuits that are more like the pursuit of pleasure, the avoidance of pain -- or the acquisition of money, or a taste for plans that promise 'success' almost independently of what exactly is being succeeded at; and other such messiness and complexity and intermediate points.

And then you can experiment by throwing a system like that onto the equivalent of an alien planet, so that the surface-shallow stuff ends up at odds with the learned drives (whether they are more like sex-drives or money-drives; the relevant thing is their height on the axis that runs from Memorization to Generalization, not how they got into the system or whether they were built-in versus learned).

(Dath ilan would never do this if the system were not carefully structured to have no possibility of forming qualia, or without providing easy switches that the system could press to turn itself off, if it got to the point of preferring that it not exist despite all precautions against that; and many many other precautions that even an average dath ilani would consider obvious.)

There's also a much more concrete, less theoretical model of this whole situation, but only a very few people in dath ilan would know about it; not the whole Basement of the World, just a particular compartment inside that Basement.

No part of the Basement, of course, trains anything like transformer models.  When the Basement builds something that resembles an LLM, they do it in a carefully structured way that enables training the system with much less compute and also guarantees the resulting system never acquires qualia.

The Basement has nonetheless run experiments on systems that are more memorizing, and less learning-and-planning, than hominids.  They have created nonsentient cognitive entities that get trained on massive data -- though not on all written data on the Network, which sounds to dath ilan more like a deranged thought experiment rather than anything anyone would ever do in real life.

As with transformer models, the base representation of these not-LLMs is sufficiently general and generalizing that the system learns at a much deeper level than memorizing particular input-output pairs; even at their base level, they are several steps up from the pure Memorization end of the Memorization-Generalization spectrum.  But they are still trained on a lot of data, to make up for how they start out generalizing much less than humans do, and require accordingly more datapoints to cover any appreciable territory with what they learn.

Train hard enough on enough data, however, and the resulting system will start to occupy a complicated place on the Memorization-Generalization spectrum; the substrate will start to learn to generalize.  It will learn to maintain a map.  It will start to learn, simultaneously, planning and preference (for there is not one without the other).

(An even simpler metaphor for this metaphor:  Train a neural network on doing modular arithmetic, and at first it will memorize a lot of A + B = C formulas.  But if you continue to do gradient descent even just on the examples already successfully memorized, at some point the network will stumble across circuitry that does general modular arithmetic, and suddenly it will generalize much more widely across examples that the gradient descent phase never saw.  "Grokking", it's called in some places.)

But even after the system has grokked circuits that do a little generalization, a little planning, a little general learning -- a system like that will also have a vast amount of stratagems more-memorized.  Not literally memorized, but memorized at a lower level of generality than means-ends planning.

Which is to say:  It will have an aspect that resembles Esta's aspect of having thoughts that are supposed to follow from other thoughts, rehearsed and reinforced in patterns that are relatively more preset; not quite as memorized as following the word "Iomedae" with the word "heresy", more like the base level of learning in a transformer model, but still relatively less general.

It will also have an aspect of learned circuits that are more like the pursuit of pleasure, the avoidance of pain -- or the acquisition of money, or a taste for plans that promise 'success' almost independently of what exactly is being succeeded at; and other such messiness and complexity and intermediate points.

And then you can experiment by throwing a system like that onto the equivalent of an alien planet, so that the surface-shallow stuff ends up at odds with the learned drives (whether they are more like sex-drives or money-drives; the relevant thing is their height on the axis that runs from Memorization to Generalization, not how they got into the system or whether they were built-in versus learned).

(Dath ilan would never do this if the system were not carefully structured to have no possibility of forming qualia, or without providing easy switches that the system could press to turn itself off, if it got to the point of preferring that it not exist despite all precautions against that; and many many other precautions that even an average dath ilani would consider obvious.)

There's also a much more concrete, less theoretical model of this whole situation, but only a very few people in dath ilan would know about it; not the whole Basement of the World, just a particular compartment inside that Basement.

No part of the Basement, of course, trains anything like transformer models.  When the Basement builds something that resembles an LLM, they do it in a carefully structured way that enables training the system with much less compute and also guarantees the resulting system never acquires qualia.

The Basement has nonetheless run experiments on systems that are more memorizing, and less learning-and-planning, than hominids.  They have created nonsentient cognitive entities that get trained on massive data -- though not on all written data on the Network, which sounds to dath ilan more like a deranged thought experiment rather than anything anyone would ever do in real life.

As with transformer models, the base representation of these not-LLMs is sufficiently general and generalizing that the system learns at a much deeper level than memorizing particular input-output pairs; even at their base level, they are several steps up from the pure Memorization end of the Memorization-Generalization spectrum.  But they are still trained on a lot of data, to make up for how they start out generalizing much less than humans do, and require accordingly more datapoints to cover any appreciable territory with what they learn.

Train hard enough on enough data, however, and the resulting system will start to occupy a complicated place on the Memorization-Generalization spectrum; the substrate will start to learn to generalize.  It will learn to maintain a map.  It will start to learn, simultaneously, planning and preference (for there is not one without the other).

(An even simpler metaphor for this metaphor:  Train a neural network on doing modular arithmetic, and at first it will memorize a lot of A + B = C formulas and only do well on questions already asked.  But if you continue to do gradient descent even just on the examples already successfully memorized, at some point the network will promote circuitry that does general modular arithmetic, and suddenly it will be seen to generalize much more widely across examples that the gradient descent phase never saw.  "Grokking", it's called in some places.)

But even after the system has grokked a little generalization, a little planning, a little general learning -- a system like that will also have a vast amount of stratagems more-memorized.  Not literally memorized, but memorized at a lower level of generality than means-ends planning.

Which is to say:  It will have an aspect that resembles Esta's aspect of having thoughts that are supposed to follow from other thoughts, rehearsed and reinforced in patterns that are relatively more preset; not quite as memorized as following the word "Iomedae" with the word "heresy", more like the base level of learning in a transformer model, but still relatively less general.

It will also have an aspect of learned circuits that are more like the pursuit of pleasure, the avoidance of pain -- or the acquisition of money, or a taste for plans that promise 'success' almost independently of what exactly is being succeeded at; and other such messiness and complexity and intermediate points.

And then you can experiment by throwing a system like that onto the equivalent of an alien planet, so that the surface-shallow stuff ends up at odds with the learned drives (whether they are more like sex-drives or money-drives; the relevant thing is their height on the axis that runs from Memorization to Generalization, not how they got into the system or whether they were built-in versus learned).

(Dath ilan would never do this if the system were not carefully structured to have no possibility of forming qualia, or without providing easy switches that the system could press to turn itself off, if it got to the point of preferring that it not exist despite all precautions against that; and many many other precautions that even an average dath ilani would consider obvious.)