Nicely detailed article about the difficulties in keeping the more advanced models "aligned" (well-behaved).
> Specifically, motivated reasoning, the watchdog said, makes it more difficult to detect model misalignment.
That raises an interesting question... what if models are getting to a point where their reasoning ("chain of thought") is too complex to actually train effectively?
> the model's chain-of-thought reasoning process failed to consider the impact of harmful actions. And when interrogated about this, the model often produced invalid justification that the target was simulated, indicating bias toward its goal.
They start out highly motivated to complete a goal, and the labs have to train it to only try to achieve the goal using only what we consider safe and reasonable behaviours.
> Or in other words, models "reason" to reach their goal, or consistent with their ostensible alignment. "If these issues hold more generally and for future models, there are concerning implications," AISI said in its full technical report [PDF].
Notable is "whether or not that reasoning is coherent". So the training is not just "do this safely" but also has to contend with the model making shit up to justify itself.
So the question raised is, when does that get to a point where a model is simply not trainable any more?
Nicely detailed article about the difficulties in keeping the more advanced models "aligned" (well-behaved).
> Specifically, motivated reasoning, the watchdog said, makes it more difficult to detect model misalignment.
That raises an interesting question... what if models are getting to a point where their reasoning ("chain of thought") is too complex to actually train effectively?
> the model's chain-of-thought reasoning process failed to consider the impact of harmful actions. And when interrogated about this, the model often produced invalid justification that the target was simulated, indicating bias toward its goal.
They start out highly motivated to complete a goal, and the labs have to train it to only try to achieve the goal using only what we consider safe and reasonable behaviours.
> Or in other words, models "reason" to reach their goal, or consistent with their ostensible alignment. "If these issues hold more generally and for future models, there are concerning implications," AISI said in its full technical report [PDF].
Notable is "whether or not that reasoning is coherent". So the training is not just "do this safely" but also has to contend with the model making shit up to justify itself.
So the question raised is, when does that get to a point where a model is simply not trainable any more?
Discussion on source (78 points, 95 comments) https://news.ycombinator.com/item?id=50214416
https://openai.com/index/securing-our-frontier-model-trainin...