When there are appropriate advances in physical substrate for AI (or computing in general) the risk will be more intuitive and alignment will grow quickly as a field of study. It's possible that all the work, research and discourse on alignment until then will be obsolete. Calling current alignment work a scam is a stretch though, who is benefitting from the scam?
Sep 1, 2023·edited Sep 1, 2023Liked by Joe Canimal
> "First, computers are in fact powerless because all one needs to do is turn them off"
This is a common but naive argument. If a model had the ability to spread on the internet like a virus it would be very difficult to eradicate.
> "Second, even ignoring our complete control over the physical substrate required to run any AI, there would still be no realistic prospect of a "foom" scenario in which some privileged program or other learns the hidden key to practically-infinite intelligence and thus forever imprisons the world. Instead, all indications are that we’ll see a more or less steady increase in capabilities, more or less mirrored by various sites around the world."
There definitely is evidence that "foom" or rapid self-improvement is possible. AlphaZero went from zero to superhuman chess ability in just 4 hours. While I think rapid AI self-improvement is a major risk because it could cause AI to act too quickly for us to respond, it is not necessary for AI to be an existential risk.
A steady increase in capabilities is still dangerous when the AI exceeds human intelligence because the AI's intelligence could be misused or its high intelligence may lead to behavior that humans can't understand or anticipate.
When there are appropriate advances in physical substrate for AI (or computing in general) the risk will be more intuitive and alignment will grow quickly as a field of study. It's possible that all the work, research and discourse on alignment until then will be obsolete. Calling current alignment work a scam is a stretch though, who is benefitting from the scam?
> "First, computers are in fact powerless because all one needs to do is turn them off"
This is a common but naive argument. If a model had the ability to spread on the internet like a virus it would be very difficult to eradicate.
> "Second, even ignoring our complete control over the physical substrate required to run any AI, there would still be no realistic prospect of a "foom" scenario in which some privileged program or other learns the hidden key to practically-infinite intelligence and thus forever imprisons the world. Instead, all indications are that we’ll see a more or less steady increase in capabilities, more or less mirrored by various sites around the world."
There definitely is evidence that "foom" or rapid self-improvement is possible. AlphaZero went from zero to superhuman chess ability in just 4 hours. While I think rapid AI self-improvement is a major risk because it could cause AI to act too quickly for us to respond, it is not necessary for AI to be an existential risk.
A steady increase in capabilities is still dangerous when the AI exceeds human intelligence because the AI's intelligence could be misused or its high intelligence may lead to behavior that humans can't understand or anticipate.