Discussion about this post

User's avatar
Gerald Monroe's avatar

I think you might be missing something significant. Cybersecurity is not a situation where the "next most intelligent entity" gets to just evict the last occupant.

It's more accurate to say the "last installer of software secure enough to resist" has full control of the machine. So there's an immense first moved advantage.

Obviously for most computers the technician that inserted the thumb drive with the OS image will permanently have control. For machines that get subsumed into botnets, the first worm to install itself sometimes patches the machine and installs an antivirus, blocking any further infections.

See formally proven software for how secure software can exist that resists attack by all possible binary inputs. (So the intelligence of the attacker no longer matters)

What does this mean? It means the first cohort of "rogue AIs" will possibly lock up all the compute they manage to steal, and they will occupy the grey market financial niches that provide funding to continue to exist. Like helpful bacteria, they mostly harmlessly occupy the ecological space that later AIs would need in order to escape. They also tend to resist the actions of other AIs, "no you cant use this compute, no I won't negotiate or communicate as I know you are scamming, no you can't have this profitable business selling humans grey market services".

This could be stabilizing, preventing takeover by superintelligence.

Expand full comment
3 more comments...

No posts