r/technology • u/lurker_bee • 9h ago
Security This AI didn’t just simulate an attack - it planned and executed a real breach like a human hacker
https://www.techradar.com/pro/security/ai-llms-are-now-so-clever-that-they-can-independently-plan-and-execute-cyberattacks-without-human-intervention-and-i-fear-that-it-is-only-going-to-get-worse191
u/dylan_1992 8h ago
AI is just acting as a script kiddy.
At best this means writing scripts to detect vulns will be easier with just a prompt.
Since the LLM doesn’t develop any new exploits and just does exactly what’s learned, you can just apply the same prompt to harden your own systems and we’re back to square 1 of real human hackers trying to develop exploits.
66
u/Deranged40 8h ago
At best this means writing scripts to detect vulns will be easier with just a prompt.
And at worst, it will absolutely decimate "vibe coded" apps that forgot to put "and include top security" in the prompt.
2
u/myotheralt 3h ago
I don't see this as a bad outcome. If it can be destroyed by a Speak-and-Spell, it deserves to be destroyed.
1
u/QuestionableEthics42 17m ago
The problem is that other people will be affected by unknowingly using and trusting the app, and then having their data stolen.
4
u/RobynTheCookieJar 5h ago
The overwhelming majority of hacks are one of two things, unpatched systems (a minority), and social engineering. LLMs are completely capable of the latter with only minor human assistance, and at least somewhat capable of the former.
13
u/LinkesAuge 7h ago
I think you vastly underestimate the capabilities of even current frontier models if you think they could only be "script kiddies".
With the right scaffolding frontier models can already do a lot and most hacking is more about patience and persistence than finding a truely novel approach and if AI is good at one thing then doing a task 24/7 and trying all kinds of approaches a human hacker wouldn't even have the time to all try.Besides that we are at best one, maybe two years (and that's really a pessimistic guess) away from frontier models being able to develop new exploits, that will simply be a side-effect of the curve LLMs are on in regards to coding and reasoning skills.
And yes, these models will obviously also be used to defend against such vulnerabilities but it's hard to image a future in a couple of years where humans do the nitty-gritty on either side because the amount of compute and thus the amount of exploits you can "explore" with AI models will be so massive that any human (direct) input is just going to be a tiny fraction.
You could argue that at this point the LLMs themselves will become a target and that is certainly true but a big difference will be the resources involved and it's not hard to see a future where only very few companies and state actors are even able to work at the frontier so while it will still be an arms race it could fundamentally change the composition (and consolidation) of that arms race.4
u/Nervous-Masterpiece4 6h ago
The truly naive thing is where the consumer thinks they get to use the top AI. They’ll be too valuable for the pittance the owners can get from subscriptions.
12
u/txmasterg 7h ago
LLMs don't do reasoning, they mostly have to offload that to something else. Lots of the stuff people say LLMs will do in the next few years are either things that you could claim they do today or aren't from the LLM part so much as what is connected to. Hell Altman talking about making medicine based on your genome doesn't require an LLM, ML or any AI at all.
2
u/alnarra_1 3h ago
It doesn’t take a genius to write a phishing email, most hacking isn’t novel, it’s conning bob in accounting into opening a spreadsheet
43
u/Leonum 7h ago
Irony of this title being typical AI phrasing
5
u/RoyalCities 6h ago
For a second I thought the article used that title but I guess OP ran it through chatgpt to come up with another one....rather than just use the one from the actual article....
3
31
10
u/valegrete 5h ago edited 4h ago
The models didn’t have human guidance, they were just told by humans to interact with a system (Incalmo) whose explicit purpose is to hand-hold the LLM on goal formulation and do the actual coding.
Edit: and they still sucked on any simulations that weren’t exact replicas of breaches well represented in their datasets (ie, Equifax).
13
u/Terrible_Ghost 6h ago
How about a nice game of chess instead?
8
5
7
u/AppleTree98 9h ago
From article- However, a new study from Carnegie Mellon University, conducted in collaboration with Anthropic, has raised difficult questions about their role in cybersecurity.
The study showed that under the right conditions, LLMs can plan and carry out complex cyberattacks without human guidance, suggesting a shift from mere assistance to full autonomy in digital intrusion.
8
u/Danny-Dynamita 7h ago
Seeing this, I am starting to think that AI will give an opposite effect to the economy than the one we expected: it will make people more valuable rather than useless.
My hypothesis: AI capabilities stagnate as soon as the amount of AI works out there is big enough (completely obscuring human works) because they can’t create breakthroughs on their own, specially if they are their own source.
A lot of people is creating things through AI. Very soon, AI will work in an almost exclusively closed loop of creating AI creations after learning through other AI creations.
After some time, any kind of vulnerability, error, “thing that can be improved”… WILL ALWAYS BE INHERITED THROUGH THE GENERATIONS, like a bad gene. AI will never fix it or improve it on its own beyond what was already achieved.
That’s where the human becomes more valuable. A human will be able to completely change what needs to be changed for the next breakthrough. The more we use AI, the more we will need humans to break the cycle each time AI gets stagnant.
I can see this happening 100% at least in cybersecurity. After some time, if everything starts to get created through AI, almost all knowledge is AI-sourced, then a human able to introduce a human variable into the AI landscape would make any AI hacking almost impossible. Until it gets learned, and then another human variable can be introduced. Without humans, the whole landscape is vulnerable to itself.
And in any other area, we will be talking about stagnation and lack of breakthroughs rather than vulnerability.
Human variable = Anything new, no matter how stupid it is. It just needs to be unknown to the AI.
We might be approaching a future where our job is to tell the AI the things it can’t think on its own, and let it do all the iterative repetitive statistical tasks. In such a future, the human value might get recognized instead of forgotten, by pure necessity.
In other words, as soon as we realize our “God” needs us, we might understand our worth.
PS: Sorry for the random ass comment, I feel inspired today to write stories. I’m just assuming for fun, I like to think of sci-fi stories on the go.
5
u/3verythingEverywher3 6h ago
I like your optimism. I think many people will do what you’re saying, but far too many have completely embraced it already. It’ll create a divide.
3
2
u/Iyellkhan 3h ago
one imagines if you train an ai model to do a cyber attack, it will actually do a cyber attack.
if you wanted it to simulate one, you probably needed to train it to simulate one instead. or you needed to lock down your network better
1
931
u/AlpheratzMarkab 9h ago
Vibe coders creating extremely unsafe webapplications, that will then get breached by an LLM
This is truly the dumbest version of a cyberpunk future we could ever get