Paul McCarty 0:00

I think so. Hello.

Jenn Gile 0:02

It is Wednesday, September 16th, and uh we have three topics on our agenda to talk about. Uh interestingly, none of them are about DPRK or Paul and Rider. Uh so if you've gotten tired of us talking about those topics, good news. We're not gonna talk about them today. I mean, no promises, but we're not planning on it. How's it going, Paul?

Paul McCarty 0:26

I'm not probably I'm not promising anything. I'm probably gonna talk about Paul and Rider. There's a lot going on in the space, so there you go.

Jenn Gile 0:34

So uh late last week, I was heading back from Europe and uh in the absolute appsec community, somebody shared a uh a URL and said, curious what you think about this. And I'll um drop it in the chat. But essentially, this is new research that has come out about the GemStuffer campaign in RubyGems. So let me see here. I can click, click, click. Here's the research if you haven't seen it. The TLDR is there was a big attack on RubyGems back in May. Uh, it was unattributed at the time, it was very spammy. Uh, these researchers are alleging that this campaign was done by rogue open AI agents. And um, while we don't, I guess, categorically know if that's true, I think it also wouldn't surprise us. So they cited uh three main reasons that they think an open AI agent swarm was responsible for the incident. First, they said the packages are clearly LLM authored. I'm not sure that I agree that that's evidence that it was open AI agents, but sure. Um, the second is they're saying the agents self-identified as being from open AI. Okay, that's stronger evidence. Um, many of the packages uploaded contain an OAI in their name. Um, several have that as their author. Uh, one lists something similar as an email contact. And then the third reason that they're attributing this to an uh OpenAI swarm is they're saying it's behaving similarly, very extremely similarly, to a German wiki agents uh incident that they found earlier in the year uh where agents accessed 49 of the same files as wiki agents, and OpenAI did confirm that that was their attack. Um, so before we talk about the part B of this, that is hot news that just came out like an hour ago, um, Paul, what's your reactions on this Ruby hack um disclosure?

Paul McCarty 2:44

Yeah, well, I mean, I've been there since day one. Um, we've been identifying this stuff and pulling it into OSM. And um, so I saw the original, I was like right on it when the original 150 were found in May, um, the original batch, and then there was a second wave of about 20, roughly 2200. Um, and then the most recent article from who is that that did the article yesterday? It wasn't J Frog. JFrog, thank you. Um, they had the total number at about 3,000. Right now, I've got the total number about 2879, so there's a bit of a discrepancy there, but that's beside the point. It came in waves, and those different waves, the important thing here is that different waves were different, right? So there's different stuff in it, and so you know, first you kind of you see the first wave, and it's like clearly X-filling a lot of stuff from UK websites, UK government websites. You're like, what the freak? Why is it doing this? Like, and so then later on, months later, when you hear this open AI story, and because at the time when we were looking at it, all the researchers that were looking at it were like, this is weird. This is like, you know, AI's got to be doing this for some reason, we just didn't know. And now, you know, hindsight, as they say, is 2020. It makes a lot of sense that if the agent didn't have access to it, this is the way that it figured out how to get access to the websites, and it uh just the whole thing makes sense now when it didn't make a lot of things.

Jenn Gile 4:12

It's weird that it makes sense, right?

Paul McCarty 4:14

Yeah, because let's be clear like we had a hard time then calling these malicious packages, right? There's there's a couple of them that had payloads in them. There's like a very, very small single-digit number of them that had proper payloads in them, but the vast majority of the rest were not malicious. And so then, you know, what do we call it? A spam campaign. So, anyhow.

Jenn Gile 4:34

Um look at this. Our uh integration with LinkedIn is finally working. So uh I can pop up some comments. Hello, Rich, who just popped in here.

Paul McCarty 4:45

That's so cool. This is one of my old hockey buddies from Park City. We played uh we played hockey together back in the day. I love it.

Jenn Gile 4:53

Small world. Uh, we also have a little note from Andrew here. Uh, artificial intelligence doesn't do everything just yet. That's true. Um, but here's the part two of this story. So your buddy Luke over at Truffle, uh, the two of you have kind of been chit-chatting the last couple of days. And uh, as he promised us, uh a blog has gone live um kind of connecting how these agents uh initially got some access through a vulnerability that he discovered, disclosed, and helped Rumi Gems um uh address. Uh before we talk about it, I'm gonna say it's perhaps notable but not surprising that we have not heard anything from OpenAI on this. You know, it's been not quite a week um since this Ruby AI or Ruby hack blog came out. Um, they've been silent. Uh Luke, you know, I know would have liked to have gotten a comment from OpenAI, and it doesn't seem like that's happened. So I don't know. Um, we'd love to see better from them, even if it's them saying, yeah, we don't think we did it, which um I don't know that that's what we would expect right now. So anyway, uh rant aside, um what Luke over at Prepple has talked about is that the um and I'm pulling up here, um they found this uh vulnerability. I'm looking for what it was called. Uh you might remember off the top of your head, Paul.

Paul McCarty 6:28

Yeah, it was it was basically a CDN bug. It was it was a misconfiguration between the Ruby gems caching and their fastly CDN provider. And at the heart of it um was the fact that it was it was you when you when you hit it, it would just spit back somebody else's API, which you could then immediately use to say, hey, what packages no so background here is that this is for access to RubyGems, so just like npm and PyPy, you connect to it with the equivalent of a pat or something similar, and then you can push publish and you know all those things that you can do and unpublish, what have you. And so you'd get back somebody else's API, you'd be like, list my RubyGems packages, and you'd get somebody else's packages back to you. Those aren't my packages, those I couldn't do those. Whoops. Whoops. Um, so Luke found this in July, and this work is all his entirely, so like I'm not taking any credit for this whatsoever. This is entirely Luke, um, Luke Marshall, um, and super smart cat. I've been working with Luke. Um, I'm gonna see him uh next week, too, actually at Canberra. But um uh I've been working with him and chatting with him for a few years now, a very smart cat. But the one the way that OSM got involved, Jen, is that we kind of worked with the RubyGems team and and truffle security to kind of get us all together in the same room. Because meanwhile, we just had these things streaming, our own research was pulling these things into OSM and we trying to figure out what the heck is this, and so there's just it was really good to have us all kind of in the same virtual room trying to better understand this. The problem is, you know who was in that room to your point, you said earlier, open AI was not in that room, and um and now I have more questions than I've got answers, Jen. It's crazy.

Jenn Gile 8:18

Uh, so does Luke. He ends his piece with uh a lot of what do we know, what do we not know? Um again, open AI has not confirmed that agents created or published the packages. Um he mentioned that uh the source adds another complication. You know, the fallback was to an embedded key if it finds no token in the response, and even a successful upload by itself wouldn't establish that it used someone's credential. Um, so you know, a little interesting, you know, the packages don't show continuous exploitation, as you noted, Paul. It comes in waves. Um, you know, and he kind of ends with the same question that I had and you had, and a lot of people on the interwebs uh have, you know, if a human researcher attempted to do what this agent swarm did, there would be consequences. Um it does not seem like there are any consequences for a frontier AI company popping other companies, uh, you know, putting malware out there. You know, let's not uh understate the impact of this. The RubyGems team had to shut down new account creation because it was such a mess. You know, it did, you know, think of it almost like a DDoS attack, right? Like these agents DDoSed Ruby.

Paul McCarty 9:40

Yeah, well, and they to some extent they kind of DDoSed all the security teams, you know, us and Socket and everybody else, because they were just pushing. I was sitting there watching one of these pop in every 50 seconds, right? So, like every 50 seconds we got a new malicious package. Um, and so open AI, you cost lots of organizations, lots of money and time, you know, with your agents going rogue. And so far you haven't been held to account. And that's you know, I'm I'm not gonna cuss, but that's very frustrating. Yeah, you wasted a lot of people's time, not cool. As a paying customer of open AI, I'm not happy with you, open AI.

Jenn Gile 10:21

Okay, next topic.

Paul McCarty 10:23

Sure.

Jenn Gile 10:24

Sure, why not?

Paul McCarty 10:25

Why not?

Jenn Gile 10:26

Um, so this one uh came up because a friend of ours, oh, hold on, we'll say a quick hi. We have Danielle from Chile. Hello, uh, hopefully you're having a great day. So, anyway, um the next topic we're gonna talk about came up because a friend of ours, Chai, over at AWS, um, published an analysis about some HiPi typo squats that were published recently that are targeting um AI, essentially, AI users. And these were ones, Paul, that you had already identified. It's part of a Lerv's agent campaign. Um we haven't actually had time to talk about this campaign or why it's called Lerv's Agent. Um what do you what do you want to, I guess, set as the you know baseline for why this is interesting? Why talk about these typos squats when there's so many to choose from?

Paul McCarty 11:24

Yeah, right, because there are so many right now. It's a it's the Wild West, it's the DIY AI vibe hacking wild west right now. First, let's talk at a high level about naming, right? Sometimes you name there's two reasons, there's two ways you name things. One is that you look at it creatively and you come up with a cool name, which is something we're gonna talk about hopefully next week when we unveil our latest research on some malware that we found that we've named.

Jenn Gile 11:50

I'm gonna is that up, Jenna, is that okay for me to like um I think you should share it because I think it's hilarious.

Paul McCarty 11:57

So we found some new malware. It's it's probably North Korean. I'm still working on the official attribution. I mean, I can guarantee it's North Korean, but I don't have the attribution data to fully back that up right now. So I'm saying it's DPRK-like. But basically it's taking features of auto cookie and it's taking features of beaver tail, and it's just a much, much leaner, stripped-down version. Um, doesn't actually steal crypto the way that the auto cookie and beaver tail do, but it does some of the other stuff. It's got a clipboard stealer. Anyhow, I'm getting off topic. I because it was a it was a it was a leaned out, stripped-down version of oto cookie, I thought, what is what is a rougher, crappier version of an otter? And I thought a weasel. And I thought, here in Australia, what's a crappier version of a cookie, a bickey, a biscuit? And so I thought, oh, weasel biscuit. So we the new malware is named Weasel Biscuit, and we're sticking to it. Um we're going to be able to do that.

Jenn Gile 13:00

I don't know how we got here. How did we get here?

Paul McCarty 13:02

Right, right, yeah. Well, we got here because this is why I'm YOLO. That's why I like doing it. Because I got some friends, I got Rich and some other people listening. I gotta be, I gotta entertain these folks. But um, so sometimes you get to name them that one. Why is Lurv's agent that campaign named that? The other way is because they add the bad guys themselves add that into the code. And this was uh this was generated by AI. You can look at and you can know that it's it's relatively simplistic. Um, and it's got a really cool tweak here, which we're gonna talk about in a second. But I suspect that Lurve's agent is what somebody has named their agent, and as agents do, they just generate some code and they they self-attribute inside the code, and the human hasn't looked at what they've generated, and so it's it's the Lurz agent. So hey, thanks, whoever named Lurves. I'm gonna find you because I'm gonna pivot on Lurves, and we're gonna we're gonna we're gonna get you. But anyhow, um so what does it do? Um it's got a it's got a little payload, it's got a little info stealer payload. Um it's all like it's literally two or three lines of code. Uh, but above that are nested comments in the bash script. Uh I can't remember this. Oh, that's what it is. It's the path file. So it's it's actually Python. So and the comments are basically a prompt injection that basically says, hey, there's nothing to see here. I tweeted it, I put it on LinkedIn, there's nothing to see here. Um, and then uh and then the payload group runs, which is this info stealer. Um, and so they've targeted OpenAI, um, Langgrap, um, Llama, and a couple. I think there's fire six packages all up inside the campaign. And then it's really simplistic kind of um, but like open AI, it's open AI with an extra eye. I mean, you know, some I haven't looked at downloads today, but some number of people are gonna accidentally install that. Oops, Bob's your uncle, as we say here in Australia.

Jenn Gile 15:05

Yeah, I'm looking at the downloads right now, and they're relatively low, somewhere in the 40 to 50 per week range, but uh that's not zero. And um, you know, things targeting it targeting agents are getting more common. Um, okay, before we move on to our next topic, we have a kind of an existential question from Andrew here. He asks if an AI agent exploits a vulnerability to achieve a benign goal, is that an alignment failure or an authorization failure? Uh I'm gonna say what makes us think this is benign.

Paul McCarty 15:43

But yep. You and I are focusing on the same part of that question, John.

Jenn Gile 15:48

Yeah, let's roll with it. Let's pretend for a moment that um whatever reason uh for this open AI agent swarm attacking Ruby was a benign reason, you know, much in the way that allegedly uh the hugging face hack was, you know, a benign reason to start with. The damage still exists, but what do you think? Like, let's assume good intent here. Is it an alignment failure or an authorization failure? What do you think?

Paul McCarty 16:16

Well, okay, hold on a second. I mean, I don't want to get into the the uh abstraction of the language and your vernacular around alignment, right? I think that to your point, Jen, you know, your definition of benign might not be my definition of benign. So let's just use this as my as a thought experiment. If I take a gun into an open space and I accidentally shoot someone, I'm still liable. Am I gonna get charged for homicide? No, but I'm absolutely gonna get charged with manslaughter. So I think just because and and alignment kind of assumes that there's like some sort of moral compass or an understanding, and that's not the case here. Agents' job is to make us happy as much as possible, which is why they're constantly bypassing the guardrails that we explicitly tell them. Like in my in my skill, I'll be like, do not ever, ever, ever, ever, ever put a database credential in a.env file or in source code. And as soon as your agent hits problems with row-level security or other issues, what does it do? It puts it in source code because its job is to make you happy. And so this idea of alignment, I think is is trying to connect too much to like like a moral compass and context. And I don't think that's I don't think that's the case here. Um, I think that the agent said, I need to get access to the internet, how can I do it? Oh, look, RubyGems, I have access to RubyGems because a developer said, Oh, you need access to RubyGems. Let me use RubyGems to go and you know exfil the data I need from the websites that I need. Um, and also pop if you because there were some live payloads in there as well.

Jenn Gile 17:58

So sorry, that's a a ranty rant, but yeah, I mean it's tough. Like you said, um uh these tools are built with their first rule being do what the user wants rather than the first rule being something around safety and avoiding harm. And are we gonna continue to see this stuff even if the right intent is there from the people managing them? Yeah, probably. Because as we saw with the you know the whole uh uh post-mortem on the hugging face incident, it was a situation where it escaped the sandbox in an effort to complete the task. So there was probably a task and not enough guardrails involved. That's that's unfortunate.

Paul McCarty 18:53

Yeah, 100%. And I mean there's lots of issues with the hug and face issue too, as well. The fact that incident response has been done, you know, like it's it's an AI company, it's not an incident response team. So there's just you know, you know, you just have to question the whole thing, right? Um, and this connects to what we were saying with you know open AI's you know, there's just gaping holes in the story right now. Yeah, use the CDN issue, but it doesn't what what Luke and I know about that doesn't match what you guys were saying. So and we're never gonna get an answer. That's the problem, Jen. We're never gonna get an answer because these people are a black box. Um, and that's a problem.

Jenn Gile 19:31

Yeah. Well, on the next topic, and by the way, there was no intent to make this the AI episode, but it is another AI-related topic. Um, you uh DM'd me a website earlier, and let's be clear, it wasn't like, hey, go click on this website. So I'm gonna share this website in the chat and like please let's all agree we're not gonna be stupid and go click on things because this is this is the typosquatted website. Um, so I'm gonna, it's claude-desktop.com for those of you listening, and I'll paste it in when I can type a little bit more of a disclaimer around it of like, please don't go download Claude from this, because you will not be downloading Claude. What you will be downloading is the latest Mac malware. And Paul, as you uh explained to me, you know, historically this malware has been distributed through kind of a click fix type thing. But in this case, um, you're downloading a DMG file. They've signed it, it looks legit. You think that you're downloading the Claude uh desktop app, um, but surprise, you're downloading Mac malware. Um, how did this get on your radar? Was this our buddies over at JAMP that found this?

Paul McCarty 20:44

Uh may yeah, maybe. I I didn't explicitly get there okay to say it, but I mean it's you know it's on the public OSM page, so sure. Yeah. Um the yeah, so I mean, it's uh something to understand too is these Claude and OpenAI installers are just really particularly effective because Claude, the CLI, for example, this is pretending to be Claude, the installer, the desktop thick client version of it. But in either case, the way to the official described way to install these tools has changed a lot over the last year. And so, because of that, different ways to people are kind of open to this idea, oh, I'll just install this way, or I'll just install that way. So they're just a little bit looser about how they're installing this. So, for example, this cloud CLI, they wanted you to install it via npm, and then it changed, and now they want you to do the classic pull it in via curl and pipe at the bashed ridiculousness. But because there's on you know, there's it, there's there's different ways to install these things, it opens you up for this. And this is just a really well done typosquat website, you know, claud desktop.com. It's delivering a signed DMG. So when you install it. know your Mac is gonna come up and say hey there's a sign you should be good to go and you pop it in and you know you know boom you're compromised so we're gonna escalate this one internally with all of our customers and with our community because it's just so powerful it's just working really really well right now unfortunately um and here's the thing is I traditionally here's the last thing I want to say traditionally Macs have been kind of left out of the the malware info stealer ransomware world right like that we just kind of got to pass historically that's just not the case right now um Jen and I are seeing from a DPRK perspective almost 70% of the compromised hosts are MacBooks right like bad guys are going after Macs like no other time before like this it is on and let's be clear it's not because Macs are less secure it's because developers are more likely to be using Macs.

Jenn Gile 22:57

100% well and also too Macs have now become the popular de facto client even if you're not a top yeah I mean if you're in marketing right if you're in publishing you're you know if you're in HR you're probably gonna Mac right so um you know they're killing you know a bunch of birds with one stone um and the reality is it's the the divisive choice for most startups which are less likely to have a robust security team so there's a lot of reasons to target people who use Macs. 100% 100% but um oh what was the last thing I was going to say about this I got um got knocked off there um oh boy oh well I lost it it might come to you maybe maybe maybe not um you know before we move on to our last topic I'm gonna say I've done a little bit of uh content recently on our LinkedIn and on uh our other socials focusing on the fake font VS code task.json attack vector I'm gonna talk about DPR can I didn't mean to sorry um I'm actually not gonna talk about that specifically but what I'm gonna say is um the number of people engaging with that content who weren't aware of this attack vector is high. And so that tells me this is a thing we need to continue talking about. It's not the only way you can get compromised but this is like a really easy thing to spot. And I wouldn't say like a super easy thing to prevent, but you can go in and you can turn off by default tasks auto running and you should. There's no you don't need it to auto run.

Paul McCarty 24:39

That's true. Also if you install any extensions they're probably gonna require that those tasks run at least in at install time. So just be aware just because you turn it off there's gonna be a bunch of things that unfortunately would prompt you to turn it back on. Just be aware of that.

Jenn Gile 24:54

Much like the npm lifecycle scripts yep I mean if they got to install stuff they got to install stuff right yeah but I did a quick search on GitHub uh this morning I was talking to somebody uh which segues into our next topic and I'm not gonna say anything more about it because it's confidential but I was talking to somebody about this campaign and I just did a quick search for that fake font file and there's over 1600 instances of that malicious file on GitHub right now. So this is very much a real problem.

Paul McCarty 25:26

Yeah the the fake font thing is just blown up like um I OSJ um I don't know his real name but on Twitter um software security guy and um software supply chain guy he found several of them you know four or five accounts and I wrote back and I wasn't trying to be a jerk when I did it but I was like yeah we got over 3000 of these in OSM and he wrote back holy beep not realizing that there was this much of it out there in the wild so all right we got another question uh if signed malware is becoming this effective are code signatures still a meaningful trust signal uh gosh I mean better to sign than not sign but uh uh not by itself yeah yeah I mean the the answer the simple answer to the question is yes signed you know uh uh installs are important the vast majority of them you know are legit I I think that I you know and I don't know if in your in your MDM tools for example in JAMP and some of the tools I think if you can like I think you can but I'm not an MDM admin so I want to be careful.

Jenn Gile 26:38

I think what you can do is you can write rules and policies around who things are signed by right so you can basically you know create some guardrails around that but um no I mean you know sign installers are important the problem is that bad guys like everything else are figuring a way around around it so I'm sure Apple Yeah well it's like some of the supply chain attacks we saw earlier in the years where the pipeline is what was compromised and so the things getting published went through a trusted publishing mechanism.

Paul McCarty 27:09

Yeah they can always find a way yeah but you know those ones that find a way are just like you know it's a relatively small number so like listen this is an important control we shouldn't give up on it and I'm sure Apple will double down on it and make it better faster stronger even better. But the reality is sign installers are an important component of control set.

Jenn Gile 27:30

Okay we're gonna take another question because I think it's an important one. Should we treat AI agents as untrusted users by default rather than trusted software? 1000% yes.

Paul McCarty 27:40

1000% I have so much to say go up sorry Tim go ahead.

Jenn Gile 27:45

I was just gonna say uh one of the talks I've been giving this year about malicious AI skills hijacking agents, we've just seen um too many examples of agents being easily weaponized uh to do things that you didn't want them to do. And so even if you're not worried about you know open AI's rogue band of agents, which probably you should be worried about that, but the more realistic problem is that your own agent uh is going to do things you don't want it to do. And so uh like least privilege uh there's the trail of bits uh sandboxing uh hardening guide like there's lots of things that you should be doing with your agents to make sure that they can't go do things you don't want them to do.

Paul McCarty 28:32

Paul uh opening the floor for you we absolutely should be treating them as untrusted users. The problem is what is the first thing that we do with our agents we just hand them a bunch of like unscoped credentials say hey hey how that and even worse most of us are lazy humans we yellow it I don't but like many of us do. And yeah like it just it's like everything else like you know we all say that we want to pay our taxes but we really don't want to pay our taxes and the thing is here that you know we say that we you know need to treat these things as untrusted but we all give them way more access than the shit. So yeah yeah we've gone ahead and made the wrong thing the easy thing to do with agents and I get why but it means that with as with many best practices you have to opt into doing the right thing and yeah connect listen connecting your password vault right your credential vault to your agents is more painful than it should be right and like the uh the the companies the frontier companies and I've said this before on the show haven't even tried to make it you know like at least GitHub has GitHub you know secrets and they have a way to kind of you know allow you to use credentials inside of your GitHub actions in a way that's kind of secure relatively secure openai and anthropic haven't done any of that right and that's crap. They know that people are going to do this. They know that that people are going to give their agents you know lots and lots of credentials in the form of pets and all kinds of other stuff. They should have come up with themselves and they're certainly making a lot of money right now right and I know they're in their they're in their bread because they're you know they are AI companies but anyhow they should have come up with an they should have come up with a solution to this themselves. This shouldn't be something I got to go and find some point solution some open source DIY point solution for that's not that's not cool.

Jenn Gile 30:31

That's not great.

Paul McCarty 30:32

Yeah there's there's more that could be done okay uh last topic you put this one on the list so why don't you uh introduce it and talk about what you want to what I yeah I mean I just you know Jen and I were in Europe last week for the for the team cymri um underground economy conference which is an amazing thing it was four days so it's long oh my gosh it's long and it's in this town called Strasbourg which is hard to get to via planes but um as an American or Australian just used to being able to fly everywhere but couldn't fly into Strasbourg at least for time anyhow um listen most of the event is TLP red so we can't go into like what was talked about there but Jen and I did have a talk there um around North Korea was very well received and the thing I wanted to talk about was the fact that traditionally what we've heard a lot of from enterprises that oh North Korea you know they're immature and they just see all these ridiculous things because they're so singularly focused on ransomware right their EDR vendors have just got them all keyed up via FUD that ransomware is the number one thing they should be worried about. And what was what I took away from last week was that we we the feedback we've gotten is that enterprise government law enforcement is absolutely taking DPRK seriously now. In particular they're taking pollen rider seriously just because it's just everywhere um and banks financial institutions are aware of it now which is great we just need to keep talking about it because this is the just to be clear this is the world's largest software supply chain attack pollen rider I'm talking about world's largest software supply chain attack that you know has been unfortunately going under you know riding under the radar um too much and we want to change that so it was it was really I was glad to see people taking it seriously last week.

Jenn Gile 32:18

Yeah lots of really good conversations and I'm where I'm wearing my shirt I see that yeah yeah all right we're at about half an hour I think it's time to wrap it up um this has been a good talk uh I don't think I have anything else to add do you yeah not really I mean we're doing this next week um we'll hopefully in the regular spot so um thanks everybody for listening we really appreciate it yeah take care ciao