Jenn Gile 0:01

Hello, we are live. It is May 28th. Paul, I can't believe May is almost over. Um what happened? I realize this is something that I think you get over a certain age, and this is just everybody says this all the time, but seriously, what happened? It's almost June.

Paul McCarty 0:20

As as you said in the past, time is a flat circle, but as you get older, the circle just spins faster.

Jenn Gile 0:24

It gets smaller. Well, I just came back uh this morning from San Francisco. I was at an AI agent security summit uh put on by Zenity. And I have to say they did a really nice job. The content was outstanding. Um, you know, it ranged from people talking about the opportunity that uh security practitioners kind of have in front of them to be using agents. Um, really great networking. I gave a talk on malicious skills that went down pretty well. And uh funny story, I haven't told you this. So I walked in the door home about an hour ago, and my husband's working away, and he's like, hey, um, fill in the blank vendor. He runs a piece of software for his company. This vendor has invited his team to be part of beta testing, an AI uh chat agent. I see your face for people who can't see the face. Yeah, that's that's the face I made. And he was like, What questions should I be asking? And initially my brain went for okay, here's how I would test it, like this is how I'd figure out. And then I was like, wait, wait, wait, wait. Did did you do a security review first? Oh, okay. We have to have a totally different conversation. And so uh, yeah, this vendor um turned on this feature for them, gave them no security documentation. Uh, we don't know anything about uh how they're segmenting data, if they're, you know, how they're training things, and no answers. And he works for a very um uh uh legacy kind of company that software development is definitely not something that they do. It's not in the tech industry. I guarantee the IT team knows next to nothing about what to ask. So um I am departing on an adventure uh to help him figure out if this thing is safe to use and then get it approved for use internally so he doesn't get fired. Um yeah, it's a wild world out there. So that's just a reminder. We've been at this for what a year and a half or so, and there's still uh a lot to learn.

Paul McCarty 2:44

People weren't pen testing stuff like this to begin with, and you know, originally enough, right? Especially small businesses and blah blah blah. But now, with just this, like how quickly people are turning these things on, these features, you know, the uh specifically AI features and AI chat features, and they're not pen testing these things. So I think there was this assumption by those of us in the AppSec community that, like, you know, with all this new code and all these new features and stuff, there'd be an equal number of you know, security checks along the way, and the opposite is happening. We're seeing less.

Jenn Gile 3:17

Yeah, and I mean this particular vendor uh is a big company, but they serve an industry that absolutely does not have cybersecurity professionals on the you know, same tier, certainly not people who know more than me. And like, let's be clear, that's a low bar. Um, it's not it's not great. Uh yeah, interesting coincidence. You know, while I was at the event yesterday, I was part of a tabletop exercise that was on almost the exact same topic. Um we had a debate about whether it was realistic, and then I walk in the door to this kind of a thing. So yes, yes, it's realistic. But anyway, let's cover the news. Um, you sent me a link, maybe it was yesterday, maybe it was the day before, about OSV deleted um 157 malware entries because they had been found to be false positives. These were malware um reports that had been reported by a team at AWS uh through some automation. And um, I think before we talk about the reaction, I think what we understand, and correct me if I'm wrong here, is there was some kind of LLM generating reports where maybe the uh reports were not vetted by a human on the like outgoing side and then on the incoming side into OSV, maybe nobody at OSV validated them either. Am I understanding that correctly?

Paul McCarty 4:48

Yeah, we don't know the exact, you know, exactly what was validated on either side, either from the AWS side or on the OSV side. That's correct. Um uh well, I guess I changed what you just said, but you do know and I mean that's the thing that happened, and candidly, I'm not super surprised.

Jenn Gile 5:12

It's not ideal. There is real harm uh when false positives make it into the ecosystem, whether it's vulnerabilities or malware. But the reaction from application security vendors was immediate and kind of brutal, and honestly, kind of not cool.

Paul McCarty 5:29

100%. I you know, I said this to a group of my friends on Signal. Like, I you know, I really think this is crappy, and I'm trying not to say any naughty words because we don't get in trouble, but I think it was really crappy that a bunch of ASPM and AppSec vendors jumped on this. First, um, because most of those vendors have never like kind of you know supported uh OSV in any way, they've never submitted a single thing to it, and yet many of those vendors, and I'm not gonna use any names, have used OSV as their primary malicious package detection source, you know, since 2023 when they turned it on. So here's the backstory. Um, AWS is the largest contributor to the the open source um uh uh software foundation, which is part of Linux Foundation, which is who collabs with Google for OSV.dev. And so AWS has been um uh submitting for a while now, and they submit so many that they recently um moved to like an automated uh submission process. So they're not doing it the way that that I used to do it, just making PRs. They now push mass submissions, and by their own admission, Chai at AWS in the PR, you know, said you know, says, Hey, we turned on some new detection rules and they're they're a little bit too um uh trigger happy, and we push some stuff that we shouldn't. So 157 of those. Now, here's the problem is that they've already started rolling back some of those. So some of those actually, those are legit, and I've since found between eight and fifty.

Jenn Gile 6:59

I saw I saw one comment about hey, you guys know this one's real.

Paul McCarty 7:03

So I've found a I've found a number more, fun number of them too. Because here's a here's the problem is that when LLMs decorate these things, and everybody inside of their malware detection engines are all using LLMs, everybody is, right? Um, and uh when you decorate the the um the findings, you know, you basically have static findings and your sandbox findings, those all come together, and the LLM kind of goes through though and goes through those things and does it it's it's it's a very gray area between, especially with some of these like crypto ones where you're seeing like you know, wallet addresses and other things like that, it's really easy for something to get ticked over from 49% malicious to 51% malicious, and then then it's decorated as malicious and it gets shipped, right? And so this idea that it's really obvious at face value if something is malicious or not is not the case. Bad guys hide stuff. So I just think it's crappy that people are piling on who are not part of the process, who are not part of the solution, right? Um, and moreover, I think the AWS basically the vast majority of submissions over the last couple years have been from AWS and from Chai's team uh inspector. And so I just want to say publicly, Chai, I appreciate your work. AWS, I appreciate your work. There's this little bit of mess up, but I mean, you know, shit happens. I would rather get a couple of false positives than you know, than the alternative, which is you know, lots and lots of um uh false negatives. So yeah, absolutely.

Jenn Gile 8:27

And I mean, I think we should be clear, like we do believe that there needs to be a solution for malicious threat intel that's not OSV. OSV is amazing for vulnerabilities, it's not really purpose-built for malware. Um yeah, that said, there should be, you know, this is a community. Let's be cool. Uh, speaking of community, we got a hey in the comments from our friend Francois at Boost. Uh, hope you're doing good, my friend. And a little uh I don't know what's going on, face. So what a good guy. I hear you. Okay, let's move on to our second uh topic on the list. Uh again, might have been yesterday, might have been a year ago. Time's weird. Uh, you shared a link from CrowdStrike. I know, flat circle, like I said. Uh so CrowdStrike uh announced that they did a big takedown of glassworm C2 infrastructure. So they didn't take down the threat actor, nobody went and made arrests, they took down infrastructure. Um, you and I messaged about it a little bit. I dug into it a little bit more. Uh, it was a collaboration between CrowdStrike, Google, and the Shadow Server Foundation, which does have some ties to law enforcement. But um, we were kind of wondering how they managed to pull this off since we didn't see any government involvement. Uh, I found an excerpt from TechCrunch where they said it's not clear on what legal or technical authority CrowdStrike and others operated to take down the operation. When asked by TechCrunch, uh the CrowdStrike spokesperson declined to comment. So uh mysterious. It's very mysterious. So we can kind of you know set that to the side and talk about what the takedown actually included, because it was four pieces of infrastructure um to kind of you know educate listeners a little bit. If you're not familiar with how malware delivery works, like it's never just or rarely just the package that you ingest. There's going to be, you know, a dead drop where stuff is getting exfiltrated to um, you know, perhaps a typosquatted URL. In this case, what it was four things, it was a Solana wallet. What else was it? I don't have my notes in front of me.

Paul McCarty 10:50

It was the BitTorrent. Um, uh what is it?

Jenn Gile 10:54

Oh, yeah, the Google Calendar stuff, which I think is fascinating. And um, they had some like uh commercial VPS providers, so yeah, wide variety of things.

Paul McCarty 11:08

This this release, like first and foremost, this is great. Glassworm is a very real problem, and I'm super happy to see this. So, nothing I'm saying here obviously is critical of this operation. I just I just want to say, having been in this space for a long, long time, this this release you know presents more questions than it does answers. It's like I don't, I don't there's so many things I want to know. I mean, the first of which is like here's this here's this paragraph from it. CrowdStrike also said there were multiple wallets on the Solana blockchain that were subject to a takeover. However, the security vendor did respond to a question from it news that did not respond from this what this entailed, such as the capture of private encryption keys. Um, and they basically sorry, they did respond and said as a result, infected machines can no longer receive new instructions or prelims. So basically, Glassworm used the same technique that DPRK innovated on first, which is so like two.

Jenn Gile 12:06

Yeah, I did drop our um article from a month ago or whatever. Uh, so it's on the YouTube already. I'll drop it over on LinkedIn later. Um, we did an article about like the four arms of one monster or whatever. Um, you know, basically uh multiple vendors detected different parts of the monster, and we realized hey, it's all one thing, so that's available.

Paul McCarty 12:33

Yeah, I think you you mentioned it, but I just want to go into a little bit more detail.

Jenn Gile 12:38

Yeah, Francois saying, Who knows? Uh Threat Actor probably dropped some plain text like anyone else, yeah.

Paul McCarty 12:44

Right. So, I mean, basically the the way bad guys do this, whether it's glassworm or DPRK, is there's multiple stages inside of this attack chain, right? And so in the middle stages, they want to use things like dead drops so they can change those, right? Because at the end, the final stage is typically served from an actual C2 service, whether that's an IP address or a domain or whatever, right? And that doesn't change a lot. So when you look at the whole attack chain, there's a lot of innovation and change at the beginning, right? At the shift left of the attack chain. There's a lot of like change and innovation and and variation there, but not much at the end. So for DPRK, there's typically five or six stages, there's like two or three bash stages alone before you get to you know the the final stages. I get it. The same thing was going, yeah, same thing was going on here with glassworm. And so they used in in the second stage or the third stage, they used these memo fields inside of these address, these blockchain addresses, specifically on Aptos, ton, and then there's a Binance thing that I don't really understand. But and so the the things that you put inside of those addresses are immutable, right? But these little memos, you can make transactions, you can like basically spend a little bit of of uh crypto to like change these things, and so those things are mutable, and so the bad guys have used those as dead drops very successfully. They specifically say in this that they took those down and they had go into no detail. So I'm like, How did you do this? This is magic.

Jenn Gile 14:12

They have collaborated with someone, yeah.

Paul McCarty 14:15

Yeah, exactly. Well, you know, obviously obviously Google was involved, so the Google and calendar thing, you know, they can they can be managed managing. Um, but there's multiple questions here that I have, and I don't know if we're ever gonna get answers because they're they appear to be pretty quiet on this. Um, and my friends at Google, I'm calling you out. You haven't you haven't helped me out here, so what's going on?

Jenn Gile 14:36

Come on, dish. Um, one thing that is almost a little bit of a throwaway uh in the CrowdStrike blog, and you kind of see it echoed in the PR that got picked up by other outlets is they talk about taking down a botnet, and they don't actually explain what they mean by that. Um, you know, we have talked about uh human botnet through a different campaign and a different threat actor. And I'm curious, what uh do you think makes this a botnet? Or do you think it is?

Paul McCarty 15:11

Yeah, I don't know. All right, so I'm gonna go into total speculation mode here. If I remember correctly, Glassworm was using the same kind of technique for force uh push code once they once they compromised a developer, they were using that technique for a while. I don't know if that's the that's ongoing. Um, but uh some maybe they're you they're talking about that. So in that scenario, Jen, they're basically talking about the botnet in the same way we are, only we're talking about you know DPRK and Pollen Rider, and they're talking about glassworm, which is a we're assuming they're Russian.

Jenn Gile 15:42

Yeah, why don't you give the uh botnet for dummies explanation? What does it mean?

Paul McCarty 15:47

Yeah, so basically, um okay, oh over the years, you know, with with uh contagious interview, uh and specifically with DPRK, 51% stack. Um the the oh, in terms of the the query. So basically uh a developer has now persistence running on a malware running on their machine, um, andor ongoing access to their uh GitHub credentials or NPM credentials, right? Some kind of persistence is in place, yeah. Yeah, so then what they do is once they have persistence on all these hundreds or thousands of developers' machines, they can then go and do other stuff onwards from there. And we we're seeing that with PollenRider. Um and um uh even you know what? Actually, Team PCP even did some of that in this second to last wave. Um uh I got some good, I got some good data out of that too, as well, some juicy data out of that as well. But the the idea there is that yeah, they have persistence, they can do onwards bad stuff. And so for Pollen Rider, which is DPRK, what they're doing is they're then going and and compromising additional repositories or forking things as that compromised user to then get their code into to popular open source projects. That's the typical kind of way that they're doing kind of worm-like behavior.

Jenn Gile 17:16

And I know there's like a big debate about if glassworm is really a worm. Um, I don't think we're gonna get into that, but you know, the whole, you know, it's worm-like.

unknown 17:27

Okay.

Paul McCarty 17:27

Well, we we need to make a distinction between worm and botnet. And the reason that you and I call what Pollen Rider is doing as a botnet is because they have control and persistence of the machines, and then they are manually going out and orchestrating things on all these different machines, doing bad stuff, right? Versus a worm, which is self-replicating and goes and finds things, right? Um, so yeah, difference.

Jenn Gile 17:49

Yeah, worth worth clearing up. Okay, I think we've exhausted what we know about the CrowdStrike thing and everything else is uh speculation. So I have so many questions. I know so many questions. The next thing on my list, uh, you popped this on me in the five minutes while we were prepping. Uh there's a an ex post about um well, why don't you explain it? Because I haven't even read the post. I got the the the 30-second version from you that there was kind of uh a nasty bit of finger pointing going on that could be pretty damaging.

Paul McCarty 18:28

Yeah, I mean the internet is a buzz about this right now, right? KCLS and Gossy the Dog, and everybody else is talking about it. So basically, here's the backstory, a simplified version of the backstory. Um a researcher known as Nightmare Eclipse um had been working with uh MSRC, the um Microsoft um you know security team uh on four zero days that Nightmare Eclipse had um had found. And there was some breakdown in communications, or I honestly I don't really know what happened there, frankly. But um old mate Nightmare Eclipse decided to just drop them. And um the problem is this that happened in a timely kind of contextual time when other researchers are just not working with bug bounty or uh research or sorry, security teams at all and just dropping stuff.

Jenn Gile 19:19

So, like there's yeah, let me read what it says. So uh Gabriel Landau, the researcher, uh reported a device guard bypass with a 90-day window. MSRC told him it met their bar and they'd fix it and they asked him to hold off disclosure for extra months. He agreed on the condition they issue a CVE. They then patched it silently, decided after the fact it didn't meet the bar, air quotes, and um never issued the CVE. So uh, in his words, they strung him along for a few months to keep him quiet and then broke their word. Uh, and then another researcher, Root Sec Dev, says he responsibly disclosed a legacy auth flaw that allowed password spraying while avoiding smart lockout five months later. Same thing. Um Microsoft uh posted about it, meaning to defend their coordinated disclosure policy, and instead it's become a thread of researchers explaining why they stopped trusting Microsoft process. And honestly, that's fair.

Paul McCarty 20:29

Yeah, it's enforced. So I like I think it's less important what specifically happened with this case and more about what what is kind of brought on. Pattern, yeah. Uh yeah, pattern, which which is that Microsoft has a has not acted in good faith with many researchers. Simultaneously, Microsoft has is under budgeted for security teams, you know. And I have a bunch of friends that just recently got laid off, um, had dinner with one of them last week. Um, and um, you know, certainly from a GitHub and NPM perspective, it's just it's a crap show, it's not the other word. It's so listen, I deal with this all the time when I report malicious npm packages and GitHub repos, they just take a long time to get back to us, and the MSRC is doing the same thing over here. And then the important thing here is that Microsoft's response is just crappy. It basically equates the security researcher to having performed criminal activity, right? It says that something I'm paraphrasing here, but it's it says something along the lines of you know, our legal team is going to, you know, uh aggressively uh prosecute, you know, blah blah blah. And that's just crappy because security researchers that disclose these things are under no legal obligations to do so, right? There's no rules about this, just to be clear, there's no laws about this. We're doing you a favor. When I turn in an NPM package or when I turn in a malicious GitHub disclose uh you know repo, I'm doing my Microsoft a favor. I'm giving them free security, you know, help. And to have them respond back in that way when they systematically has been under investing in this space and not dealing with people researchers well. It's just a really bad look. And Kevin Beaumont, Gossy the Dog, you know, brilliantly put this together saying, you know, Microsoft, this is going to hurt you more than it's going to help you. You might want to rethink this policy of coming out hard swinging. This is on you, right? We're giving you free help. So don't turn your back on that.

Jenn Gile 22:32

Well, and you know, there's a lot of narrative in the security industry right now about AI slop and bad bug bounty reports. And this is clear examples of that not being what's happening. You know, it's one thing if you're getting junk and you can't wade through it, but these are, you know, from all information available, you know, these are not junk. No, these are not these are total legit. These are legit reports that were you know received and acknowledged, and then the handling of it is poor.

Paul McCarty 23:06

Well, and yeah, on that last part, GitHub then closed, shut down a nightmare eclipse GitHub uh account. Yeah, burned it all down when they have a long 20-year history of you know, specifically uh of allowing security researchers to post pox. They you know internally there was a lot of support for that. And here, because it specifically affects Microsoft, they shut it down, right? So, yeah, not a good luck. Not a good luck. This whole free speech, it's kind of Elon Muskie, right? Like free speech until it affects me negatively, and then uh no no.

Jenn Gile 23:41

Say what you want until I don't like what you say.

Paul McCarty 23:43

And then GitLab, yo, I gotta call you out because um Nightmare Eclipse went over to GitLab, uh, took all their repos over to GitLab and GitLab, and I have heaps of friends at GitLab, and I'm very disappointed in the fact that you guys shut that down. That's pretty crap.

Jenn Gile 23:57

Ugh. Well, uh, let's end on I don't know about a high note, but a useful note. Um, you and I did a collaboration with Feedly recently, and um I'm pretty happy with how it turned out. I think it's really useful. We wrote up a guide on how to use GitHub as a collection source, and where this idea came from was a workshop that you delivered definitely at B sides, uh, but various places, and it's based on a technique that you use. And if correct me if I'm wrong, this is the technique that you use to discover a lot of the pollen writer uh malware, right? Okay, so uh I'll share the link to that, but yeah, give us the where this came from and why should people be using this?

Paul McCarty 24:48

Yeah, I mean, basically, these are just my simple techniques to forward hunt inside of GitHub looking for bad guys, you know, and looking for the the scale of DPRK and glassworm and other stuff inside of GitHub, right? So ultimately it's just a set of search patterns strings as well as a process around that. And so I did an all-day training on it in and I and I did GitHub, VS Code, NPM, and PyPy in one day, which was too much. I knew it was gonna be too much, but I got really caffeinated and I talked really quickly.

Jenn Gile 25:16

You're really excited, yeah.

Paul McCarty 25:18

Yep. And at least one person's come to me and said that they've taken what I taught them in that class and now has built actual production forward hunting in their ecosystem, and they're actually using that to find stuff. And I just think that's amazing, right? Like that's um it's really dope. So this Feedly one is kind of a condensed version of that. You did a lot of great work on this. Thank you very much, Jen, for kind of taking my big expansive thing and making it much more kind of easy to to um access and and you know more more to the point. Uh so I appreciate that.

Jenn Gile 25:50

Anyone who listens to this podcast understands the difference between you and I, I think. I'm the the crib notes and you're the the uh uh I'm not you can wow you can wax prolific too as well.

Paul McCarty 26:04

I've heard you do it lots of times.

Jenn Gile 26:07

I can ramble. Um but yeah, I'll make the offer. Uh if anybody uh wants the workshop, uh, I think we're up open to delivering it elsewhere. Did you put it in for uh DEF CON or anything, or did you do other stuff for that?

Paul McCarty 26:23

No, I might actually I got a few days here to do some of the um the villages. So I might I might do it for some of the villages. For example, um adversary village, I might do it for that, but um I just got to kind of wrap it up in a specific way for each one of those.

Jenn Gile 26:36

Yeah, well, uh, for anybody who's putting in um uh CFPs for the week of fun in the summer, um Adversary Village, uh AppSec Village, those are closing, I believe, May 31st. So those things in. I think besides Las Vegas is closed already, DEF CON's closed already. So we should start hearing pretty soon what those schedules look like.

Paul McCarty 27:02

And and sorry, my brain, you know, uh I definitely put in Yeah, sorry, the the light bulb went off while you were talking. I definitely put in this um GitHub search thing, um, hunting in in GitHub for a four-hour workshop. So I did one last year that was similar, it was red teaming the software supply chain, which was very successful at DEF CON. They're four hour workshops. Um, and I put in for the same thing. I haven't heard back yet, so let's cross our fingers.

Jenn Gile 27:27

Fingers crossed, people keep asking me if we're speaking, and I'm like, I don't know, but we'll be there regarding.

Paul McCarty 27:32

Oh, I got double rejection from Black Hat.

Jenn Gile 27:35

Oh, bummer. Well, I guess I'm not surprised it's hard to get into Black Hat, and it feels like it's getting harder. Um, be interesting to see what is getting accepted. I don't think it's personal, Paul, but at the same time, it sure feels like it, doesn't it?

Paul McCarty 27:51

Right? It really does.

Jenn Gile 27:52

Yeah, no, we're not salty.

Paul McCarty 27:54

Okay, we're not salty at all.

Jenn Gile 27:56

We're at about 28 minutes or so, so I think this is a good time to wrap it up uh before we just start bagging on black hat. Um, we'll see you all in a week. Take care.

Paul McCarty 28:10

See you guys. Thanks for listening. Appreciate it. Cheers.