[VIDEO] Watch "Stopping Cyberthreats - Stories You Can Learn From" webinar or jump to your preferred topic below.
From cryptominers hijacking the company phone system to unsuspecting employees giving up their passwords to phishing sites, cybersecurity experts Mike Hamilton and Drex DeFord share stories of how our experts have helped clients stop cybersecurity threats in minutes, & avoid lost records, extortion, & disruption.
Learn the intriguing details of how the cybersecurity threats were detected, investigated, confirmed, and how they were removed from the network in a matter of minutes.
In this webinar (recorded in October 2019), you’ll hear:
- The latest threats we’re seeing in the SOC, such as IoT botnet credential-stuffing
- Timely Incident Response in action, including recommendations for improving your incident response plans
- New ways organizations are monitoring and responding to threats
- And other high priority risks in today’s connected world
Jump to:
Historical Approach to Cybersecurity
Story 2: Loss of Admin Credentials
Story 3: User Credentials to Phishing Site
Story 4: Confirmation of a Non-Event
The following is the full transcript of the webinar.
Mike Hamilton: 00:03
Welcome everyone. We're going to talk today about stopping cyber attacks. I think more correctly what we're going to talk about is stopping the impact of the actual bad effects from cyber attacks and security events on your networks. We intend to do that through a series of stories about things we have observed that may be relevant to you. I'm here with my colleague, Drex DeFord.
Drex DeFord: 00:28
Hi. How you doing, Mike?
Mike Hamilton: 00:30
I am doing well, it's good to hear your voice. Sorry I'm not in the same room with you this time, but it'll happen again soon. So, a little about me, I am the founder of Critical Insight. CI stands for Critical Insight. Everybody got so sick of typing that, that we had to change the name. So, we're Critical Insight. About the last 10 years I spent in government. I was a policy advisor for Governor Inslee here in Washington State. Prior to that I was the Chief of Information Security for the city of Seattle for about eight years, a little under eight years, and I was all private sector before that. I was the managing consultant for [VeriSign 00:01:09], and had a variety of private sector consulting contracting jobs, a year at TRW, a year at Applied Materials. I've been all over the place, and going way, way back in time my job was algorithm development for hyperspectral's remote sensing of the coastal ocean as a NASA oceanographer at the Jet Propulsion lab, which was the best low paying job a person can ever have. Drex, what about you?
Drex DeFord: 01:40
I don't have nearly as exciting a story, I think. I'm a retired Air Force officer. I was enlisted, I went to school at night, finished my degree and wound up being commissioned as a hospital administrator, and specialized in healthcare IT. I was a CIO at a small hospital, and then the Air Force School Health Care Sciences as the deputy CIO, then I was a regional CIO with 14 hospitals across the southern US. I ran one of our large medical centers, and then I became the Chief Technology Officer in DC for the US Air Force health systems worldwide operations. It was a lot of fun, but 20 years had gone by, and being the luckiest guy on the planet, I was recruited to Scripps Health in San Diego as the CIO, and then to Seattle Children's Hospital and Research Institute, and then to Steward Healthcare in Boston. And about four and a half years ago I hung up my own shingle as an independent consultant, and I am also the Healthcare Executive Strategist at Critical Insight.
Mike Hamilton: 02:43
Yes, you are, and we are lucky to have you, and let me just say that.
Drex DeFord: 02:47
Oh, thanks.
Mike Hamilton: 02:48
I really enjoy spending time with you. I'm looking forward to today's chat.
Drex DeFord: 02:54
Thanks.
Mike Hamilton: 02:54
So, here's what we chat about. It's intentional that Drex and I are on the call today, me being the government guy, and Drex being very, very experienced in the health sector, those are sectors that we intentionally market to, and prioritize in our work at Critical Insight. Mission really does drive our work, and there are a number of things that make people happy about where they work, and we categorize those as purpose plan potential. Purpose, that's our mission, and we created this company specifically to serve state local government, health, things like maritime ports, public utilities, things that are really, really critical to the way we live our lives.
Mike Hamilton: 03:52
A letter from a credit card company is one thing, I can't trust my drinking water, or 9-1-1 didn't work, are completely different. Recently we've seen in the health sector in particular things like denial of service, attacks against health center call centers, and the impact there could be loss of life. So, mission really does drive our work. These are two of the hardest markets to work in, I can tell you, but that does not dissuade us from our mission. Drex, you've been in this business a long time, what are your thoughts historically on what got us here?
Drex DeFord: 04:31
Sure. I go back and look at the work that both you and I have done in different lanes for a long time. You used to sort of hand roll firewalls before they were actually even a thing. I think that sort of reflects sort of the historical approach of protecting a fortress from a cyber security perspective, we have always taken the approach of we are going to put up a hard wall, we're going to keep the bad guys out, we're going to protect all the things inside the walls, and that's exactly how cyber security's going to work. And as a result, we wound up purchasing lots of different products, we train our team on lots of different products. Sometimes they're specifically security people, sometimes they're IT operations people that we've also tasked with cyber security responsibilities.
Drex DeFord: 05:23
And they are tasked with a ‘defend the organization’ job, that's it. And in the end, sometimes it feels like we buy a bunch of hardware, we buy a bunch of software, and we hope it keeps the bad guys out. But as time has progressed here, we've sort of figured out that that's not necessarily been the case. Building that hard, crunchy outside doesn't give us necessarily the result we're looking for. It's expected, right Mike? It's kind of a new standard, but it doesn't necessarily do the whole job.
Mike Hamilton: 05:52
Yeah. I think that in different venues we talk some about this, but way back in the 20th century, you're 100% correct. The mandate was to keep the bad guys out of the network. We're inclusive, so it's bad people. Today, it's more manage the risk around what is considered to be a foreseeable event. And the ground has really shifted. All that hardware, those tools that you were talking about, they are constantly, constantly messaging, alerting, look at me, something's going on, better look at me. And recently, we've seen the result of the failure to prioritize the human resources necessary to ingest and evaluate all of that messaging, and some of that has ended up in accusations of negligence against C-Suite individuals because they were the executives that failed to take steps to mitigate a foreseeable risk.
Mike Hamilton: 06:56
And target is, I think, the poster child for that. Everybody in the C-Suite there is new because they had technology, and it was doing a good job of seeing something's happening here, you better take a look, and they didn't have anybody to take a look at it. So, the historical approach to cyber that you just went through, has ended in this just enormous collection of tools that are out there, and all they're doing is talking at you all the time, and so now we got to close that loop, and we got to talk about how we're going to evaluate all of that and make sense out of all of that. And meanwhile, this is happening. You want to talk about this?
Drex DeFord: 07:37
Yeah. You're exactly right. All these tools that we purchased, all these services that we purchased, man, they just pound you with alerts, more noise than ever before, more alarms and notifications. And then not just from inside the organization, but sometimes from outside the organization because you're connected to so many partners. You get other requests for investigations, there are more breaches than ever before. The security demand is relentless, the tempo and intensity that we talk about on this slide is rising. And at the same time, especially in organizations that have tasked their IT operations folks to be responsible for security operations, and security work. There's more requests than ever before for them just on the operations side.
Drex DeFord: 08:24
Modernize IT, do upgrades, improve the architecture. Here's another list of another 50 digital projects that we want to do. It feels like especially in healthcare right now. We're going to do a massive critical system implementation in electronic health record, or rev cycle system, or an ERP system. And we have to support and run that, and the thing that you're seeing more than ever before, is that our internal customers are partners to be engaged in the provision of great healthcare to our patients and families now have a zero tolerance for downtime and outages. We have created a great situation for them, deploying electronic health records and other tools, they've become very reliant on those even though they complain about them sometimes, and the way that they work.
Drex DeFord: 09:18
They really have gotten to the point where they just don't know how to operate well, or effectively, or safely without them. And so, these two items have kind of crashed together. A huge security demand, a huge operations demand, and it puts a lot of the operators, right? It puts a lot of the CIOs and CISOs and their teams in a very difficult situation. How do you prioritize? How do you maintain the expertise you need to do all of these things, and the balance that you need to do all of these things to make sure that you can deliver on that mission, that critically important mission, specifically in healthcare, of delivering great care to patients and families. Never letting them down.
Mike Hamilton: 10:00
Yeah. You make a good point about the digital transformation that's going on, and I know it's preceding a pace in the health sector, local government is also really going down the smart city path and adding telemetry for traffic management.
Drex DeFord: 10:17
Right.
Mike Hamilton: 10:18
Right? Just building, energy maintenance, all kinds of things. And that's making the attack surface bigger, and bigger and bigger. There's just more junk hanging on your net ... and it's good, it creates all kinds of efficiencies, it controls cost, but done without regard to security, you're basically setting yourself up to have a real bad day. And I know I see a lot of that going on. I would say just with respect to the intensity as well, threat actors are making a lot of money right now. And, some of those threat actors are actually nation states that pose as organized crime, and we know the country of North Korea actually monetizes its cyber operations, and they use that to fund weapons of mass destruction, right?
Mike Hamilton: 11:08
This is the stuff that's going on in the background here, and nation state level capabilities are pretty serious. And when they package themselves as organized crime, man, what are you going to do? That's kind of over and above the anti-virus that everybody has on their desktop that we're all hanging our hats on.
Drex DeFord: 11:29
Right, but Mike, can't I get cyber insurance against some of these things? Isn't that a-
Mike Hamilton: 11:36
Well no, let's talk about that. I'd love to talk about that. Because in my view, the way that you wield insurance, that's one of the ways that you handle an identified risk. Right? So, I've got this risk, and I see that there's some likelihood that this bad event will happen, and there's some impact there, so this is a high risk. Okay, what do you do with that? Well, you can accept the risk, say, "Okay, well if that happens, we're just going to recover from that." And that's a way to handle it. You can avoid the risk, you can remove the condition that created the risk in the first place, I can mitigate that risk through the application of controls, or you can transfer that risk through insurance.
Mike Hamilton: 12:19
And in my view, you should transfer risk with insurance, after you've done the other three, and you've raised your risk bar as high as you can, such that the thing that you are covering is a ... right? You don't want to insure against rain. We live in Seattle so I can say this. It's that really low frequency, high impact event. And if you have applied controls and avoided risk, and et cetera, et cetera, so that insiders, hacktivists, unsophisticated criminals of opportunity, potentially organized crime, and you're starting to get pretty good at being able to keep those guys out. The nation state level stuff, that terrorist level stuff, that's what you should insure, right?
Mike Hamilton: 13:10
So, in my view, that's the way insurance should be applied. And as you know Drex, sometimes insurance doesn't work. What was their name, was it Anthem? That had the huge events? And the one that I know is Mondelez, which is the EU grocery chain, it's a retailer. And after the Russians tried to poke the economy of Ukraine through a tax preparation software company that ended up creating collateral damage everywhere, Mondelez was out of business for a few weeks and went to their insurance company and said, "We're insured against this, right?" The insurance company said, "Well, that was an act of war, and no, we're not really going to cover that." So, that's still in the courts. You had a ... what was the quote from, it was Anthem or Banner, or somebody? They don't insure stupid.
Drex DeFord: 14:09
I'm trying to remember.
Mike Hamilton: 14:09
That's it.
Drex DeFord: 14:09
Oh, yeah. They don't insure stupid ... yeah.
Mike Hamilton: 14:11
Yeah. Because their controls were so bad. So again, unless you've gone through that process of actually applying mitigating controls, if you just want to go out and buy insurance and think, "Hey man, I just got a policy. Everything's going to be great", it's not necessarily going to be great, because they don't insure stupid. The failure to mitigate your own risk is not viewed as something that they want to compensate you for.
Drex DeFord: 14:37
Yeah, we see examples of this, and for example, healthcare organizations who, and other organizations too, not just healthcare organizations, who declare to their insurance companies that they do backups, and they do them regularly, and they're offsite, and they're secure and all those things, but then when the time comes to actually do something with those backups, like restore them, they've never actually tested the restoration process, and those backups don't work, and that's another major problem in all of this. So, all the things that you're doing, every piece of work that you do from a reducing risk perspective, you got to make sure that you go out and test it, and test it, and test it right. Don't have any self-inflicted injuries in this piece of the world, because the insurance companies, the more claims there are, the more they learn, the more they figure out how they can deny and not to make them the bad guy, but don't over-rely on the insurance companies.
Mike Hamilton: 15:37
Yes. Do not over-rely on the insurance companies. And right now, it's an overused term, but they are feeding in a trough right now, and without the benefit of 200 year old actuarial tables that talk about the likelihood of a cyber incident resulting in actual loss, they kind of got a finger in the wind here. So yeah, they're collecting data at your expense.
Drex DeFord: 16:09
Yeah. Almost in real time, that's for sure.
Mike Hamilton: 16:11
Yeah. So, let's talk about today's biggest weakness, and I'll go ahead and let you try first here Drex, but we know what it is, right?
Drex DeFord: 16:20
Yeah, yeah. I think when we think about that whole insurance situation, we think about the way that we try to build a fortress and protect the entrances of the fortress, and protect the data that's inside the walls, the bad persons, the bad guys are very, very sophisticated, they're very smart, and they sort of figured out that the people part of this is the real weakness. That's how they're going to get through. So, today's biggest weakness are the end users. And the first response I get, I even read an article today that make sure that your end users are trained, and trained and trained, and I totally believe that.
Drex DeFord: 17:02
They absolutely should be trained, but what we've done is we've taken a person who actually does patient registration, or they're actually delivering care to patients, or they're doing research, or they're ordering supplies that are filing claims, and we've given them now an additional responsibility to become a frontline cyber warrior. They have to actually protect the organization from the bad guys. And that's not really what they do. That's not really what they do professionally. And end users are ingenious, they will figure out ways to get around the rules and the policies that you put in place, because they really have one major mission, and that is to get their job done.
Drex DeFord: 17:46
And if it's in healthcare, it's to get care to patients and families, and so they'll figure out ways around the barriers that you put up to secure the organization, they're really ingenious like that. They're reading emails and they're surfing the web, and they're getting snagged in all of these nets that are out there, that create a situation where they accidentally, maybe, accidentally give away the crown jewels of the organization, right?
Mike Hamilton: 18:18
Mm-hmm.
Drex DeFord: 18:19
These are the people who have access to the most valuable assets in the network, the applications and the databases, and all the confidential information, and all those databases inside your network. And realistically, they're also humans. We can rely on equipment if we've set it up properly, we can rely on software if we programmed it properly, but humans can make mistakes, even when they're well trained, and even when they have really great intentions. And so, the bad guys, the bad people, will get through. They'll get through the wall. And that's a real challenge that I think everyone faces today, right?
Mike Hamilton: 18:59
Yeah, and so isn't it true too, that in the health sector, surgeons, physicians, have an extraordinarily long leash with respect to the latitude that they have with their computing devices? Isn't that true? And doesn't that exacerbate the problem? We all know that medical IOT and network medical devices, and that's a thing now. But isn't it also true, I'm talking about the human weakness, isn't ... I don't want to say excessive access given, but aren't there some users that drive kind of the policy to be a little more loose than it needs to be because of their need to have all this latitude?
Drex DeFord: 19:44
Sure. We talk a lot about people, process, and technology, and all of this, and the technology turns out usually to be the easy part. It's the people and process parts that are hard. The people part is the kind of section that we just talked about, people make mistakes, they're humans, they have good intentions. The process part includes a lot of things like politics, and you're exactly right. And a lot of health systems today, the physicians may actually not be employees of the health system they work in. They're referring patients to that hospital or to that organization, but they're not actually an employee of that health system.
Drex DeFord: 20:24
In fact, some states have rules that say specifically they can't be employees of that organization. But they still refer patients, and they bring a lot of dollars to the table from that perspective. And so, they wield a significant amount of weight when it comes to asking for particular kinds of exceptions to rules, assuming that those rules exist around cybersecurity. And sometimes organizations feel compelled to allow those exceptions to happen, and they do those without maybe completely thinking through what are the compensating controls we're going to put in place for that, or what are the unintended consequences that can happen, because of the exception that we've made. And so, we probably have more of that in healthcare than other industries, and it's definitely a challenge for us.
Mike Hamilton: 21:14
Yeah, well you bring ... I want to add two quick points before we move on here, because you hit on a couple of things that I think bear a little bit of drill down. And one of those is credentials. And when you talk about training users, right now credentials are under attack. There is good data out from Microsoft and their Office 365 monitoring, and Microsoft knows what the links look like that you're getting delivered in email what the attachments look like and stuff like that. And they have been able to chart this massive rise in the number of sites that are out there, that are built to look like Microsoft so that they can gather your credentials in a phishing attack.
Mike Hamilton: 21:53
So, you get something in an email, and it leads to a site, and Microsoft will say, "Hey, that looks like Microsoft, and that's not us." So, that's what they officially say. That has gone way up, and the number of sites that push malware on you have gone way down. Almost sympathetically. It's an incredible negative correlation. So, what this tells you is that credentials are so valuable to threat actors, and we know that there's lots of abuse of credentials that have been disclosed in the Yahoo breach, and some of these others. And there are actors out there that are going around trying to find out if you use the same password on Yahoo that you use at work.
Drex DeFord: 22:34
Right.
Mike Hamilton: 22:35
Right? All this is going on, so that tells you right there, there's a problem with credentials, and Microsoft and Google, and a number of other organizations have said, "Hey look, if you use your phone as a multifactor authentication token," maybe with one of the applications like Microsoft Authenticator and Duo, and there's a whole bunch of those, 99% of the phishing problem goes off a cliff. And man, that's a little change you can make to get a big outcome. And then the other point I want to make is the one where you're talking about well, your job is at a hospital, it's patient care. But you have access to surf the web, read your email, read your home email, stuff like that.
Mike Hamilton: 23:17
When I was at the city of Seattle, we took a lot of measurements, and I could prove that 40% of the compromised assets on the network came from the use of personal email, right?
Drex DeFord: 23:28
Mm-hmm.
Mike Hamilton: 23:28
So, we're paying a lot of money to clean up Outlook, and Microsoft is dropping all the bad stuff on the floor, that's all great, and right next to Outlook on somebody's computer screen is a web browser open to, who knows? Comcast home and Bubba's email service…
Drex DeFord: 23:41
AOL.com.
Mike Hamilton: 23:43
So, what good did it do to spend all that money to clean up Outlook? And so, my point here is, and this is why I went to ... I wanted to ask about who kind of drives policy in healthcare, we all have this policy of de minimis use. You can use the organization's computing assets for your personal use if it doesn't cost extra money, affect your productivity, or cause security problems, I can prove that it does all three. So, changing that policy of de minimum use, and saying all personal use will be on a personal device, drives another 40% according to my measurements, right off that same cliff.
Mike Hamilton: 24:20
There are two changes that you can make that move the needle unbelievably in terms of the preventive controls you have in place. Prevention is not all you need to think about though, because the bad guys are going to get in. They are going to get in, and I know this because part of our business is information security consulting, and we have penetration testers, and we always get in. Every hospital, every bank, everything we ... no, there's one bank we didn't. But for the most part, we always get in. So, what that tells you is you are secure until your ticket is punched. And after that, you need to see what's going on, on the network.
Mike Hamilton: 25:04
If your wire transfer system is talking to Uzbekistan, you should really know that and go yank the wire out of it, because this stuff is going to sneak in. It doesn't matter what you have in place. And your new job right here, is to find the break in, and put an end to it. And I don't know if this is a common philosophy in the health sector, it's really rapidly becoming adopted in the public sector, I can tell you that, and when I went to the last Gartner conference, it was the risk and security management summit.
Drex DeFord: 25:41
Right.
Mike Hamilton: 25:42
They talked about augmented automation. Right? Human oversight over the possibility of taking a semi-automated act to stop something that's in progress, and this is where the focus is shifting despite the fact that venture capital companies are willing to fund any magic technology that you can impress them with, with a three slide PowerPoint deck, right? And as well as these tools. So, back to your original point, the tools are still proliferating, but the mindset is starting to change to be more about detection and rapid effective response. Is this reflected in the health sector?
Drex DeFord: 26:24
Absolutely. People, process, technology. The technology is as good as the technology is, and I think you said it very eloquently, there are tons of tools, and there are more and more tools every day. You buy a ton of them to try to figure out how to cover the waterfront, and defend the border, and realistically there's little holes and cracks between all of those tools. And there's no one tool that kind of takes care of the entire waterfront. So, a lot of this ultimately does come down to the people and process parts. Do we have the people to do the work, or more importantly, do we have the process, do we have the partners to be able to look at what's happening in our network, and see when we've been compromised very quickly, so that, and I've heard you say this a bunch of times, so that you can put out the fire when it's just a campfire and not when it's a wildfire.
Drex DeFord: 27:21
That's the stories from the front lines we're going to talk about, right?
Mike Hamilton: 27:25
Yeah. And, I'm going to just talk really quickly about the people issue that you've just mentioned because the people to do this are in short supply, and expensive.
Drex DeFord: 27:37
For sure.
Mike Hamilton: 27:37
Oh, and I can tell you with a good deal of authority, they're a bit flaky because I am one of them. And, right now, especially the younger folks that are coming up into this job, they know they can change jobs every six months and double their salary, and they do. So they're super hard to hold onto. One of the things that we have discussed is HR churn, and that kind of hidden cost in managing your own security rather than outsourcing it to a company like ours that becomes that focal point for the resources. Just briefly, and we can talk about this maybe on another get-together, but the way that we source our employees is by monitoring for free, down-market cities and counties.
Mike Hamilton: 28:21
Because A, they don't have any money, and B, they still make your drinking water. And then, rather than processing that in our security operations centers, we partnered with a number of universities in our state, and they built curriculum around the data that we collect. So, university students are getting their book learning, and they’re test-taking, and they'll get a university degree and maybe a certification, more importantly they're sitting down in front of a live-fire. And against critical infrastructure in their own neighborhoods. And the urgency to get that right is sky high. And when they pop out of there, we hire them. We put them through more training here because our commercial stack has additional capabilities.
Mike Hamilton: 28:58
But this is all about people. So, the stories that we will go through here have to do with the people that I was just talking about, who are working in our security operations centers here in Bremerton, and over on the other side of the Cascade Mountains in Ellensburg, both places with very high quality of life. And here is some of the things that we've found that we got in front of before that little fire burned the house down. Okay, so first one, yeah. This was a health provider that was really growing rapidly. And, they activated ... And so this is a trend right now.
Mike Hamilton: 29:44
There's a big consolidation going on in the health sector just for the sake of economics, isn't there?
Drex DeFord: 29:50
Absolutely. There's a ton of merger and acquisitions going on right now, and we definitely see, from the CIO seat, we definitely see a lot of the challenges and issues associated with the security tied to our decision to make acquisitions of other clinics or hospitals. And remember, a lot of the time the reason that those hospitals and clinics might be available for acquisition is that they haven't made a huge amount of investments in their infrastructure or their security protocols. And so, they may already be owned before we ever try to buy them.
Mike Hamilton: 30:31
Yeah. Well, and this particular one, they bolted a new clinic on, and immediately, immediately our analysts started to see unencrypted health records. And we have playbooks on this kind of thing. And so, our business associate agreements say that in the event that we have incidental access to protected health information, our responsibility is to notify the covered entity, which we did in about five minutes. And they were able to take that clinic offline right away, and no harm, no foul, nothing needed to be reported. They got that fixed, they put things over an encrypted tunnel, and were right back to work.
Mike Hamilton: 31:14
This was I think, with us in their corner, they avoided a bad public report in the middle of a bunch of acquisitions, which would have been financially very detrimental to the organization.
Drex DeFord: 31:30
Mm-hmm.
Mike Hamilton: 31:31
So, what do we take away from here? This is the kind of thing, seeing unencrypted protected health information, is not the kind of thing that any of those security analytics tools, none of the stuff you can buy will tell you that it's seeing that. Especially if it's going out your front door, maybe you have some data loss prevention or something like that, but what's a health record look like, right? How do you tell it that? It's a human that has to be in that loop. And the fact that our analysts are educated enough and trained to know what protected health information looks like, and what the rules around that is, I think we saved them a bunch of embarrassment, and frankly a lot of money.
Drex DeFord: 32:16
Yeah, yeah. It's part of the reason that over time I've become such a huge fan of Critical Insight, because it's not just about the technology. You have a ton of great technology under the hood, that's kind of given, and it's certainly exciting and fun to be a part of that, too. But it is the people and process parts of this, it is the fact that we bring humans to the table, to look at this stuff, see what they see, and then the great sort of Toyota production systems standard work that you've created about how to respond to these things. And how to make sure that you're only responding to the ones that are actually a challenge or an issue, or a real problem for that health system, or that clinic. And being able to help their people solve the problem. We don't just throw it over the fence, right? We actually give them some specific direction about the work they need to do to solve the problem that we found.
Mike Hamilton: 33:10
Yeah, that's a good point. The guidance a long with the pointer to the bad thing that's going on I think is a real differentiator. So, let's do story number two. Because this one is a good one as well. So, this is, without specificity, I will tell you this is a large manufacturer. They make really big things. And, the analyst, we caught a big amount of data going out their front door, right? Which that raises a flag because that's just not an average thing to happen. And on inspection, it turned out to be basically their user database, with a preponderance of administrative account details, including things like hashed passwords.
Mike Hamilton: 34:02
So, right away, so a little bit of backstory: we know that the country of China has just said publicly we want to dominate the market that this particular manufacturer is in. So, our radar is up just from the threat intelligence perspective, looking for things that may portend there's intellectual property being stolen, their manufacturing secrets for manufacturing these very large things. So, we contacted them immediately, and said, "You need to force a password change for everyone. We just saw everything go out the front door." They were very appreciative of the fact that we told them this, and it turns out later one, when I looped back with them, that this was a penetration test.
Mike Hamilton: 34:50
This was their annual penetration test, and we found that going on, and it was actually no harm, no foul. Now, the lesson learned here, and this is from my perspective of being in this business for 30 years, the rules of engagement with your service providers in information security have got to be really crisp, because that was not ... that should not be allowed. You do not exfiltrate client authentication credentials if that's not part of the program. And so, we actually recommended that this customer not use that service provider anymore. And I don't think they will.
Mike Hamilton: 35:30
So, and I don't know if you see this much, Drex. Have you seen that happen in the health sector?
Drex DeFord: 35:35
Totally overzealous, right?
Mike Hamilton: 35:37
Yeah.
Drex DeFord: 35:37
That is really, that is a step too far. You want to be able to ... you want to make sure that whoever you're partnering with is able to say, "We found it, and here's where we found it." But don't take it out of my network. That's just dangerous.
Mike Hamilton: 35:53
Yeah, yeah. Okay, so number three. So, this is in my space right here. This is a public sector organization, and this is a large county. And, this particular county, we saw an unencrypted password go out in response to a password prompt. So, this could've been, hey, just an informational thing, whoever you're doing business with out there really needs to encrypt this. But we dug in and saw, oh, the site that they gave their password to, looks just like Microsoft, but it's not.
Drex DeFord: 36:36
Mm-hmm.
Mike Hamilton: 36:37
So, this person just gave some actor the ability to walk right into the network without the use of any fancy packaged exploit or anything like that, establish a foothold, and potentially disrupt water purification, waste treatment traffic management, and all those other things, that this county does, including 9-1-1 and elections. The door was open to that. So, five minutes later, we had initiated contact and forced a password reset on that user, which turned this rapidly into a tree falling in the forest. It was nothing. But, the rapidity with which that had to occur, if that actor had that person user credentials and logged back in and then started to implant back doors and things like that, then we would've had to find some other way. That could've escalated very quickly.
Drex DeFord: 37:33
It definitely could.
Mike Hamilton: 37:34
Yeah. The lesson learned, to me there, is this ... the credentials, and then the mining and the gathering of credentials is something that our radar needs to be really tuned to right now. I talked about the Microsoft data, and there is a technique called credential stuffing. That's just pervasive right now. And credentials are being used, and they'll go, "I'll take a simple password, and I will try, sequentially, every user at your organization, try that password." And it doesn't work for one password, then I start over with another password, and that way nobody gets three login failures within a certain period of time, nobody gets locked out, and nobody knows this is going on unless you are really watching and tuned for that credential misuse.
Drex DeFord: 38:27
Right.
Mike Hamilton: 38:28
Yeah. Any other lesson you can take away from that one?
Drex DeFord: 38:32
I've just, I've seen so many of those go so horribly wrong. I had a client who had a network administrator, and again, they had done all the right things, right? They had segregated sort of the username and password, you had one credential for doing all of the internal network stuff that actually is really risky and if exposed, can create a lot of damage to the organization and then the individual has another credential to read email and surf the web and do those kinds of things. The individual had actually sort of gone in and broken the rules on his own, and said, "I have one credential for everything." And then that's the credential that got breached.
Drex DeFord: 39:19
And so, you can imagine that scenario where, man, if somebody gets those internal credentials, and they've got access to literally the crown jewels, it gets really ugly, really fast.
Mike Hamilton: 39:34
Yeah. It does. And just another quick story that you just reminded me of, it's the managed service providers that do IT support for a lot of networks that don't have IT staff. They are under attack as a one-stop shop to penetrate all the organizations that they serve. And there was one near us here in Washington State, and I got a call from them. And he said, "We have a system that is in constant VPN contact with open connections to 5000 of our client's servers, and it’s been penetrated." And, I said, "Do you have insurance?" And he said, "Yes." And I said, "Call your insurance company, because they're the ones that are going to dictate what happens next. I would be happy to jump into this, but I know that you have a legal obligation to call them first."
Mike Hamilton: 40:29
But that's in my neighborhood right here. But this is pervasive. So, losing credentials in your network is bad. But you can control that. Your third-party security, which is really becoming a thing that everybody's focused on right now, is just as bad. And that's a little harder to control. So, today's presentation is not going to address that, but we have answers for that. So, let me just talk about our last story here. So, this one I'm proud of. Confirmation of a non-event. And this was another public sector organization, and the analyst saw evidence, the kind of thing nobody should ever see. And we have an affirmative obligation to report this to law enforcement if we ever see this.
Mike Hamilton: 41:16
And, without specificity, I will tell you this is not the first time, this wasn't the last time. This stuff happens. So, we saw this, and the investigator was able to review a packet capture, because part of our system keeps on customer premise a packet capture so that we can review that and we can replay things, and we can do 100% confirmation.
Drex DeFord: 41:42
Right.
Mike Hamilton: 41:43
On review, we could replay this, and we saw this was somebody ... it turned out to be a law enforcement officer in a police department in this public sector organization. Searched for a poison oak remedy. The sites that came up in the search order, the link number one redirected to another site that was trying to force this imagery. We could prove, we could prove that the user was never exposed to the imagery. We still had to do our reporting thing, but we were able, with this packet capture, to exonerate that user and say there was no intent here, and in fact there was no exposure.
Mike Hamilton: 42:27
Now, that confirmation of this thing never happened, that was extremely important. And again, I'm very proud of that. So, the lesson learned here, is the access to a packet, there is no substitute for the ability to answer the question what happened here, that a packet capture. So, the packet capture that we keep on customer premise, for periodic examination in the context of an investigation, I think is some of our secret sauce, and right there is where it really paid off.
Drex DeFord: 42:58
For sure. Boy, I can't imagine. That would be a tragic mark to have on your record, right? It would undoubtedly be grounds for being fired. To be able to prove that something like that didn't happen, you really saved this person's life, and maybe literally, right? It's a huge opportunity that Critical Insight brings to the table that doesn't exist in most other services.
Mike Hamilton: 43:37
Yeah. Well, so we are about at the end. I'll put our contact information up here. And by the way, if you follow me on Twitter, buckle up. I have a lot of weird hobbies. I actually, I'm a drummer in a punk band, and so some of my tweets are about things like that. But I think we can get into the QA part of our presentation. And have any final thoughts you want to leave everybody with before we get into some QA?
Drex DeFord: 44:11
No, I'm looking forward to it. I think we're about to have a good conversation, so thanks everybody for listening and attending, and we'll talk to you in just a few minutes.
Mike Hamilton: 44:19
Yep, yep. Thanks a lot, everybody. And we'll talk to you real quick here.