Verizon Double-Take Data Breach
Brian Krebs recently reported on a Verizon data breach here: https://krebsonsecurity.com/2016/03/crooks-steal-sell-verizon-enterprise-customer-data/
I don’t know for sure if the bad guys in the most recent breach actually used an exploit to get the data, but I can tell you that the Verizon MongoDB that I found was completely unprotected by any password or authentication. All you needed in order to access it was a MongoDB client and the IP address.
The big difference is that the one I discovered was then responsibly disclosed to Verizon. However, it took them a month to plug the hole. It never made the news, but now I wish that I had made a bigger deal out of it. Maybe that would have spurred some changes which could have prevented the breach that Krebs wrote about.
Let’s start with my initial email to Verizon, which I sent to EmergencyCyberReport@verizon.com on December 22nd, 2015 and titled “Urgent data exposure notification”:
I've stumbled upon what appears to be one of your databases. It is currently configured for public access without any sort of password or authentication at all. In reviewing some of the data present, I have come across, for example, secret Verizon encryption and authentication keys (PSKs).
There are also some access tokens and password hashes for various services and accounts. This database also includes massive amounts of data and metadata for DVR, VOD, and Fios Hydra services.
If you believe this is indeed your database, I can provide the IP address and port number that you will want to check and secure. Please do respond promptly, as I'm sure Verizon does not wish for this data to remain in the public's view.
And here is the response I received almost one month later (on January 19th, 2016):
It appears that you sent us a find and no one has responded. Please confirm either way – if you have not been contacted then I would like to introduce myself – I’m the Director of Corporate Cyber Security for Verizon and will be your POC.
I look forward to hearing from you and working with you on your find.
Verizon Corporate Security
I sent back a response letting Jim know that I had not heard from Verizon and wasn’t even sure if the database was still exposed (it was). I also asked Jim if I had included the IP address in my initial report and if there was anything else they needed from me.
That message prompted the following response from Jim:
I think I have everything. Give me a day or 2 to rattle some cages in our IT shop and get this looked at.
Thanks for your patience and again I apologize for our confusion over here
That evening it occurred to me that I had not actually sent Verizon the IP address of the exposed database in my initial report. I started to wonder if Jim really did have everything. So, I checked to see if the database was still leaking. To my surprise, it was. That’s when I fired off another email to Jim:
I just checked, and was surprised to see that the database is still exposed. While you're rattling those cages, you might want to point someone towards IP address [REDACTED] on port 27017 (the default MongoDB port). That's where the completely open database can be found. Attached is a screenshot that should also help.
Thanks for getting the ball rolling on this one.
The same evening, Jim responded with the following one liner:
Thanks for helping us out - even if we're slow out of the gate. I'll be back with you shortly
That’s where the story went cold for a long time. I had not heard back from Jim until March 28th, 2016 when the Verizon PR staff heard that I was planning to post this article.
I want to be very careful here and make sure that nobody thinks I’m disparaging Jim. He has a very intense job and oversees a lot of people and systems, so I’m not trying to give the impression that he unfairly brushed me off. I’m sure things fall through the cracks all the time at every corporation the size of Verizon.
Jim and I actually had a very nice phone conversation last night. It turns out that the Verizon PR team is claiming that the MongoDB I found was only a test environment with fictitious customer data, non-sensitive reference material, unique encryption keys, and solely used passwords specific to that test environment (i.e. move along, nothing to see here).
So I asked Jim about that PR claim. Jim told me that he never personally had the opportunity to see the database that I found, but the PR people are indeed quoting the official report that had been passed up the chain regarding the incident.
That’s when I let Jim know that I still have over 50 gigs of data dumped from the MongoDB that I found, and that at least some of the database tables are actually marked as being production (i.e. not test data).
Jim understandably became very interested in the 50+ gigs of Verizon data that I still have. We agreed that it would be best for me to zip it down and securely upload it to him so that Verizon can review it and verify if this database really was just a “test environment” with no sensitive data. I did so last night.
Readers may be curious as to why I don’t necessarily believe Verizon’s claims at this point. I mean, if this database really was exposed to the public, then it would make sense that nothing important was in it, right?
Well, the thing is that companies, when caught with their pants down, almost always claim that the exposed data is fictitious, or just a test environment. It’s an easy excuse that, if believed, gets them out of a lot of potential embarrassment and liability. I’d say that 90% of the breaches I find are initially denied as just “test data”. But I’d also say that the vast majority of those do indeed turn out to be real breaches in the end.
The truth is usually a mix. Much of the time it really is a test environment, however, that test environment has usually been populated with real, production, data. So, in essence, it is a hybrid breach involving real data in an environment that no one was taking seriously. This is a potently dangerous practice that I’m finding more and more.
As you can see from the screenshots posted with this article, there was some ominous stuff here. My eyebrows raised when I realized there were password hashes exposed for accounts named “VzAppUserRW”, “VzAdmin”, and “VzSuperAdmin”.
To the victims of Verizon’s most recent data breach, I want to say that I’m sorry. I’m sorry that I didn’t make a bigger fuss back in December and January when I found the other exposed MongoDB. If I had sounded the alarm on that one, then maybe things would have been tightened down just a bit more and perhaps this latest breach, the one that Brian Krebs is reporting on, wouldn’t have happened.
***UPDATE***: After one more conversation with Jim Matteo, from Verizon, I have additional details to report. For starters, I was right about the “hybrid breach” scenario in this case. It turns out that there was indeed production data here in somewhat of a test environment.
According to Jim, there had been some kind of service disruption in one of Verizon’s network services around November 6th, 2015. That’s when this test environment was put together and populated with, at least some level of, production data. It was used to troubleshoot and resolve the errors, but then wasn’t properly taken down after the problems were fixed. In fact, some Verizon employee, whether intentionally or unintentionally, ended up opening up the database to the entire world. That’s when I found it and reported it.
Jim assures me that even if this particular database had been fully leaked to the public, rather than just me discovering it, then no one really would have been harmed. I hope that’s the truth and I have no particular reason to disbelieve Jim. Although it still makes me curious if making a bigger deal out of the MongoDB I found would have resulted in tighter security measures surrounding the MongoDB that Krebs has more recently reported as stolen.
In the end I have securely purged all 50 gigs of Verizon data from my storage and notified them as such. I’ll also be keeping Jim’s cell number handy just in case I find anymore little treasure troves bearing the Verizon logo in the near future.
Follow the latest security news and stay up to-date with information about security breaches with MacKeeper Security Research Center.
Attention - Portions of this article may be used for publication if properly referenced and credit is given to MacKeeper Security Researcher: Chris Vickery.