Yesterday, a group named HTP claimed responsibility for accessing Linode Manager web servers, we believe by exploiting a previously unknown zero-day vulnerability in Adobe’s ColdFusion application server. The vulnerabilities have only recently been addressed in Adobe’s APSB13-10 hotfix (CVE-2013-1387 and CVE-2013-1388) which was released less than a week ago.
As a result of the vulnerability, this group gained access to a web server, parts of our source code, and ultimately, our database. We have been working around the clock since discovering this vulnerability. Our investigation reveals that this group did not have access to any other component of the Linode infrastructure, including access to the host machines or any other server or service that runs our infrastructure.
Credit card numbers in our database are stored in encrypted format, using public and private key encryption. The private key is itself encrypted with passphrase encryption and the complex passphrase is not stored electronically. Along with the encrypted credit card, the last four digits are stored in clear text to assist in lookups and for display on things like your Account tab and payment receipt emails. We have no evidence decrypted credit card numbers were obtained.
Linode Manager user passwords are not stored in our database, but their salted and cryptographically hashed representations are. Despite the uselessness of these hashes, as you know we expired Linode Manager passwords on Friday.
There were occurrences of Lish passwords in clear text in our database. We have corrected this issue and have invalidated all affected Lish passwords effective immediately. If you need access to the Lish console, you can reset a new Lish password under the Remote Access sub-tab of your Linode.
For users who have set an API key, we’re also taking action to expire those keys. We’ll be emailing API-enabled users with that information.
We take your trust and confidence in us very seriously, and we truly apologize for the inconvenience that these individuals caused. Our entire team has been affected by this, leaving all of us, like you, feeling violated. We care deeply about the integrity of Linode and are proud of the work that we accomplish here for you. This unfortunate incident has only strengthened our commitment to you, our customer.
Please feel free to contact customer service via our ticket system or support@linode.com if you have any questions or concerns.
Comments (78)
Thanks for keeping us updated on this. It’s sad to see exploits taken advantage of, I hope we all learn something from this 🙂
When you say “There were occurrences of Lish passwords in clear text in our database”, does this mean there were only a few? Or a lot? Or all?
Should I be worried about anywhere else I’ve used my Lish password? If it’s only a few, are you able to notify those few directly?
Hope it’s a good passphrase on that private key!
Though I’m not happy about this, I truly believe Linode has done its best to secure my server and data anyway.
Ah.. zero-day vulnerability in ColdFusion. Does Linode has any plan to migrate to a more, say, popular/modern tech stack? Frankly speaking I know nothing about ColdFusion, just feel it’s kinda.. outdated, and it looks quite dangerous to lock into one specified product.
Can the passphrase survive in an offline attack?
It was claimed that whilst the credit card information was encrypted, the public and private keys for this were stored on the same machine rendering it useless – is this true?
Also, there have been a number of claims of fraudulent bank account withdrawals on cards stored with Linode – have you investigated this thoroughly?
Thanks for the update, please help to address following questions more clearly:
1) does said hacking party have access to your public/private key for credit card encryption (are private key, as previsouly suggested, stored on this server as well)?
2) do you store anything else sensitive besides the credit card no. in the compromised database (for example personal contact information, or even worse, credit card CVV)?
3) whether there is a plan to allow making payment without storing credit card number permanently on linode?
What was the complexity of the password for the private key? Could it be found in /usr/share/dict/words or was it sufficiently unguessable and long?
I don’t know what’s more embarrassing about this whole debacle: that the breach happened and -since you haven’t denied what HTP said about this- the private key was stored on the same server as the credit card numbers that were being encrypted by that key; or that it’s 2013 and a Linux company are using ColdFusion on their signup form. I mean, ColdFusion guys, seriously?!
And can you give us an indication, on a scale of: “We’re screwed if they have a rainbow table of common dictionary words” to “Ain’t gonna be cracked before the Sun dies” of how strong that passphrase on the key is?
While I’m glad you’ve posted something, this leaves a lot of questions unanswered.
For me, the main issue of this post is that it’s generally vague about a subject that needs to be crystal clear.
– Did they get access to credit card information or not?
– Was my lish password stored in plain text? (how about a % of passwords stored in plain text)
– What other information was accessed in the database?
Also, a lot of us probably reads hacker news, and in extension the irc logs from “ryan”. Can someone publicly deny his statements such as the one about storing credit card public and private keys in the same directory?
Finally, I also read a lot of tweets regarding credit card usage that would be nice to get out of the dark as well.
Thanks for understanding,
/A long-time customer
Furthering Eivind’s question, this is the only thing that matters to customers at this stage: they have encrypted credit card info, and they have the private key. The attackers can now brute force the private key passphrase with impunity, with as much parallelism as they can command.
So what are we talking about? Do we expect they’ll have cracked it in a few weeks/months? Days?
You guys run a great service, and I don’t think I’m going to move anywhere else because I know that what you guys offer is pretty much unparalleled, but I do need to know how important it might be for me to invalidate my credit card number or not. 🙁
@Kymair
A little over ten years ago ColdFusion was completely rewritten in Java, and these days is almost unrecognizable compared to what we used to write in the old days, these days it looks very much like Java. Also these days there are alternative ColdFusion Markup Language servers, including Railo, which is open source, so you’re not locked in to the Adobe product unless you’re using a few of the tags that are only supported by Adobe.
Arlen’s question is one I want cleared up as well.
Thank you. I’ve felt like Linode does its best to be transparent and I’ve defended your service to my colleagues in the past couple of days in that regard. This post confirms my trust in your work ethic and commitment to customers.
Thanks for your hard work.
Can you let me know :
1. If I should be worried about my credit card details being leaked.
2. If we need to reset user passwords for all our linodes
3. If our applications running on the linodes could have been compromised.
+1 arlen
Thanks for the update and transparency.
so, is this linode’s fault or adobe’s ?
Elvind and Arlen summed up perfectly what is on my mind. Please advise us on this ASAP as I need to know whether or not to cancel my credit card.
> The private key is itself encrypted with passphrase encryption and the complex passphrase is not stored electronically.
I smell bullshit. If you didn’t store the private key “electronically”, how could you possibly use it to encrypt credit card numbers? Surely you don’t have a secretary sitting there manually entering CC numbers as people update them in the manager, do you?
I was giving you the benefit of the doubt as this mess unfolded. This one sentence made me consider that you may in fact be lying through your teeth 🙁
Brute-force attacks depend on the encryption algorithm used + the keysize. I hope Linode will give us some info on this. I assume they used a proven algorithm from a trusted vendor, and not something they hacked together themselves…
@Tim – We’ve only had two reports of customers claiming fraudulent activity on their cards, neither of which corroborate a larger issue.
@Eivind – our private key is stored only in encrypted format. The passphrase is not guessable, sufficiently long and complex, not based on dictionary words, and not stored anywhere but in our heads.
I greatly appreciate the update. I know a lot of folks have been complaining about the time it took to get this out there, but to me it’s worth the wait to know what’s really going on.
@XXX:
I think they mean they were using a private key file which is passphrase encrypted. To access the private key file you need to type in the passphrase. This passphrase is not stored electronically on their system.
@caker, reiterating XXX’s question, if you don’t store the passphrase anywhere, how do you encrypt the CC information automatically?
Thank-you for this update.
I will be requesting a new card number from my CC company and you should be recommending as much to everyone.
It is clear that something needs to change in your security and coding practices that recognises the huge target you are now. And to prevent this being the last straw for some of your customers you need to be open about those changes.
@XXX, @Raúl Pedro Santos: only the public part of the keypair is required to perform encryption
http://en.wikipedia.org/wiki/Public-key_cryptography
Thanks for the update Linode. Although I know everyone wants the updates NOW NOW NOW, I can certainly understand Linode wanting to get all their ducks in a row before making a statment like this, which I would expect.
It is even more evident of their dilligent work when you see the owner making a blog post at 3AM, and still repsonding to comments at 4:45AM.
CAKER: Are their any further plans to implement 2 factor authentication, or other steps to further improve security? Oh, and go take a nap.
@Raul, @XXX:
It’s called public key encryption. The private key is only used for decryption.
@Raul: you use the public key to encrypt and add CCs, then monthly when you’re billing you type in your [hopefully long and complex] passphrase to decrypt using the private key temporarily
I thought it’s been clearly stated that there had been a very lot of chances that the hackers could break the encryption of the credit card information to make use of it.
So, I just invalidated my CC and it cost a little money and brought me a little inconvenience.
Hope that Linode stuff could learn something from this and don’t store credit card infomation any more. Just make the billing before the time coming.
@Raúl Probably by keeping it in memory and entering it manually when the server/daemon boots.
Another customer here, I revoked my credit card yesterday. As discussions on Hackers News indicate there are still unanswered questions. At the very least you have to stop using Adobe products in your infrastructure and thoroughly review your security procedures.
There are some questions which you MUST answer: what is the risk of credit card numbers being decrypted? Are previous customers at risk in any way? What is the risk of passwords being decrypted (remember people reuse passwords so they may be exposed elsewhere). Also the Lish passwords, how exactly were these in clear?
RE: Credit Card Info.
If you guys read the blog post, caker stated the CC information is encrypted with asymmetric encryption. That means the PUBLIC key is used to encrypt the data and can ONLY be decrypted with the private key. Doing it in this manner means a “For Your Eyes Only” approach to encryption.
Essentially, credit card operations are run manually by a user with knowledge of the private key when it is appropriate to run billing.
Gaining access to the private key DOES mean that linked should rotate the keys used for this process, but it also means that CC information is not readily compromised. CC information should be reasonably safe, IMO.
@Raúl To encrypt information, you need the public key, not the private key with a passphrase. To bill people continually, you need credit card gateway tokens that can only be used to bill for this specific merchant and are useless otherwise. I see no issue here.
Thank you, Linode, for the update. You continue to have my full support.
@XXX and @Raul Pedro: you should both read up on public key cryptography. You don’t need to know the secret key (or its passphrase) in order to encrypt something against it. That’s the entire point of PKI.
Correction: (it’s 6am and I’m on my phone)
s/linked/Linode
And knowledge of the private key pass phrase.
Consider the way the CCs are stored as the reverse of how you normally perform certificate generation. And the pass phrase sounds reasonably complex enough that they won’t be able to crack it easily.
This is much better than most businesses handle and store your credit card information and I’m sure Linode will take even further action in the future to prevent this from happening. Caker and team aren’t idiots, but they are only human.
As an FYI for those wanting to know, various US government agencies offer up to 10 business days to patch security flaws: that’s 2 weeks. To use an attack within 7 days of release is crazy, but not as common. There’s no way to humanly be able to prevent this short or taking sites offline as soon as public availability of attack. And even this does not resolve 0-days.
Using Coldfusion was only a mistake if you assume that no other developers make mistakes when coding. Even an application like Wordpress, which had thousands of developers’ eyes and hackers attacking it continues to have security flaws, one recently alerted by the US CERT. So I don’t think a custom solution is necessarily the answer because even the “many eyes” approach hasn’t worked. But I’m sure Linode is weighing its options.
@Raul (and others) you can encrypt the credit card numbers with the public key, no passphrase required. But decrypting them would require the passphrase. So at billing time, someone could log in, enter the passphrase and run the billing program. Alternatively they could use gpg-agent to store the passphrase (fairly) securely in RAM. So their setup sounds plausible but not that secure.
However, if the server is compromised, it would be possible to put a keylogger in there, or monitor RAM, while the monthly billing is done. Then you get the passphrase and all the credit card numbers. Those attacks would require either root access, or being logged in as the same user who is entering the passphrase,or some related privilege escalation attack. I have no idea whether such an attack was done, but it feels like it was possible.
An improvement to security would be to store the private key on a few USB keys, and only plug them in when you need to do billing. But when you need to launch a new server, they need billing immediately, so that would make the gpg-agent scenario more likely.
Even better would be to store the credit card details with a third party who specialises in credit card storage and not storing them yourself.
I’m also disappointed that any passwords were stored in plain text.
@XXX, Raúl: For those who don’t get it, the whole point of asymmetric (public/private key) encryption is that you can encrypt data with the public key, without needing the password (and so put it in the DB on update in the manager) but you can’t decrypt anything without the private key (which is protected by the passphrase).
I presume the Linode folks only enter the password to run the billing itself, which happens on the first day of every month and so makes the scheme handily secure – if the system is set up as described (and with a decent RNG behind the key’s generation :P) CC data should be just fine.
I will invalidate my credit card, however I don’t want linode storing my CC anymore. I hope for some other way for payment implemented in the near future!
Paulo, in PKI http://en.wikipedia.org/wiki/Public-key_infrastructure the private key is not required to encrypt.
looks like this is the source of rumours:
http://turtle.dereferenced.org/~nenolod/linode/linode-abridged.txt
but i couldn’t found any info about stolen credit cards.
Definitely frustrated by this, but it looks like you’ve done the best you can in making a responsible disclosure to your customers. No doubt my private information has been violated worse by companies who then swept things under the rug. Thanks, I guess.
The passphrase must be stored in RAM, as think about all those automated times you upgrade and cause immediate billing only seconds later. It always knows my CC number and charges it right away.
Adobe should have stuck to Photoshop. LOL @ Cold Fusion. Did you guys port the code from Visual Basic? :p
“Linode Manager user passwords are not stored in our database, but their salted and cryptographically hashed representations are.”
Are you guys planning on reassuring us you used a hash that’s robust against offline attacks? I would definitely feel better about the compromise if you were using bcrypt, scrypt, or something similar.
You handled this well, I’m confident in your security practices. Incidents like this wont dis-wade me from using your services as long as you continue to be upfront about them. Keep up the good work guys, shit happens.
_Rob
Anyway, the Linode Manager is awesome. I was looking at one of your competitors and their control panel was horrendous by comparison.
This news doesn’t bother me. I’ve only had one fraudulent charge on a card in my life and the transaction was flagged immediately. Of all the places you buy things, who’s as cautious as Linode?
For what it’s worth, I’m completely satisfied with your response.
Thanks!
Please Linode add more payment options like PayPal.
@everyone jumping in to correct me (and XXX), you’re right, my bad, I do know a bit about cryptography and PKI and should have given it a second thought instead of asking away.
There are still unanswered questions, though, and it would be nice of Linode to provide some further explanation, if nothing else for the mere fact that not doing so will only contribute to a negative image for them.
A satisfactory response.
Wonder why Linode does not just completely store CC info with a 3rd party and charge with a token. PCI compliance is such a pain when you are storing CC info on your own servers.
These things happen. Keep up the good work and keep shoring up your security!
Thanks for the update, and good luck with the remaining cleanup work.
I’m less concerned about the specifics of this actual breach than I am about the security practices which turned something less serious (a web server zero-day) into one that was connected to information for both payment data, and plaintext passwords for server access.
Can we expect to hear how your security policies and audit procedures will ensure this doesn’t happen again?
Also, would you consider handling payments through a third-party processor so that Linode is only sending and receiving transaction tokens, instead of storing sensitive financial information on servers accessible through your web frontend?
Thanks for keeping your users informed. As usual great customer service.
Knowing that username, password, api key and credit cards were accessed by hackers (encripted or not) makes anyone nervous and paranoid.
Although, knowing that you’re doing your best to ensure our safety thru legal and technical means, just reasures my love for your services.
Whatever you write, people will make more increasingly technical moans in the comments here.
Thanks for the update. I went ahead and revoked my credit card as a security measure. I use a randomly generated password for all my sites so not worried about that. Please let us know in the future if you have made additional changes to prevent this from happening again.
Not to continue to moan too technically or anything, but the question about Lish passwords hasn’t been addressed at all.
– Do you know which accounts had plaintext Lish passwords in the DB?
– Do you intend to let those people know?
So you straight up lied to us in the email you sent:
LIE # 1: ” We have found no evidence that any Linode data of any other customer was accessed.” — BLATANTLY FALSE.
LIE # 2: ” In addition, we have found no evidence that payment information of any customer was accessed.” — BLATANTLY FALSE
Even if you did know your statements were untrue when you sent the email, where is the follow-up email retracting these statements?
This is unacceptable.
I don’t know that I have a choice but to start looking for another provider — NOT because of the compromise, but because of how it has been handled thus far in a dishonest manner.
@Skavoovie they didn’t lie with this though. At the time the email was sent they had no evidence. That changed since the email. Hence the followup with this blog post and alert.
Based on my interpretation of both the e-mail and the blog posts, “any Linode data of any other customer” would be the data on each individual VPS, not Linode’s customer data. One host was supposedly compromised, but not others. This appears separate from the Linode Manager compromise.
As for “payment information”… well, that’s a bit harder to debate. If the e-mail does indeed refer to a separate compromise (of an individual VPS), no “payment information” may have been lost in that breach. Of course, the loss of credit card data, even if encrypted, would definitely count as “payment information” in my book, although I’m sure someone could get pedantic on semantics. If we’re not talking about two separate breaches… well…
And a quick reminder to those getting heated and calling for answers: If law enforcement is already involved, Linode may not be ALLOWED to be completely transparent, at least until any criminal investigation is complete. While I want the same answers you do, bear in mind that once the police and/or lawyers get involved, companies have to be very careful in how they respond publicly.
As for me, yeah, I’ve killed my CC on file. First time I’ve ever had to do that. 🙁 But for now I have no plans to move anywhere else. Overall, I think Linode has handled this well and has been much more transparent than most other companies would be. That said, how much of that confidence will remain will depend heavily on what comes out of this in the next days and weeks.
You guys at Linode have offered such an exemplary service over the years that I was definitely waiting to hear your side of things before forming any opinions of this situation. And I’m satisfied with your response. It’s unfortunate that such things occur, but I think we can all agree that shaking our fists at Adobe would be better served. Just make sure to place all Mountain Dews, Dr. Peppers, energy drinks, and similar carbonated beverages into your opposite hand first.
Don’t put your trust re: your credit card number in Linode (or any other vendor you use it with); put it in your bank or (better, at least in the US) credit union, which is on the hook for fraudulent charges.
You are in *much* more danger of identity theft from a restaurant employee (again, in the US) who takes your card away from the table to run your tab. (Not a knock against restaurant employees, just stating a fact; many/most? places now in Europe for example use wireless readers that they bring to the table, so your card never leaves your sight, but who’s to say where the reader is sending the data…)
Honestly, you ID theft virgins… It only hurts the first (few) time(s).
As a long-time Linode user I applaud their integrity and efforts on *our* behalf. As previous posters have said: (a) zero-day vulnerability – hard to defend against; (b) they are only human, just like the rest of us; (c) I’m 100% confident they will learn from this experience and take steps to insure appropriate improvements are made.
Linode rules!
—
drsteve
Those of you who don’t understand public key cryptography, can you please take a break on the rants and go read what it is and how it works before you continue going on about how it makes no sense. It’s actually a great design for PCI compliant credit card storage, allowing the cards to be encrypted and tokenized by the website without the website keeping any useful information to decrypt the database. This is far superior to people who just use symmetrical encryption on the card number and attempt to control access to the key with ACLs, passwords, etc.
Based on their more detailed explanation of the storage of payment information, I am confident moving forward with my card on file at Linode. Considering the fact that most cards don’t hold you liable for fraud charges, and the significant number of cash cards and online use cards out there, we all have plenty of options to keep our money secure. I just like to know that they put the forethought into using PKI to store the cards, and it paid off when they got hit with this 0-day.
As far as using ColdFusion, I don’t care what web platform they employ, there will always be 0-days. That’s why a good architecture is the real defense, which again paid off for them here. The only thing I found disturbing was the LISH passwords in plaintext, which they say they’ve addressed.
With that my confidence in Linode is restored, stuff happens, life moves on. Now if we can get our memory upgrades in Fremont that would be great! 😉
Like quite a few other people here I’m quite confused… is this a second incident that was unrelated? That doesn’t seem to be the case because this message refers to the forced password reset last week.
The message has a lot of grey areas as pointed out by others… we need clarity. It does no good to know that the database or CC information was encrypted if the key was also lost, or if the key didn’t follow best practices.
In this day and age security is just as important if not more so than host/network reliability and availability.
Thanks for the update. An unfortunate incident, but we appreciate the additional detail about what happened.
Our main concern is the security of the access to the Linode Manager – we don’t want to wake up one morning to find our Linodes deleted. Please consider implementing some form of two-factor authentication in the future.
Now that this unfortunately happened.
One of these two things, preferably both should properly be implemented.
1) Two factor authentication. We’ve been asking for awhile :/
2) Another PayPal method, EX PayPal. Storing CC’s is a dangerous thing these days.
So I asked about a lot of these questions a while back. Mainly, I’m curious how far the systems were actually penetrated and whether or not the fact that I could purchase upgrades and get them automagically charged while my card numbers were somehow protected by a password only Linode knew but the panel figured out by means of magic. I was told that it was under investigation, so I ask again, more than a week later. What is going on?
Also linode, Cold Fusion? WTF. You’re about a decade or so behind the times. Maybe even two.
@Sorressean: That’s a bit harsh. CF is still a supported product (and it wasn’t always Adobe ;). While other mistakes may have been made, IMO not moving away from CF was not one of them – there is no web framework that has never had a security flaw. (Of course the open/closed source debate is another matter).
I’ve just reviewed all material regarding this breach. And I must say it does make me feel very uncomfortable. I don’t know what to trust anymore, if we are going to trust what the ryan(hacker) says from chat. Everything is compromised. Credit card data is decrypted (from chat log: ” They did try to encrypt them, but using public key encryption doesn’t work if you have the public and private key in the same directory”) and they can also access the content of your virtual machine (for example if you had your mail or private code repositories configured). Its totally beyond me that Linode staff could configure their system so badly, when even I could do it better?!! And my skill level is very low… Actually it seems the system was so poorly configured that it even makes me suspicious that the story of the hacker is fake. Then again something definitely happened :-/
If nothing else I would really like only one thing from Linode.
Please say to us:
Do we have to invalidate our credit cards?
It is not a problem to do it. It can become a problem if we don’t do it but should! Please Linode I hope you are not giving us a sense of false security!
Next time this happens, please be consistent about your method of communication. I received an email on the 12th stating that nothing was f*cked, then never got a notification about the update / blog post. I was under the impression nothing was compromised, but am currently rebuilding all servers due to new information that wasn’t received in the same manner.
I do appreciate the transparency, but attention to detail and consistentcy is very important in these situations.
I am dying to use linode.com from years, I do have credit card. But I use my credit card on internet only via paypal. I can not give CVV numbers etc to each and every site on number I purchase from; if in years to come they get hacked.
Its far better for me, add another virtualization layer upon my creditg card by using paypal.
I am just unable to understand why on earth linode.com does not accept paypal or netbanking from India etc. It will vastly expand their user base in an instant.
A real worry for anyone doing online trading.
Basic one would be keep anything away from billing that does not need to be there.
Eg install the machines according to NSA/CSS standards.
http://www.nsa.gov/ia/mitigation_guidance/security_configuration_guides/operating_systems.shtml
Or at least use that as a guide point to improve your companies own security.
If in doubt chuck it out.
In most cases if it’s not there they can’t steal it.
Even if it means re inventing the wheel slightly keep recurring billing separate and offline local access only.
What ever it takes it’s worth doing to keep customers safe.
Authentication servers is another interesting subject.
Not a huge issue in small scale operations but you can see with programs like wireshark and even your logs what ports the smash and grabbers are going for.
I was thinking SELinux may help here (as if I know what I’m talking about), perhaps the database can be running on another (ahem) linode so that the data is isolated.
You guys are awesome. EVERYONE gets broken into but not everyone reports it, you guys are on the short list!!
Please keep up the vigilance and innovation.
It seems a lot of people are missing the other side of this coin here.
Are there a lot of unanswered questions? Yes. Are you going to get answers to most of them? Probably not.
Answering a lot of the questions in these comments would involve divulging a lot of how their infrastructure is designed, and what security measures are in place; not a very smart thing to just drop into public-facing comments. Security breaches are embarrassing to everyone – most companies reporting breaches where caught not even using salt / hash, so props to them.
Like i saw in another comment here – give them time to get all their ducks in a row. In the meantime, do the sensible thing – assume all data is compromised, and change your passwords / encryption keys, and pay attention to your bank account.
[…] is how they handle it and communicate with their customers. Linode has been plagued by multiple security issues in the past. Failure to disclose incidents in a timely way can be costly, especially in the […]