- Role-based, attribute-based, & just-in-time access to infrastructure
- Connect any person or service to any infrastructure, anywhere
- Logging like you've never seen
Solutions Architect Cameron Delano joined PlayStation Network in 2013, just as the PS4 was about to help it rocket from a small gaming platform into the gigantic digital media entertainment service we know today. And he was still there in 2014 when Lizard Squad tried to ruin Christmas… twice. Here’s what happened.
These days, we think of PlayStation Network as a huge gaming system that millions of people rely on for the gaming portion of their day. But what was it like when you started?
Back then it was a very two-legged beast. Everybody likes to think of PlayStation Network as being highly available, but when I started there, it was really just in a couple of buildings at a data center in Las Vegas. We would update one stack while the other was serving traffic, and then once that update was done, we would move everybody over to the new stack and then update the other. It would usually take a day and result in significant downtime and unhappy gamers, of course.
You were at PlayStation Network for the Lizard Squad DDoS attacks. Can you tell us what happened?
The first attack happened in December 2014. It was widely publicized that the Lizard Squad took down the PSN, but that’s not really the whole story of what happened.
The attack, of course, caused major issues, but really, we also kind of DDoSed ourselves because on the back end, the applications were very chatty. We were running all internal and external routing through devices that were really not up to the task. So while the DDoS attack was filling up that front pipe and not allowing customers in, we also had a significant back end issue due to the internal traffic.
But even though you mostly knocked yourself down, going down was a big deal, right? What was the aftermath like?
The first thing we did was get our routing correct. We increased our capacity and resolved a lot of the internal issues. But what really helped us was re-architecting the entire front end of it. This is the part that my team was directly involved with.
So we did a series of things. The first is, we put everything behind Akamai, but kind of stealthily. We created digital properties with origins that went to AWS based — HAProxy servers, creating a proxy layer that would then bleed off more of that DDoS traffic. Whatever was getting through Akamai would get caught by AWS and not make it back to us in the data center. We also changed our DNS to DyneDNS, which is a SaaS DNS platform.
We had all this ready to go, being unused. We were running business as usual, and when Lizard Squad announced on Twitter, after we get so many likes, we're going to start our attack. We sneakily changed all of our TTLs and went and changed all of our DNS to no longer point to our data center, but point to those Akamai properties. So as soon as they kicked off that huge DDoS attack… nothing really happened. There was no effect at all… and we had a happy Christmas.
You said that working there, you were definitely in the startup life. Do you remember that fondly?
At the start it was a lot of fun and very much like a start up. It was very work-hard-play-hard—many days of staying late to the night. We all had PlayStation 4s on our desks, and we would just play games with each other after work during the PS4 launch, which was a lot of fun.
We moved the PlayStation Network from the data center fully into AWS, and those long days turned into even longer days. It was an intense and exhausting experience. A couple months after that move, I left to start a new adventure.
Now for the same seven questions, we ask everybody.
What’s your favorite word?
おやすみ Oyasumi. It means good night in Japanese.
Your favorite sound?
The little grunt that my dog makes when I scratch behind her ear.
Your favorite superpower.
Patience.
Favorite emoji?
Save or spend?
Spend.
Who's your hero?
My wife. She came over here from Japan and married me, which is a heroic event in itself. She didn't know the language fully and then made a career for herself.
And what's your personal slogan?
Fail fast.
That’s a good one. Ladies and gentlemen, Cameron Delano. Thank you so much for the interview.
If you’d like a chance to meet more great people like Cameron, check out our latest job openings.
About the Author
Maile McCarthy, Contributing Writer and Illustrator, has a passion for helping people bring their ideas to life through web and book illustration, writing, and animation. In recent years, her work has focused on researching the context and differentiation of technical products and relaying that understanding through appealing and vibrant language and images. She holds a B.A. in Philosophy from the University of California, Berkeley. To contact Maile, visit her on LinkedIn.