Token Security Podcast | SoFi Head of Infrastructure Peter Tormey

This episode we sit down with Peter Tormey, Head of Infrastructure at SoFi. The crew talks PII, security and what it takes to maintain privacy at-scale for the new model of finance. Peter leads the team that manages and develops a HA Postgres infrastructure using CoreOS utilizing K8s to orchestrate over 100 microservice databases.

Listen to the podcast:

About This Episode

Justin McCarthy

Justin McCarthy is the co-founder and CTO of StrongDM, the database authentication platform. He has spent his entire career building highly scalable software. As CTO of Rafter, he processed transactions worth over $1B collectively. He also led the engineering teams at Preact, the predictive churn analytics platform, and Cafe Press.

Max Saltonstall

Max Saltonstall loves to talk about security, collaboration and process improvement. He's on the Developer Advocacy team in Google Cloud, yelling at the internet full time. Since joining Google in 2011 Max has worked on video monetization products, internal change management, IT externalization and coding puzzles. He has a degree in Computer Science and Psychology from Yale.

About Token Security

At Token Security our goal is to teach the core curriculum for modern DevSecOps. Each week we will deep dive with an expert so you walk away with practical advice to apply to your team today. No fluff, no buzzwords.

Transcript:

‍Max Saltonstall 0:00

Hey, thanks for joining us today I'm Max with Google Cloud and with me is Justin from StrongDM. Justin McCarthy from StrongDM and we are joined today by Peter Tormey from so far thanks for joining us today

Peter Tormey: Thanks for having me

Justin McCarthy: Peter, I believe you lead data operations over at SoFi so your day to day deals with actually pretty sounds like pretty sensitive information because of course SoFi is a financial institution right?

‍Peter Tormey 0:28

Yes, yeah, we store all kinds of good fun sensitive information and have to make sure we've got it secured in the best possible way.

‍Justin McCarthy 0:41

So today's topic is application secrets and Peter could you start us out what is a secret?

‍Peter Tormey 0:48

A secret is anything for me that the general public doesn't including your general audience inside your organization.

‍Justin McCarthy 1:01

Could this be like last quarters financial results or next quarter's financial results because I really want those.

‍Peter Tormey 1:11

No, and I think that would just lie most under like PII and Sarbanes Oxley, SOX. Applications secrets are anything that your application requires to connect to other services or data stores. You know obviously passwords and private keys things like that are secrets depending on your risk tolerance and how you view security you can even go down the road of saying that IP addresses and ports could also be secrets. We don't want to just publish those out on the internet for the general public. Secrets could be you know it could be a very broad definition so you have your things that you definitely consider secrets, passwords, and encryption, keys, things like that, you know, but depending on what your personal view, I guess it could be a lot of different things.

‍Justin McCarthy 2:16

Okay. So things that your application needs to run that you'd rather not be tweeted out?

‍Peter Tormey 2:23

Yes. As a broad definition. Yes.

‍Max Saltonstall 2:26

Yeah. important pieces of information that gain access to critical data and that data should be kept confidential. Yes, absolutely. If you don't want it on Twitter, it might be a secret.

‍Justin McCarthy 2:39

So Peter, it sounds like you've been at SoFi since the early days and you know, thinking back to when when that team was that that lean and overworked team but when it but when you still had a pretty high standard of security to maintain really if you if you had some advice for another team that was in a similar situation, what would be the first thing you would encourage them to do with respect to secrets. So that's what's the quick crawl, walk run for introducing the secrets management?

‍Peter Tormey 3:11

Yeah. So the first thing I would do is trying to find a way that your applications aren't grabbing and storing secrets in plain text. I guess what I would what I would say is don't hard code your secrets in your application that would be the first step let's let's not hard code secrets at a minimum use some type of Config Manager right chef answer or something like that. That's the best thing to do. You know, right right from the start. If you're already at that point, and you're already using some type of configuration tool in your store your secrets in that configurations when they will pass your secrets to your applications. You're doing good. But the next thing I would say is how can you encrypt those? Having your secrets. And config management is a good step. But even if you can encrypt your secrets, you're in a much better position. If your config file gets leaked to or accidentally posted somewhere, you know, in Slack or GitHub or something like that. Is it in plain text? You know, that doesn't do you much. Good. So can you encrypted Third, if you've got an encrypted How can you have you prevent anybody from accessing even the encrypted secrets without some type of authentication, whether that be tokenized authentication for application multifactor authentication for humans, you know, for people, employees, things of that nature, but how do you limit access to that a secret to just those people and processes that needed that has kind of how I would approach it I guess from a crawl, walk, crawl, walk, run type of approach.

‍Justin McCarthy 5:08

And then in terms of when you were actually contemplating your options for your secret management, just what were some of the key inputs to selecting the project currently using vs in a really rolling around are using an old

‍Peter Tormey 5:23

Yes, we're extremely agile. And so as part of that agile methodology ours is kind of a trial and error. So we literally I remember a single day we decided to switch from Chef to Ansible. That same day, we spun up Ansible. We got that working. Didn't like it; tore it down on the same day and we just literally iterated through different content management secret. We landed on Vault. We liked the way it worked it took us two or three days to figure it out. Figure out which one we liked what we were just extremely agile. spot them all off and tore down until we found will be like.

‍Justin McCarthy 6:10

No big proposal process you tried it on and the shoe fit yeah actually

‍Peter Tormey 6:19

Yes if anybody knows that's been a part of a startup like that there's there's no RFP there are no proposals, there are no committees, there's nothing like get the job done and figure out what works.

‍Max Saltonstall 6:32

It seems like you've gotten to a place where you feel really comfortable with how you're storing these keys, the secrets and protecting data of all of your users. What about combating an insider threat and the ability of one of those operators who have access to a lot of secrets to do something bad with them.

‍Peter Tormey 6:51

So the nice thing is we don't have to have a lot of people that have access. There's a very core group of people that have access to that information. Outside of that a lot of our developers and other people that need the secrets, they simply give the path for those secrets and where they're stored and they never actually have any direct access to the secrets. So from that standpoint, it's not a huge concern on the core group of people. If we didn't have somebody that wanted to do something malicious becomes a little bit harder. They act quick enough. But one of the nice things with Vault that we really like is as soon as we're aware of an issue, we can lock the wall so our vault has the ability to be completely shut down. No Ingress no egress no nothing. It is as the name implies locked Vault, you cannot get into it or out of it. At that point we have to set up so that it takes some number of people to reopen the vault each with individual tokens and do that so if we did have a person at that actor, it'd be very as soon as we're aware, within seconds or minutes, that both can be locked down, that does prevent us from creating application services, things like that from accessing those secrets. But the way we've architected our applications is that once they have their secrets unless they are redeployed, or something of that nature, they have those secrets so we don't have to really have to worry about the applications themselves falling over. So say I would lock down the vault and then take any other measures we need as far as rewriting secrets and keys and stuff after that.

‍Max Saltonstall 8:47

And when you're thinking about that, what to you is the most important data that you need to protect you and your team are operating and deploying each day.

‍Peter Tormey 8:57

Whether the data itself. Like I said SoFI is as a financial institution, so a lot of very sensitive financial information that needs to be protected. But also the applications that connect whenever any of our data stores obviously they're connecting with user accounts, passwords, tokens, that nature, and we'll make sure that those aren't compromised either because it was just-

‍Max Saltonstall 9:24

Otherwise, Justin loses his mortgage. And and we're all sad.

‍Justin McCarthy 9:27

I am a happy SoFi customer. Thank you for lending me all that money.

‍Max Saltonstall 9:34

So what do you what do you look for as you're pushing this problem in a large scale across all the different kinds of secrets and keys? What are you looking to do as you organize all that?

‍Peter Tormey 9:48

Well, from my standpoint, I deal with a lot of our data stores. Part of that dealing with those data stores is encrypting that data. We use a lot of different technologies to create that data. But at some point, you ultimately have to store your encryption keys. You know, when you store those, you can store them, obviously, in a secure manner, not just in some text file somewhere in some config file necessarily. So we use a lot of Hashicorp products Vault for that that allows us very secure access to the secrets that we need for our applications, passwords and things of that nature allows easy to use for it as well.

‍Justin McCarthy 10:40

So one thing I know that often comes up with secret management is the concept of, I think, HashiCorp calls it a secret induction, but the idea of bootstrapping. So in other words, how to the machines that are privileged access the secrets How do they get those privileges so there are a lot of different ways to do it, but you always need to do it carefully. So can you share a bit about, um, how do you begin that bootstrapping?

‍Peter Tormey 11:06

So our applications they all use in the console. So the applications will store the console path to where their particular secret is stored. And then they're given a token. And that token gives an application access to one or more different paths within Vault. So this does a couple of things for us if any particular application were token is compromised, the secrets that might be able to be accessed are limited to just with that application.And then secondly, it prevents applications from accessing other data stores or other applications that they may not necessarily need to be accessed. In fact, our applications of the console on the work with data ops and DevOps and in their application they'll be given the path to their secret were secrets that they need. So then instead of when the application comes up the call from console uses the token that has been granted for Vault will grab that secret typically store that as needed be variable within the application you can there have access to the information that it needs.

‍Max Saltonstall 12:26

Why do you think storing with the config manager is less safe, less operationally sound.

‍Peter Tormey 12:34

For us, for starters, that the config management isn't necessarily the secrets aren't exactly secret encrypted. They're all in plain text which isn't great. So our particular setup with vault is that everything is based 64 encoded and then encrypted at rest as well. So we have we have a little bit of extra security around that, and then as well, the config management now you have the possibility that different applications can get ahold of different parts of your config and, and potentially grab secrets that you don't need to have. So using vault to store those secrets makes that a lot less likely harder for them to do. I guess we have had some applications become very chatty. And there's a lot of talk over the wire that ultimately would cause that CD to have some issues basically coming close to DDOSing ourselves.But other than other than that, you know, we have a little bit of work to get our configuration, correct, make sure we have the right number of nodes and, you know, it's the high availability basically. But once that was worked out, there wasn't a lot of issues. There's no there's no latency for it. No, no no appreciable latency to it. We pass into token into Vault and it reaches back our secret and we're up and going.

‍Max Saltonstall 14:04

When you're thinking about how your team operates, you're able to automate a lot clearer. And you've made a pretty efficient process with vault in with console. What are your teams still doing that you wish they didn't have to do manually.

‍Peter Tormey 14:19

When we split up in your data store? That data store obviously needs to have usernames and passwords created for the applications. We still have to manually set those we have no way currently or at least not that we've automated no way to just set a password say in Vault and have the data store itself spin up, reach out to vault and set its own passwords and secrets. We still have to manually go in, you know, to a database for instance, and say, Hey, you know, create this user for this application and set this password to what we've set it as in Vault. We do use us Consol templating a lot for different things. So it is possible that in the future we could actually use console templates when writing out database configs and things like that to potentially have that reach out to vault and grab the secrets that it needs and right the data store configs from that we're just we're not there you see companies and organizations almost on a weekly or monthly basis that they've been hacked and data has been leaked. You know, that type of stuff that there's a lot of different things you can do to mitigate that but storing application secrets or you know, keys that type of stuff, storing those in plain text and just sitting next to me you're kind of just asking for trouble so why store them in plain text if you don't have to the great there are great tools out there that they'll store them you know for you they'll encrypt them, you know, that type of stuff, make it much harder to get those secrets versus just some some GitHub repo out there that, you know, even if it's private GitHub repo, we get config management stuff and you know what type of deal I've seen too often people even in slack channels and stuff like that accidentally post config management their config docs and there are all their secrets.

I've seen that happen at previous companies I've worked for people just accidentally saying oh, you need this information here. Here's the config file and oh crap there are all my secrets too... This just helps alleviate a lot of that we just don't need that. We don't need to store it in plain text anywhere. It's just not needed even within you know we ever were heavily into Kubernetes & Docker and as well. Even within our Docker containers and Kubernetes manifests, we just referenced Vault we never have to actually store any of the secrets we reference Vault and we're good to go.

‍Max Saltonstall 17:11

for people listening along who are thinking all right, these are good security practices and I know what to do and I'm on track what's the next thing what's the extra piece of their checklist you want to make sure they include-

‍Peter Tormey 17:23

At SoFi we take the onion approach right so security is all about as many layers...

‍Max Saltonstall 17:27

It makes me cry

‍Peter Tormey 17:31

And putting as many layers as you can around your infrastructure and around your data but the one layer that seems to get people in the most trouble is their user security. So I would say definitely that that is something to take a look at, you know in that regard, and how you just handle workspace security and things of that.

‍Max Saltonstall 17:55

And when you say user security here, you're talking about your employees, right, the people don't have access to the secrets/ Not the clients, the customers of your services with the internal user base. They, if they their accounts were compromised could let some of those secrets or some of that raw data leak out or be exfiltrated badly, right?

‍Peter Tormey 18:13

Yeah, absolutely. You're absolutely right. Yeah, you're gonna have people in your organization that have access to that sensitive information to the tokens to your vault, or whatever your secret store is to your passwords in keeping that side of it, secure. Everything we do here. So if I require multi-factor authentication, we can't get on to our network without multi-factor authentication. You cannot get on to you can't get to the Vault without multi-factor authentication, you know, so, so things like that, keeping it secure from the human side. Computers are pretty good at being secure humans, not so much. We're kind of bad at it. So everything you can do to help protect those secrets and keys from the human element is really-

‍Max Saltonstall 19:04

awesome. Thank you. Thanks very much for joining us today and talking about security and SoFi and secrets I really appreciate it.

‍Peter Tormey 19:11

No problem. Thanks for having me.