Managing Secrets with Vault, with Seth Vargo

Welcome to a new episode of Syscast. My name is Mattias Geniar and today I’m joined by Seth Vargo from HashiCorp to discuss Vault and managing secrets in a variety of ways. Hi there, Seth.

How are you?

Hey, Mattias. I’m good. Doing well over here in Pittsburgh.

Thanks for having me. Super excited to be here today.

Likewise, and nobody will notice that we did this intro twice because I forgot to hit record. Great, we’re off to a good start. So Seth, you work at HashiCorp.

It’s a firm that I think most of the listeners have definitely heard from because you do a lot of interesting open source projects that I’m actually quite sure each and every one of us has used at some point. Could you introduce yourself, what you do at HashiCorp and what’s your day-to-day business like?

Sure. So I’ve been at HashiCorp about two years now. My background is in engineering.

So I am an engineer by trade with the computer science degree and all. I’ve touched about every open source project that we’ve ever worked on. So if you look at the Git log, you’re bound to see my name in there.

More recently, as the company has grown, at one point we were just… two to three to five people, and now we’re up over 40. I’ve taken on a number of different roles. And most recently, my job is the head of education and the director of evangelism.

So my job entitles running the entire training curriculum that we’re building out this year, in addition to speaking at conferences, doing things like podcasts, and helping users understand our tools at a very deep technical level. So I always like to describe evangelism as as marketing that doesn’t cost any money to the user. So I do a lot of teaching people how to use our tools for free in the hopes that they’ll adopt them.

I think that’s about a dream job as you can have, speaking at conferences, doing podcasts. Sounds like a really fun job to do. I asked you to come on the show and talk about Vault.

Vault is probably easier to explain by you. Could you give us the five-minute introduction? What is Vault?

Sure. Vault is HashiCorp’s solution, a recommended solution to secret management in the modern era. Vault works at a number of levels.

If you take a look at Vault’s threat model on vaultproject.io, we clearly call out the problems that Vault is trying to solve. But what it really breaks down to are two isolated threat areas. The first is internal threats and then external threats.

External threats are what we typically think about in secret management. Somebody sitting at the keyboard typing lots of commands, trying to hack or decrypt the data. internal threats are more like rogue employees or unauthorized access the difference between authorization and authentication within an organization so you know if somebody has access to the api credentials for example and then they leave the company how do we make sure that they no longer have access to those credentials do we invalidate them do we generate new ones do we audit etc So Vault provides a new and innovative approach to secret management in that it treats secrets as leased values, meaning they have a TTL associated with them. So unlike a traditional secret management solution where you generate a credential, like a database credential, maybe it’s a username and password to the database, You put that in the secret management solution and it’s very static.

The only way it gets updated is if an operator or a developer goes into the system itself, generates a new username and password, updates the data, etc. There’s decentralized keys and very limited visibility into that operation. Vault’s approach to that same task is to allow Vault to generate those credentials for you.

So Vault actually generates Postgres usernames and passwords and returns them as a result. And those credentials have a TTL associated with them. So they’re only valid for, let’s just say, 30 days.

And then after 30 days, those credentials are revoked and the application or the operator or the developer has to go request new credentials from Vault. So this makes the leases enforceable. And if someone leaves the company, you can revoke those leases early.

So even if it had a 30-day lease, we could revoke it at 10 days if that employee leaves the company, and those credentials are no longer valid. So Vault provides a centralized source for secret management all in one place. So we’re not worried about static secrets living here and living there.

Instead, everything is dynamic. And the dynamic nature makes Vault a highly scalable solution for large enterprises that have a ton of secrets with lots of different permissions. But it’s also a simple enough tool that even a small two or three person company can adopt it and it can bring great value to the organization.

That sounds great. I think nowadays most of us as developers either store our credentials in a simple INI file that we commit in the repository or we store it as environment variables or it’s something that we manually… keep up to date on our systems. It’s not something that scales well.

And as you mentioned, it’s one of the first things that we stopped doing. Perhaps the first few users that leave the company will rotate the keys, but eventually that becomes a burden, a drag. People stop doing it.

I’m very sure that there are a lot of systems out there where developers have left the company, but still have access if they wanted to. So something like Vault would be a perfect solution for that. Managing secrets obviously implies some kind of password or authentication or private key that gives you access to that data.

How does that work in Vault? Could you describe the authentication and security mechanisms that Vault has?

Yeah, sure. Vault, like most encryption systems, uses an encryption key under the system. It has a symmetric encryption key.

That encryption key encrypts the data in the durable storage backend. Where data is written The most logical there is disk, but Vault can also write to a database or it can write to in-memory. The encryption key is what’s responsible for encrypting that data once it’s sitting at rest on some type of durable storage.

That encryption key is a static key. Vault has commands that allow you to rotate that key at any time. And it uses a key ring so that it can automatically upgrade over time as new data comes in and data is rewritten.

That encryption key itself is actually encrypted by what we call the master key. And the master key never exists in Vault. Instead, the master key is decomposed into a number of shares.

And a threshold of those shares must come together to regenerate the master key. And this is done via an algorithm. It’s a computer science algorithm called Shamir secret sharing.

And the analogy I always like to use here is if you had a physical key to a door, you know, a deadbolt or a door lock, and you cut that into five pieces and you gave five people a piece of that key. If somehow any three people could come together and put their pieces of the key together, super glue them back together, that could unlock the door. That’s actually how Shamir’s secret sharing algorithm works.

So what we do whenever we initialize a vault, the process of initialization generates a master key. That master key is then split into a threshold of keys that get distributed to a number of users in your organization. And then that master key never exists again.

It only ever existed in memory. In order to decrypt the encryption key, we need the master key. And in order to get the master key, a threshold of users have to come together and enter their shard or their piece of that master key in order to regenerate it.

And this is Shamir’s secret sharing algorithm. In Vault, that shard, that piece of the key, is what we call an unseal key. So when a vault is created and initialized, it comes up in what’s called a sealed state.

The sealed state means that no requests are served and the data in vault is not available. It’s currently encrypted and we need to generate the master key so that we can decrypt the encryption key so that we can read and write the data. Only after a threshold of the key shares have been entered can we generate the master key, and then that master key can decrypt the encryption key, and Vault can be operable.

And this is the process known as unsealing. So if you’ve ever seen a movie where the president or the prime minister… has to enter the secret codes to disarm the nuclear missile, and three people have to enter their code in order to disarm the missile. That’s actually very similar to how Volt internally works.

And this is part of that internal threat that I talked about earlier. The Shamir secret sharing algorithm prevents one person from having complete reign over the system. Instead, there’s a system of checks and balances in place. such that no one operator can go rogue without the other operators in collusion or intervening.

The precedent comparison is a good one indeed. The nuclear missiles requiring five or more keys. As I recall, the amount of keys or the amount of parts in which a key is distributed is something that’s configurable when you first initialize Vault.

So if you would like a single key, perhaps the most dangerous option, a single key to decrypt or allow access to the Vault, that’s also possible.

Yeah, that’s true. The number of keys and the threshold are both configurable values. So here, the default is five and three, so it’ll split into five parts, and then three of those parts have to come together.

Here at HashiCorp, we actually have about 10 keys, but we also have a higher threshold. So there are 10 people who have an unseal key, and then I think five or six of them have to come together in order to unseal the vault. And that’s really a business decision or an organizational decision about how you should split that You do want to have a high availability of key holders.

So that means if you have a lot of people who travel or are in different time zones, you’re going to want to split those key holders up across the organization. However, one thing that’s really nice about Vault is that just because you’re a key holder doesn’t make you an administrator in the system. So for example… we could give our designer an unseal key.

And our designer only needs to enter that unseal key once. And he doesn’t actually need access to the vault at all. He just needs to be able to enter the unseal key.

So this isn’t just a developer or an operator thing. You could give an executive assistant an unseal key. You could give a marketing team an unseal key.

And they really just have to run one command to enter their unseal key, just as if they were unsealing a bank vault. All they have to do is insert their key. They don’t have to understand the mechanics of the bank vault.

So this allows you to distribute the keys among more than just your engineering teams.

Correct. So everything in Vault is policy and path-based. So everything is an API server.

The Vault CLI actually interacts with the JSON API directly. It doesn’t have any special magic. And everything is a path, a URL in Vault.

And the policies in Vault are tied to those paths. So you simply give somebody access to a path, and then they have access to the secrets that are contained on that particular path. The result there is that when you authenticate to Vault, you are assigned a policy, and that policy dictates what you can and cannot do in Vault.

The way you authenticate to Vault is done a number of different ways because it’s a pluggable system. So the most basic authentication is just by using username and password. The same way you would log into a website, you type in your username and password, you’re given a token back, and that token, just like a cookie, is how you’re identified moving through the system.

However, we do have more complicated workflows that tend to scale at larger organizations, things like LDAP, where your OU and CN define the policy that gets assigned to you. So anybody who’s in LDAP who’s a member of the developer’s OU gets a certain privilege assigned to them by being part of that organization. in LDAP. Similarly, we have GitHub authentication.

So just like the OAuth flow in the browser, you can supply a GitHub API token during authentication to Vault, and Vault will be mapped to a series of teams on GitHub. So anybody who’s a member of the developer’s team on your GitHub organization gets a certain privilege attached to them or policy attached to them on the Vault server whenever they authenticate to the system.

Okay. How do you define the root access, the administrator? Is that something that you do the moment that you configure Vault for the first time?

Is that something that you can add or redistribute later?

Yeah, so there’s a couple different answers there. The first time you initialize the Vault, Vault is going to give you out the initial root token. This is kind of the root user, the pseudo user, if you will, and it can be disabled after you’ve configured Vault further.

Usually what we see is that a small number of people are given admin or root access to the Vault via some policy, and then a number of other users are given developer access or operator access. It’s possible to regenerate that root key if you lose it or if somebody leaves the company, the same way that it’s possible to regenerate those unsealed keys that we talked about earlier. And that process is actually very similar.

It involves a threshold of the unsealed keys coming together to regenerate a new root token. And that root token does have kind of unrestricted access within the system.

Okay, that makes sense. You touched on it a bit earlier, I think. Vault has a storage backend.

It has to store the encrypted version of the passwords, of the secrets, of the API keys. What’s that like? Is that something that you’ve written yourself?

Is that an open source project you’re relying on?

So everything in Vault is open source. And this is just part of Vault’s core. So we have what’s called a logical backend or a logical storage backend.

And that’s how we refer to the durable storage. It’s pluggable. So you have to implement a series of API calls.

And then once you implement those, you can be a pluggable backend. Vault itself has support for the file system, Postgres, console, in-memory, and a few others that you can see on the website.

Right. The data is encrypted at rest, even while the vault is unsealed. So that means even if the vault is unsealed, if somebody were to gain access to the file system, they would just have encrypted data.

They would have to do some type of brute force operation or complex operation to try to decrypt that data, if it’s even possible. And we do use the latest ciphers and algorithms available. So AES, CBC-256 for the majority of vaults encryption.

Sure, that’s a great question. And the high availability is tied directly to the storage backend. So Vault’s high available architecture is designed in a way that each Vault instance shares the same logical storage backend.

So what that means is in order to have highly available Vault, you have to have a storage backend that supports leader election. So something like console or etcd are two examples that come to mind very quickly. The reason for that is, let’s say you have three vault servers.

When you initialize the first vault server, it’s going to configure console to store your data. When you bring up a second Vault server in the same cluster and you attach it to that storage backend, it no longer needs to be initialized because the folders and the setup are already there. Instead, that server immediately goes into standby mode because the backend supports leader election and says someone else is currently the leader, and that’s the Vault that we initialize first.

And then as we bring a third one on, that same process is repeated. We still have to bring the Vault up, we still have to unseal it, but the moment it’s unsealed, it goes into standby mode. Then any requests that get made to that standby vault server get forwarded to the leader.

And that’s because the backend supports high availability. If our leader should lose election, maybe somebody pulls the plug, it has a bug, something, it falls, it crashes, whatever. Then one of the two other servers will acquire that lock. as part of the leader election algorithm, and they’ll start servicing requests.

And as you bring new vaults online, all you do is bring them up, unseal them, and they become part of the cluster if they’re sharing the same storage backend.

There’s really no advantage at this time for putting a load balancer in front of it because even if you load balance to a standby, the standby is going to forward that request to the leader. Currently, standbys cannot respond to any requests involved. This is an optimization that our team is looking at.

So putting a load balancer like an ELB or HAProxy in front of it isn’t going to help you out at this time. So it’s best to either query console to say, you know, give me the vault server or just hit any vault server and they’ll forward the request on to the leader if it’s a standby.

Sure. So data center, vaults fill your domain is the data center. So it’s similar to console in that you wouldn’t want to namespace by data center.

Instead, you would run around one vault instance per data center, likely. It depends on your definition of data center, but you’re probably going to want one vault cluster per data center. In terms of naming things, there are no… best practices here, other than policies are tied to paths.

So whenever you name things, if you want to restrict access to a particular path, you can put it in a subfolder by ending it with a slash. So for example, if we write to the generic secret backend, which does behave very similar to Redis or Memcached in that it’s just an encrypted key value store, what you write in gets written, encrypted, and when you read it out, it gets decrypted and read back to you in plain text. If we put a secret at secret slash foo, that path is accessible to anybody who has access to secret slash foo.

But we can also put a secret at secret slash foo slash bar, and then we can further restrict those paths so that only developers have access to the bar subfolder, but other people have access to foo and some other secrets that might be contained in foo. So the naming scheme is actually, I wouldn’t focus on that. I always recommend that you start with your policies and say, these are the roles in my organization, developer, operator, designer, marketing, accounting, et cetera.

And here’s the access that I want them to have. So if you look at your current solution, you might have one password or LastPass or sharing a text file via Dropbox. And try to figure out who needs access to what and design the system really on paper and pencil or in a Google Drive and say, like, these are the organization roles and this is the data they need access to.

And then from there, it’s all path based. So you say, OK, I’m going to put all of the accounting data in slash accounting and then I can make a policy that says accounting has access to slash accounting and they get access to that data.

Okay, that sounds like a very transparent system. Good idea, I think, to approach this from the organization point of view. I think a lot of us would more easily look at this as, say, I’m a developer, I have an application, I know X, Y, and Z are secrets that I would like to store.

We probably look at it in a very development mindset kind of way, but this is more of a people and logistic problem than a technical one. So I like that idea. Does this mean that you can also tell Vault that there’s a difference between read-write and read-only access?

Yes, definitely. Vault actually has, I want to say, five different capabilities, which is what we define your ability to perform an operation. So you have read, which explicitly grants you read.

You have write, which explicitly grants you write. And if you give somebody write access, they have only write access. So if I give you write, you cannot read.

I have to explicitly give you both read and write capabilities in order for you to be able to perform both a read and a write. We also have capabilities for update, capabilities for list, and then delete. So they actually map to HTTP verbs.

So get, G-E-T, corresponds to read. Put and post both correspond to create and update. Delete corresponds to the HTTP verb delete, and list corresponds to the HTTP verb list.

So there’s a mapping there, and as we said before, it is an HTTP or HTTPS API, actually maps directly to the HTTP verbs that are used to make those requests. And everything is denied by default. So if I create a policy that is effectively an empty file and I assign that to you, you have no access in the system.

You can’t perform anything because we didn’t assign you any paths.

Yeah, so list is actually something new in Vault 0.5. It allows you to traverse a folder or subfolder to include the list of things that you have access to based on your policy. And this is especially useful in the generic secret backend where you may have a series of folders and subfolders and you just want to effectively LS or DIR if you’re on Windows and you want to see what are all of the folders in this folder or what are all of the secret names in this folder.

The list operation is never going to return the secret values. That’s actually going to require a separate HTTP call, the read operation. So it’s possible to give somebody list operations, list capabilities without giving them read capability.

And that would let them see that there are secrets there, but they wouldn’t be able to read the values. And a thing here might be like a UI where you want to display all of the secrets, but then you want to prompt for authentication once the user clicks on the secret name so that you can show them the data.

Okay, cool. Indeed. Vault secrets, everything eventually ties back to privileges, which if we put on the CIO hat, probably means someone using Vault will look at auditing.

Is there a way that you can see who accessed what and who tried to access something but was denied access? Is there auditing at work?

Yeah, so out of the box nothing is audited in Vault because audit backends require configuration. There are two configurable audit backends, file which logs to the file system and syslog which logs to syslog or rsyslog. The audit backends can be enabled using the audit enable command on the CLI and they only have to be done once even in a highly available environment.

Every single request and response, remember, it’s just an HTTP API server, is logged in Vault. You get in JSON, you will see the request payload and the response payload. Obviously, the first thing that probably comes to mind for a security conscious user is, wait, doesn’t the audit log include all of my secrets?

If it includes the response and the request object, that might include the secret data. Vault actually scrubs the audit log and performs an HMAC shot on all of the secrets. that are both requested and returned from the API. And this is incredibly important because what it means is that you can ask a Vault to generate the HMAC for a secret if you know it, and then grep the audit logs or search the audit logs for that particular value, but they’re not reversible.

So for example, if I want to search the audit logs for a particular Postgres username and password, I can give Vault the plain text username and password and ask it to generate the HMAC SHA for the audit log. And then I can grep the audit log for that HMAC shot and see any and all operations that involve that particular username and password.

You guys really thought this through. It’s a nice one. Makes sense from that point of view, too.

Can everyone see those access logs by default, or is that also something that’s policy-defined that you can limit?

So it is up to the organization to do that. If you’re piping them to the file system, anybody with SSH access and permission to see those folders obviously has the ability to see those audit logs. That being said, the audit logs don’t actually contain any sensitive information.

I say this… Very lightly, but there’s nothing that would prevent you from putting the audit logs in a GitHub gist and putting them publicly. They don’t contain any sensitive information that would be damaging to an organization.

I don’t recommend that. That’s definitely not a best practice for sure, but there’s nothing that would be harmful there. So I think it’s up to the organization itself to secure those audit logs.

One thing you can do is you can use Vault’s transit backend to feed the audit logs through the transit backend, which operates kind of as encryption as a service. It’ll encrypt the JSON audit logs and then write them out. You could write them back out to disk as encrypted files.

And then whenever you want to decrypt them, you post back to the transit endpoint with the encrypted value and get the plain text back. So that could be part of maybe your log rotate script is, you know, once the audit log fills up on disk, you pipe it to the transit backend and then write out the encrypted file to disk. And then whenever you want to recover it later, you pipe it back into the transit backend and get the plain text value out.

That does make things like grepping and searching a little bit harder. But if that’s something you’re interested in, you know, you might want to look at syslog or potentially a solution like Splunk that would allow you to do some more advanced querying or an elk stack that would allow you to do more advanced querying on those logs.

Yeah, I have a really detailed blog post that talks about Atlas, which is HashiCorp’s commercial product, the enterprise tooling, and how we use Vault in Atlas to encrypt data. So Atlas is written in Ruby on Rails, and it has a database that backs it. And part of Atlas is you have to give us… your cloud credentials because we run Terraform for you.

So you have to give us your AWS keys, your Google keys, your DigitalOcean API keys, and that’s very scary for some people. So what we do is we use Vault’s transit backend such that in the database, we never store the plain text values. Instead, we only store the obfuscated values, or sorry, the encrypted values that come from Vault in the database.

And then whenever we need the plaintext values back, we use a permission-based system, Vault’s policies, and we get the plaintext values back, but they’re revocable at any time. And that whole process is audited.

Nice one. That ties in with one of the things I wanted to ask you. What are the use cases that you see Vault being used in?

You’re using it yourself in Atlas. Are there other big projects you know that use Vault or interesting use cases?

Yes, but I can’t talk about them. It’s not really NDAs. I mean, some of them are, but Vault is security in general is one of those tools where half the battle is identifying the software used.

And a lot of these companies that are very security conscious just don’t want to admit that they’re using the tool. Simply because by saying publicly, you know, we use Vault, that immediately gives a skilled attacker an attack vector. And they can start going through Vault’s code base, trying to find vulnerabilities, you know, trying to attack that specific technology.

Whereas this is one of those cases where less is more. One thing I will say is that we have had Vault’s code base audited twice by an independent company. security firm and we have fixed all of the issues they have found they’ve also been very very minimal issues more denial of service attacks as opposed to security related issues and like linking leaking passwords and we fixed all of those issues in a very timely manner and the current version of vault pretty much passed with flying colors on the security audit

You rely on it on yourself, so you have to have faith in your own tool. That sounds good, too. I’m sysadmin by trade, so I do most of my work in config management tools like Chef or Puppet or Ansible.

I think I read your post on the Vault project blog, too, about integrating Vault with Chef. Could you describe the use cases that you can have with a config management tool? How could you use Vault to let that store your credentials?

How could that work?

Sure. I go into detail about this, like you said, in the blog post with lots of code snippets as related to Chef. But the idea is that Vault uses this TTL-based short-lived leases, as I talked about earlier.

And configuration management is generally run either once at boot time or on some type of service interval, maybe every 30 minutes or every hour. And the problem is that push versus pull model doesn’t work together very well. So if you think about Chef, because it’s something I’m personally familiar with, you know, you run Chef every 30 minutes as a cron job or as a service.

But if you have a secret that expires every five minutes, you can’t rely on Chef to renew that secret because the secret will expire before Chef has a chance to come in and renew it. And this is where we make a number of open source tools. Console Template and mConsole, which combined with Config Management, create really powerful application deployments.

So instead of having Chef retrieve secrets from encrypted data bags or communicating directly with Vault, Chef instead installs one of these tools. These tools run as runtime tools. They operate as services on the system, constantly in communication with Vault.

They speak directly to Vault and Vault’s latest APIs, and they manage the application lifecycle for you. So from the auditing and everything, you would use configuration management to write the files and the tooling that would then interact with Vault moving forward. Does that make sense?

It does. It’s sort of a duplicate config management system. Duplicate sounds too negative, but… you use your chefs your puppet to manage the console templates which in turn will do its configuration with vault and i think in return it will write on disk some kind of username password or whatever secrets that was stored stored in vault is that right yeah and then from there your application can pick up that file or you know your service can read that file or consume that file and similarly with m console except instead of writing to a file it writes to an environment variable Okay, which was also one of the questions I was thinking about is, let’s say you have a Ruby or a PHP application.

The easiest way for that application to get its credentials would be something like a console template that has written an INI or a YAML file that is parsed by the application. At the same time, Vault is sort of an API. You could essentially have the application query its secrets in real time.

That’s something I’m guessing is not really recommended by performance, or is that doable?

So it’s doable. The thing I caution here is the closer you get to integrating with Vault, the further you get away from customization. So the closer you are to Vault, the more control you have over the flow.

You know, when secrets change, what do I want to do? Do I want to drain database connections, etc. ? However, the closer you are to Vault, the harder development becomes.

Because if you directly integrate into your application, now all your developers have to run a Vault server to work on the application locally. And all of a sudden, you have to run 10 services just so you can edit one HTML file. And that’s not a really good development experience.

The counter to that is if you use a tool like console template, you get a config file and that config file is managed out of band of the process. The developers don’t need to be aware of console template or vault at all. Instead, they know that their data is going to come from a config file and that that config file will have a particular format and that they pull their database credential from this particular value.

And then if we take a step down even further, when we talk about security, generally people recommend environment variables because writing things to disk is generally regarded as less secure. What mvconsole is going to do is mvconsole is going to pull that data from the vault, never write it to disk. Instead, it’ll spawn a subprocess and inject environment variables into that subprocess, and that subprocess can be your application.

So in this case, developers only need to set one or two environment variables And they can get up and running. And those environment variables don’t have to be related to Vault. Or they can have same defaults, such that if they’re not set, they fall back to some developer credentials.

And this makes the process really easy and follows one of the modern best practices of storing secrets in the environment instead of writing them to disk.

Yeah, definitely something to keep in mind. You also, you know, you only want to inject secrets into the environment that are actually used by the application on that note. So, you know, you might have a million secrets involved and that doesn’t mean you need to put all million of them on the machine.

Instead, you should only request the secrets that are actually in use by that application. So, you know, if something like that does happen where you accidentally forget to turn debug off and somebody gets a production dump, that you’ve only leaked a subset of secrets as opposed to everything. And with Vault, luckily, you can revoke and regenerate those secrets really quickly.

And then combine that with the audit log, you can see if those secrets were leaked and are being used. But when you revoke a secret in Vault, it actually revokes the secret in the underlying system. So those credentials are no longer valid.

i think that that’s definitely one worth um pointing out the the the thing that you mentioned that it’s um revoking secrets in the underlying system that’s according to the documentation because i haven’t used vault yet so this is highly theoretical for me let’s say you’re using a mysql or a postgresql in vault there are root credentials to manage mysql and postgresql And you could have Vault generate new username and new password on its own and expire that from the database system on its own. And neither the developer nor the sysadmin even know which credentials are being used to communicate to the database. That’s an insane use case, I think.

It’s a really powerful one that sort of manages to get away with how a sysadmin would give the credentials to a developer.

Yeah, and you can have an application request those credentials, meaning a human never actually sees what those credentials are, and they only exist in memory for short periods of time. The other thing I do want to point out there, because I think it’s one of the coolest pieces of Vault, is that where possible, Vault actually uses the system’s underlying functionality to expire those secrets as well. So in Postgres, when you create a user, there’s a clause you can pass called a valid until.

And that accepts a timestamp. And in Vault, what you can do is whenever you create the user, Vault will tell Postgres that this user is valid until a certain date, and that date corresponds to Vault’s lease. Then if a skilled attacker or anything takes Vault offline, even if Vault is unable to make the SQL call to revoke those credentials, the underlying system, Postgres in this case, will revoke those credentials automatically.

So we do our best to make sure that the system is highly distributed and highly available. But then we also rely on the underlying system themselves to kind of double check our work.

Not as many as you would like. We’ve requested it a few times. The one that we really would like to see added is Amazon’s IAM credentials.

There’s currently no way to say that a credential expires in the future, and that would be really useful. That’s a thing that has been requested outside of Vault’s community. Vault will make the API call to revoke a credential after the TTL, but it’s something that we would like to see those credentials auto-expire from Amazon in the future.

Okay, a practical question then. If you’re an organization that’s working on a single large project that you want to put out, it probably makes sense to have one or multiple, for redundancy reasons, vault instances running to manage that application. Let’s say you are a web development shop and you manage 10 very large projects.

Would you run them all in the same vault instance or would you have a single vault instance per project that you manage for your clients?

I think that’s difficult. It depends on the separation of concerns there. Depending on the clients, I would say you would run a vault cluster for each application if they’re truly separate isolations.

I view it as if this application, one client’s application, should never have any communication with another client’s application, then… I consider those to be separate data centers and therefore should have their own clusters, their own networking, their own everything. I understand that if you’re running a VPC, for example, that might not be an ideal scenario, so I would run potentially a names-based vault server.

Again, everything’s path-based. so you can put things in namespaces and then provide access just to that particular namespace the same way that you know in my sql you can say that this user has access to these databases you could say that this application has access to these paths and you could restrict data that way so if you think about naming scheme ahead of time and with sufficient thoughts a single vault instance would be perfect and secure for managing multiple projects Yeah, for sure. And when you’re thinking about naming, I find it helpful to actually play with the service. And one thing we really tried to do in Vault was make it very user-friendly.

So you can go ahead now, go to vaultproject.io, download the tool. And when you run vaultserver-dev, which is short for development, you’ll get a fully unsealed, ready-to-go Vault server on your local laptop with instructions on how to use it. So out of the box, it works.

It’s not production ready. Obviously, it’s development mode. But it’ll give you a really great interface where you can play with it, try things out, break things.

And when you’re done, you just close the Vault server and it dies because it’s an in-memory server and it never writes anything to disk.

That’s a great way to get started. Speaking of which, your website, vaultproject.io, has an interactive in-browser shell, if you would like to call it like that. I found that to be really interesting to just get a hang of the project, the command line, what you can do.

Obviously, it’s limited, but it gives you a really nice feeling of what Vault is and its possibilities. That’s a really nice move to put it there.

That’s cool. I had a couple of more questions, but somehow they just popped out of my head. Is there anything else you would like to plug?

Are there cool ideas with Vault that we’re missing out on that we should know about?

I mean, the surface area on Vault is huge. It’s really hard to discuss everything in 20 or 30 minutes. We do cover a lot of it on our website.

As you said before, it’s vaultproject.io. There are both examples and then the API documentation there. There are a number of client libraries.

I don’t think we’ve discussed that. So it is an HTTP API. So we do have Node, Ruby, Golang, Rails, Java, client libraries that do allow application level integration, or if you want to build your own little command line tool or own little utility, there’s probably a package of your choosing.

And if there’s not, you can use your native languages interaction with HTTP and HTTPS to make calls to the Vault server.

So plenty of tools to integrate into your application. Even though, as you mentioned before, it may not be the best case to do, but that obviously depends on the environment and the current situation. Yeah, definitely.

Okay. I think I’ve got a lot of content to plug in the show notes as well. So if anyone has any more questions about Vault or how it works, there’s a lot of documentation on the site and I’ll include all of those links into the show notes.

One last question. I think you mentioned it briefly, but I’m not quite sure if I fully understood it. So the moment that you start a vault instance or service, you’re asked to initialize it and to create the seal that would allow you to unseal it, those key pairs that you mentioned.

You mentioned that they could be changed as well. How does that process work internally? That’s time consuming to rehash everything or how would that work?

No, it’s an online operation and it works just like the unsealing process. It’s called re-keying. So it requires a threshold of unseal keys to come together, just like unsealing the vault.

And there’s a nonce value that’s provided. And then once the number of key holders have entered their unseal key, a new group of unseal keys are generated that can be distributed and you can change the threshold and the number of keys during that time.

Okay, cool one. Because the moment you start it, you sort of unlock the vault, which allows you to lock it again with a different set of keys. Right.

Okay, makes sense. Great. I’m all caught up.

Seth, I’d like to thank you a lot for this very interesting introduction. I knew nothing about vaults coming in besides looking a bit at your documentation. Coming out of it, I think Evolve is a really interesting project to look at for anything that’s secret related.

So once again, thank you very much for your time. If people would like to find you online, how could they get in touch with you?

So my Twitter handle is just my full name. So S-E-T-H-V-A-R-G-O. And you can find me on the internet pretty much everywhere under my first and last name.

I will be at HashiConf EU next month. There will be a number of Vault talks and there is a Vault training as well. Myself and the principal engineers who work on Vault will be there if you have any questions.

And then we also have our conference in the United States in September. in Napa, California, where we’ll be doing some Vault workshops and lots of talks about Vault and security management, particularly around containers and in the modern era. So if that’s something you’re interested in, please let us know.

That’s cool. You’ve got a lot of traveling ahead of you. That’s clear.

A part of being an evangelist life, I’m assuming. Yes. Okay, Seth, thanks again.

Take care. And if you’re interested, there are a lot of interesting HashiCorp products vagrant among console. If you ever have more time, I’d love to discuss them with you.