AWS Systems Manager Parameter Store
At Entasis we help our customers understand and modify their technology stacks so that scaling is a function of cost not a function of technology. However, when we look at scaling one of the largest problems is the consistency of an environment. When we are running 100 application nodes then the opportunity for one node to go sideways and start handling traffic differently is much higher than if we only have 2 application nodes. So we work with our customers to help them abstract the configurations of their services from their code to enable their applications to auto-scale regardless of your hosting location. We are very familiar with public cloud, private cloud, multi-cloud and hybrid cloud. If you need help in this area please reach out to our team at sales@entasistech.com.This article is part of our series on Scalable Solutions. This article doesn't require knowledge of the other articles however please check out our other articles - Amazon Route 53 Basics and AWS Certificate Manager. Today we are going to look at one of the central components of AWS Systems Manager, and frankly, this is one of the easiest to implement. The Parameter Store is a centralized location to store configuration data. This helps us to know how our applications are being configured across an entire fleet. This allows us to avoid more brittle solutions such as file deployment, or shared file systems for configuration data. So let's dig in. For the purposes of this article, we are going to use the CLI to handle all of our interactions with the Parameter Store. Depending on your solution you will want to use the appropriate SDK to interact with the values in the parameter store. Additionally, you have the option of using another solution (there are a few out their or you can roll-your-own) that will load all of the values of the parameter store as environment variables, then the application can read its settings from the environment. For details on Parameter Store: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-paramstore.html
Parameter Store Values
- String – this is an unencrypted string.
- StringList – this is an unencrypted list of strings. This must be comma-separated with no spaces.
- SecureString – this is an encrypted string. This is encrypted with a KMS key if you use this your application is also going to need IAM permissions to use the defined key to decrypt items.
Recommendations:
- Always use SecureString, in my experience, it doesn’t add much complexity but it does provide the protection of having the data encrypted by default. This provides additional protection in the case of “grey” data which isn’t definitely sensitive (credentials) but can be sensitive in the wrong hands (connection endpoints, ports, etc).
- Use isolated KMS keys for each application and/or environment. If one application is compromised you don’t want it to be able to be used as a launch platform to other applications (imagine if you were able to pull down all configurations from all applications after compromising a single machine).
- Admin oversight is critical here you really need to look at what admins can do once you start solutioning for this, can your read-only admin decrypt the parameters in the parameter store for all of the different applications. How do you secure this and ensure that there is no leakage.
- Credentials can be stored in Parameter Store if SecureString is the key used. However, if you are using RDS (or other select AWS Services) Secrets Manager might be a better fit. The line should be if Secrets Manager can rotate the credential then it is better to use Secrets Manager for that credential. Then you can use a shorter rotation window (database credentials rotated once a day as an example). This limits the blast radius in the attempt of a compromise.
- Store all configurations in Parameter Store. All of them. If you put anything in local configuration files then that opens you up for divergence.
- Use descriptions in your parameters.
- Also, don’t include Parameter Store calls in your workflow. For example, if you have an API call that your workflow has to make then don’t perform a get-parameter for each invocation of that API. Rather refresh the local values on service reload/refresh/restart, then log any problems and fail to start the service if necessary.
Writing a Parameter in the Parameter Store
This pattern will use the default key of alias/aws/ssm, you can use your own created key bypassing the –key option. The output of these is simply the version number of the string created (always 1 if it is a new string).
$ aws ssm put-parameter –type [ String | StringList | SecureString ] –name [ name ] –value [ value ]
Now first we are going to create a parameter of the String type which is unencrypted. We will call it “naked string” and it will have a value of “unencrypted string”.
PS> aws ssm put-parameter –type String –name nakedstring –value unencryptedstring<br />
{<br />
"Version": 1<br />
}
Next, we will create a parameter of the StringList type which is also unencrypted. The big caveat with these are that your strings cannot contain any special punctuation as each string is separated by a comma. If your string contains a comma you are better off using a String or SecureString and separating from a string into multiple strings with logic (or by having each string in their own parameter). We will call this “nakedstringlist” and it will have a value of “unencryptedstring1,unencrpytedstring2”.
PS> aws ssm put-parameter –type StringList –name nakedstringlist –value unencryptedstring1,unencryptedstring2<br />
{<br />
"Version": 1<br />
}
Finally please forget the other two and just use this one. This is an encrypted string (everyone say yay!) and it requires a little more work on the IAM side but the security it provides is definitely worth it, additionally, it also makes it a sane proposition to store credentials here. We will call this “clothedstring” (since we are dressing it in encryption) and it will have a value of “encryptedstring”.
PS> aws ssm put-parameter –type SecureString –name clothedstring –value encryptedstring<br />
{<br />
"Version": 1<br />
}</p>
<p><strong>Updating a Parameter in the Parameter Store</strong></p>
<p>Now no doubt at some point if you are using Parameter Store you will need to make a change to an existing parameter. This is essentially the same process as above, we just need to tell it that we want it to overwrite the existing value.<br />
<pre>
$ aws ssm put-parameter –type [ String | StringList | SecureString ] –name [ name ]–value [ value ] –overwrite
This is simply going to update the parameter called “clothedstringlist” with the value of “updatedencryptedstring”.
PS> aws ssm put-parameter –type SecureString –name clothedstringlist –value updatedencryptedstring –overwrite<br />
{<br />
"Version": 2<br />
}</p>
<p>Now we see the output of the version number indicating that this has been updated to a new version.</p>
<p><strong>Reading a Parameter from Parameter Store</strong></p>
<p>So of course putting parameters in place is wonderful, however if we can’t get them back then it is massively non-helpful. So lets do that next.<br />
<pre>
$ aws ssm get-parameter –name [ name ]
Here we are going to retrieve our parameter called “nakedstring”.
PS> aws ssm get-parameter –name nakedstring<br />
{<br />
"Parameter": {<br />
"Version": 1,<br />
"Type": "String",<br />
"Name": "nakedstring",<br />
"Value": "unencryptedstring"<br />
}<br />
}
Next lets retrieve our parameter called “nakedstringlist”.
PS> aws ssm get-parameter –name nakedstringlist<br />
{<br />
"Parameter": {<br />
"Version": 1,<br />
"Type": "StringList",<br />
"Name": "nakedstringlist",<br />
"Value": "unencryptedstring1,unencryptedstring2"<br />
}<br />
}
Of course who can forget my personal favorite parameter called “clothedstring”.
PS> aws ssm get-parameter –name clothedstring<br />
{<br />
"Parameter": {<br />
"Version": 2,<br />
"Type": "SecureString",<br />
"Name": "clothedstring",<br />
"Value": "AQICAHhVdKbFht5ReQADBixWDt1CeIfSYZwDVCmC9QqTnFypxAENa7JnNj6xxG+VDWrVw8JDAAAAdDByBgkqhkiG9w0BBwagZTBjAgEAMF4GCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMOF4jFo0YR2cKU2xMAgEQgDFEmBYI8sWEIzSIJoLWU4xZBIMO/WNloQcIsVj7OPuk5VjqYxJJATyZYuUCgMeU3DUY"<br />
}<br />
}
So of course if we retrieve an encrypted string then it must come back to us encrypted. However that is not useful.
$ aws ssm get-parameter –name [ name ] –with-decryption
Lets pass it the option to also decrypt the parameter so we can use it.
PS> aws ssm get-parameter –name clothedstring –with-decryption<br />
{<br />
"Parameter": {<br />
"Version": 2,<br />
"Type": "SecureString",<br />
"Name": "clothedstring",<br />
"Value": "updatedencryptedstring"<br />
}<br />
}
Now one other interesting thing to mention. We have versioning on all of this, so lets say we want to go look at the previous version of the parameter.
PS> aws ssm get-parameter –name clothedstring:1 –with-decryption<br />
{<br />
"Parameter": {<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "clothedstring",<br />
"Value": "encryptedstring"<br />
}<br />
}
This versioning of course brings up an interesting point to ponder. If we have a parameter that starts out as a String (unencrypted) with a value of “Password1234” and we then realize our mistake and update it to be a SecureString (encrypted) with the same value of “Password1234” our previous version is still stored in unencrypted form. In this scenario I would encourage a few things…
- Change the password, not just because my example is a really bad password, but rather because it is now stored unencrypted in the cloud.
- If changing the password cannot be accomplished then a better approach would be to delete the parameter and recreate it as a new parameter that is encrypted. This will enable you to make the changes in your app which have you so dependent on that password not changing.
Delete a Parameter from Parameter Store
So now to set ourselves up for the next phase of this article we are going to delete everything we created.
$ aws ssm delete-parameter –name [ name ]
Earlier we created and updated 3 different parameters called “nakedstring”, “nakedstringlist” and “clothedstring” the delete-parameter does not actually return anything if it is successful so I am just going to give you them all at once since the explanation is not necessary.
PS> aws ssm delete-parameter –name nakedstring<br />
PS> aws ssm delete-parameter –name nakedstringlist<br />
PS> aws ssm delete-parameter –name clothedstring
Now we no longer have those parameters.
Parameter Hierarchies
Parameters can be stored in a hierarchical fashion, doing so can make it easier to craft scalable IAM policies which will allow applications to only access their parameters and not other applications parameters. For example /app1/param1, /app1/param2, /app2/param1, /app2/param2 we can create an IAM policy to allow read for /app2/* only and grant that policy to the role supporting app2, with a corresponding policy for /app1/*.
This helps us to secure our implementations from each other, however, it also makes it easier to code our applications to. For example in the above non-hierarchical examples we would need to read a parameter for each and every configuration, now keeping in mind that these configurations can be anything from connection strings, to credentials, to memory settings, to whitelists, or anything in between it is not inconceivable for us to have 20 parameters per application. Which means 20 calls, which means 20 delays (or opportunities for delays at least), when we utilize hierarchical parameters we can turn this into one call for all parameters at a given path. So the call can actually be something like /app1/* (or “give me all parameters having to do with app1”).
We can further segment it by environment and tiers, so here is a rough example of a hierarchy for an application called “squirrelbox”:
/squirrelbox/dev/database/readstring
writestring
/squirrelbox/dev/application/debug
maxmemory
/squirrelbox/dev/web/port
url
Creating our Squirrelbox Dev Environment
Let’s create the parameters that I outlined above with some dummy data to demonstrate how this might work in practice.
PS> aws ssm put-parameter –name "/squirrelbox/dev/database/readstring" –value "ReadDatabaseConnectionString" –type SecureString<br />
{<br />
"Version": 1<br />
}<br />
PS> aws ssm put-parameter –name "/squirrelbox/dev/database/writestring" –value"WriteDatabaseConnectionString" –type SecureString<br />
{<br />
"Version": 1<br />
}<br />
PS> aws ssm put-parameter –name "/squirrelbox/dev/application/debug" –value "true" –type SecureString<br />
{<br />
"Version": 1<br />
}<br />
PS> aws ssm put-parameter –name "/squirrelbox/dev/application/maxmemory" –value "500M" –type SecureString<br />
{<br />
"Version": 1<br />
}<br />
PS> aws ssm put-parameter –name "/squirrelbox/dev/web/port" –value "8080" –type SecureString<br />
{<br />
"Version": 1<br />
}<br />
PS> aws ssm put-parameter –name "/squirrelbox/dev/web/url" –value "app.squirrelbox.io" –type SecureString<br />
{<br />
"Version": 1<br />
}
Now lets assume that we have three teams: DBA, APPDEV, and WEBDEV. Now these teams are going to have different permissions for example perhaps the DBA’s can read/write to the /squirrelbox/dev/database/* however they have no permissions elsewhere. APPDEV’s have read to /squirrelbox/dev/database/* and read/write to /squirrelbox/dev/application/* and no permissions elsewhere. WEBDEV’s have read to /squirrelbox/dev/application/* (there would likely be a API endpoint parameter in there too that they would want to read) and they could read/write to /squirrelbox/dev/web/* with no permissions elsewhere.
Reading Parameters by Path from Parameter Store
Now let’s take a look at a few different ways to collect our parameters by path. This first example will only grab parameters that are under the path one level deep.
$ aws ssm get-parameters-by-path –path [ path ]
In our environment, this actually returns no parameters, since we have nested them deeper than that.
PS> aws ssm get-parameters-by-path –path "/squirrelbox/"<br />
{<br />
"Parameters": []<br />
}
If we add the recursive option this will give us all of them.
$ aws ssm get-parameters-by-path –path [ path ] –recursive
In an actual implementation IAM policies would determine what we see in the tree.
PS> aws ssm get-parameters-by-path –path "/squirrelbox/" –recursive<br />
{<br />
"Parameters": [<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/application/debug",<br />
"Value": "AQICAHhVdKbFht5ReQADBixWDt1CeIfSYZwDVCmC9QqTnFypxAHnlf92F7rVUYHWw+hiyd4WAAAAYjBgBgkqhkiG9w0BBwagUzBRAgEAMEwGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMe00Pz+/4yyVqtCUEAgEQgB82SyAFaxf2NC2m/RqOB4BI2Ug6+VsYX/sEmWovU7IZ"<br />
},<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/application/maxmemory",<br />
"Value": "AQICAHhVdKbFht5ReQADBixWDt1CeIfSYZwDVCmC9QqTnFypxAG7XidJjVTYL7xzz8sWxw3iAAAAYjBgBgkqhkiG9w0BBwagUzBRAgEAMEwGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMrt1iN9zqFLLwdzTyAgEQgB+fXMhRQJTgtMz5DDiQmjuN92xPAS9p4Ay0AEp78fmP"<br />
},<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/database/readstring",<br />
"Value": "AQICAHhVdKbFht5ReQADBixWDt1CeIfSYZwDVCmC9QqTnFypxAFAR2CpRSd7DJYMJs1+TD+bAAAAejB4BgkqhkiG9w0BBwagazBpAgEAMGQGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM9EZjxkJYZdRSPtHLAgEQgDdTEeHutrjQQH/S7cbTxjKF08XeIZPr+L5GtEuji9+njA/3subjWLJxlPTActjesYxmRrMAvmW5"<br />
},<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/database/writestring",<br />
"Value": "AQICAHhVdKbFht5ReQADBixWDt1CeIfSYZwDVCmC9QqTnFypxAGUJc6avKhbHFvdaYxTrALzAAAAezB5BgkqhkiG9w0BBwagbDBqAgEAMGUGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM/gavBkD1mzK6jTfTAgEQgDipBXD5HiVn9ImrYhBm8fnJOpoULlpFcShWT0YkoTnYvRKbVgKt+UoJtXWqy0NVLKVbvvSxfTmn+w=="<br />
},<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/web/port",<br />
"Value": "AQICAHhVdKbFht5ReQADBixWDt1CeIfSYZwDVCmC9QqTnFypxAHuyyqfVhe0yTjYZn0sTcH3AAAAYjBgBgkqhkiG9w0BBwagUzBRAgEAMEwGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM1t2/dsd3Y3mbvUz3AgEQgB/ExDPrZ0Xc0gOT3I1DUOiK0Rny/WFi32DhYuo2otoC"<br />
},<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/web/url",<br />
"Value": "AQICAHhVdKbFht5ReQADBixWDt1CeIfSYZwDVCmC9QqTnFypxAH2X9/mCJqpdzelmbSV9DRfAAAAcDBuBgkqhkiG9w0BBwagYTBfAgEAMFoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMOua5pSAK35Albf1jAgEQgC3tWc60afeByeKUQk20jBFwbLYzQ6GVzqtlVOhmLFNlU/6Tk6BUfjT0qQGj+30="<br />
}<br />
]<br />
}
So to see the unencrypted values we actually need to use the –with-decryption option like we did in our previous examples.
$ aws ssm get-parameters-by-path –path [ path ] –recursive –with-decryption
So lets take a look at what the DBA team would see.
PS> aws ssm get-parameters-by-path –path "/squirrelbox/dev/database/" –recursive –with-decryption<br />
{<br />
"Parameters": [<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/database/readstring",<br />
"Value": "ReadDatabaseConnectionString"<br />
},<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/database/writestring",<br />
"Value": "WriteDatabaseConnectionString"<br />
}<br />
]<br />
}
The APPDEV team would see the following.
PS> aws ssm get-parameters-by-path –path "/squirrelbox/dev/application/" –recursive –with-decryption<br />
{<br />
"Parameters": [<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/application/debug",<br />
"Value": "true"<br />
},<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/application/maxmemory",<br />
"Value": "500M"<br />
}<br />
]<br />
}
Finally the WEBDEV team would see the following.
PS> aws ssm get-parameters-by-path –path "/squirrelbox/dev/web/" –recursive –with-decryption<br />
{<br />
"Parameters": [<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/web/port",<br />
"Value": "8080"<br />
},<br />
{<br />
"Version": 1,<br />
"Type": "SecureString",<br />
"Name": "/squirrelbox/dev/web/url",<br />
"Value": "app.squirrelbox.io"<br />
}<br />
]<br />
}
If you use the Parameter Store to store all of your configurations for your applications it enables you to have a very resilient system that stays consistent across many nodes. In the event of configuration changes these changes can be deployed wide very quickly with a rolling restart of services to maintain up time.
If your organization needs help making your applications more elastic by nature, resilient in design, and performance on demand please send me an email sales@entasistech.com architecting scalable solutions is what we do.
