Disclaimer: This blog post is not related to my current job with NIWC Atlantic or the Department of Navy whatsoever.

The Backstory

Infrastructure as Code...what? DevSecOps...Huh?

While scrolling through Tech Twitter a few months ago, I kept seeing buzzwords/phrases I knew nothing about, but being a lifelong learner, I was intrigued to know more. One tweet I stumbled upon was from Forrest Brazeal and The Cloud Resume Challenge. I was hooked.

I knew there were multiple cloud providers out there, but a few months before I came across this tweet I had decided to get into cloud computing through learning about AWS services and passed both the Solutions Architect and SysOps Administrator Associate exams. With that knowledge, the Cloud Resume Challenge gave me a reason to stick with just AWS for a bit. The best way I learn new material is by reading books and taking courses, so off I went asking my friends in the industry for their advice.

Introductory Learning through books/courses

My friend Alan told me if I was interested in IaC and DevOps in general I should start with The Unicorn Project and The DevOps Handbook both by Gene Kim

Reading through the Cloud Resume Challenge I also took note of tools/technologies I had no experience with and looked up introductory courses on A Cloud Guru. I took hands-on introductory courses coding in Python, GitHub basics, and building IaC through HashiCorp Terraform.

After reading a few books and some classes under my belt, I felt ready to work through creating my cloud resume...woohoo 🙃

The 5000-foot view

Below is the initial through process sketched out, along with my 5000-foot diagram that went through a few modifications during the planning/development phases. I don't want to give away everything to spoil the challenge for others trying to complete it themselves, but I will say Google, Terraform documentation, and Stack Overflow are your friends. 😉

Take #1 - Rough Sketch

Basic concepts are there, need to flesh it out

Take #2 - Initial Diagram

Initially I used just an HTTP API GET Method to return visitor counter total, which works but isn't super exciting. Also didn't lock down security headers or the API Call

Take #3 - Final Diagram

To make it more exciting, I decided to create an AppSync/GraphQL API behind an API Gateway POST Method, along with securing the API call using WAF ACL rules to showcase my understanding of the newer API structure

Even after being in the IT industry for over 15 years, the one aspect that was most challenging and most rewarding moving into cloud computing was so many AWS services being decoupled from one another. Sketching out how each service needs to talk to one another helped me complete this project in bite-size pieces.

Reading through Twitter there are lots and lots and lots of Cloud Resume submissions, I needed to figure out how I could make mine stand out just a little.

What to focus on/how can I stand out?

In keeping with DevSecOps best practices, I wanted to make my cloud resume as secure as possible throughout the process. Both in what I make viewable in open GitHub repositories and how AWS services interact utilizing the practices of least privilege and separation of duties.

10 Ways I completed the Cloud Resume Challenge as secure as possible

1. Don't expose API keys on GitHub/front end of the website - Not putting API keys on GitHub is a basic online open-source security posture. I used an API Gateway HTTP API in front of my GraphQL API and passed the API key as well. This gives me benefits of both API architectures (throttling from HTTP API, while also showing off newer GraphQL skills that I learned).
2. Use AWS Parameter Store for storing sensitive variables/Personal CMK encryption - I didn't want to expose any sensitive info on GitHub into my AWS architecture. Variables are stored as a SecureString value, encrypted with my own KMS key.
3. Use Terraform Cloud for secure remote state storage - I didn't want to store my state locally on my machine, I use 2-Factor Authentication in Terraform Cloud.
4. Lock down Security Headers - Used Security Headers best practices. Obviously doesn't mean I'm 100% secure but nice to see an A+ rating on Securityheaders.com and Mozilla Observatory.
5. Follow the rule of Least Privilege necessary - I locked down my roles/policies for specific resources ARNs and limited view/update capabilities in DynamoDB/Lambda/etc.
6. Lock down CORS - Wow CORS is frustrating, but I know I need to make it a best practice to lock down my allowed origin list to my domain.
7. Custom header to block direct public S3 bucket access - I pass a custom header value in CloudFront that is secret so that the public cannot directly access my S3 buckets.
8. Enable OAI on S3 - Restrict availability through Origin Access Identity to limit S3 bucket access to only the CloudFront user.
9. Subresource Integrity for all scripts - Subresource Integrity verifies that the scripts called from my website have not been tampered.
10. WAF ACL Rules to block GraphQL vulnerabilities-

    Disclaimer: I got this working and enabled but decided it was too expensive for me/overkill for my needs in securing AppSync queries.

My main goal in using WAF ACL Rules was to block the two potential possible security holes in GraphQL queries:

1. Introspection - Way too much behind-the-scene information about my API; this is enabled by default and shows the schema/all possible queries/mutations to run against your API.
2. Nested queries - Overtaxing the API by nesting a query within the same call.

I did enable the 5 free Managed WAF ACL rules, however more interestingly I created a rule to block both my concerns with introspection/nested queries. The rule does as exact string comparison so that no other query/mutatation/schema lookup is allowed:

"Name": "exact_visitor_counter_query",
"Priority": 0,
"Statement": {
  "ByteMatchStatement": {
    "SearchString": "{visitor_counter{body}}",
    "FieldToMatch": {
      "JsonBody": {
        "MatchPattern": {
          "All": {}
        },
        "MatchScope": "ALL"
      }
    },
    "TextTransformations": [
      {
        "Priority": 6,
        "Type": "NONE"
      }
    ],
    "PositionalConstraint": "EXACTLY"
  }
},
"Action": {
  "Allow": {}
},
"VisibilityConfig": {
  "SampledRequestsEnabled": true,
  "CloudWatchMetricsEnabled": true,
  "MetricName": "exact_visitor_counter_query"
}

Top 5 Frustration Points

Now for the "fun" things that made this Cloud Resume Challenge a challenge for me personally

1. Waiting on CloudFront invalidations - Impatience in the waiting on clearing the cache.

2. Testing my website with Security Headers and CORS enabled - Wow this makes local testing much more difficult when everything is locked down. Thankfully VS Code has extensions for a built-in HTML viewer, and most browsers make it trivial to disables CORS temporarily.

3. Terraform and DynamoDB not playing nice with my visitor counter value - Terraform wanted to reset my counter after every Terraform Apply, used a lifecycle rule to ignore counter incrementing as a change in infrastructure. Still researching if there is a better way to ignore this safely.

   resource "aws_dynamodb_table_item" "create_vistor_counter_item" {
   table_name = aws_dynamodb_table.create_visitor_table.name
   hash_key   = aws_dynamodb_table.create_visitor_table.hash_key
   item       = jsondecode(data.aws_ssm_parameter.table_item.value)
   
   lifecycle {
    ignore_changes = [item]
   }
   }
   

4. Terraform 'for_each' not playing nice with sensitive values. Since this project is hosted as a static website using an S3 website endpoint, I made concessions with my bucket variables being nonsensitive and removed this requirement. 😬

   for_each = nonsensitive(
    jsondecode(data.aws_ssm_parameter.buckets.value))
   

5. Initially I set up an HTTP API GET Request using API Gateway. However I remembered reading in the Cloud Resume Challenge book that not many people have tried using AWS AppSync to retrieve the visitor counter, so I decided to work my way through learning it. Love the simplistic code it took to get the value from the Lambda function, definitely worth the challenge to learn.

However, the frustration comes with securing a public-facing GraphQL API endpoint. I did decide API key authentication (easiest to implement, not the most secure authentication method) to secure it. I take the Shared Responsibility model to heart and implemented a routine changing of the API key.If this was a project that required heighten security, I would not use an API key but implement authentication through Cognito. Unfortunately AWS Parameter Store doesn't allow for an automatic method to change stored values, however using a combination of AWS Secrets Manager and Lambda you can make this happen.

The Path Forward and New Opportunities

The Cloud Resume Challenge has been quite rewarding by getting hands-on with many different AWS services. I do look forward to working my way through the “extra credit” challenges from Brazeal’s book, but I am also looking out for other projects to flex my DevSecOps/Cloud skills. I'm also on the market for DevOps-type jobs, so drop me a line if you have any leads, thanks!

Give thanks

I need to give a shout-out to the following people for helping me through my DevSecOps/Cloud resume journey:

• Shawn Melton for pushing me to get on GitHub
• Alan Crouch for book recommendations/great DevOps advice
• Jana White for design/color scheme assistance
• Kevin Wu for the title artwork