Embedded CloudFormation: using stacks in a non-compiled context

First off, I want to define the two ways in which I've learned to create CloudFormation templates:

1. Compiled: this method uses a client/server model that very much looks like the chef-client/chef-server model.  In this method small chunks of JSON are used to kick off a template build.  This template build process creates a large template file from many component parts.
2. Embedded: this process uses no pre-compile magic, but instead uses a "root" template to kick off n-number of sub-stacks using the AWS::Stack template resource.

Now let's define a use case that can help better illuminate the situation.  I'm going to use my favorite project of all times for this: CloudRim!

CloudRim has a fairly standard layout:

1. Contained in a VPC with 3 subnets:
1. Ops ( 10.0.0.0/16 - us-east-1b ): This is where the chef server, proxy, and jenkins slaves live, this is also the "jump box" also known as the "threshold box."
2. Primary ( 10.0.1.0/16 - us-east-1b ): This is where the primary, production application will run
3. Secondary ( 10.0.2.0/16 - us-east-1e ): This is where we put our HA backup for this region.
2. The application itself is a node.js application that utilizes multiple instances, each instance on it's own port, all ports are tied together with ASG's, ELB's, and finally Route53.
3. The data storage layer is a sharded MongoDB database where the mongos bits are running on the application servers.
4. Everything is configured with chef.

When we use the compiled version we'll end up with a group of stacks that looks like this:

1. VPC
2. OpsAuto
3. Application-A
4. Application-B
5. MongoDB-A
6. MongoDB-B ( repl set )

When using the embedded approach we get a slightly different layout:

1. Root
1. VPC Subnet Ops
2. VPC Subnet Primary
3. VPC Subnet Secondary
4. Application-A
5. Application-B
6. MongoDB-A
7. MongoDB-B ( repl set )

They look basically the same, however, one of the advantages that comes with using the embedded approach is that everything can be removed by removing the Root stack.  Obviously that could be a bad thing as well, depending on the use case.

Some people might be tempted to state that the repl set should be handled in a different stack, and I'm inclined to agree with that statement.  However, for the purpose of this document we'll stick with this layout so we can keep everything together in one logical construct.

The embedded approach is very compelling, however, there is one aspect of this that happens to be a significant drawback: troubleshooting.

The embedded approach is what's known as a "derived" approach, which is to say that you have to think of every single element as an abstract idea.  This is very difficult for those that are just starting out with the technology as it forces you to have to visualize how things are going to play out, but never actually seeing the final, compiled version of the template.  Troubleshooting in this environment is very difficult, even for a seasoned veteran like myself.

In contrast, the compiled approach is specifically targeted at avoiding this problem.  The compiled approach is much easier to debug, maintain, and supports a much faster iteration cycle.  Templates are used to make things faster and more agile, however, the draw back to this speed is that it requires a compile layer ( client / server ) in order to actually work.

Either solution can be integrated into a CI/CD system, and both systems have an equal pro/con list.  Choosing which method you go with really comes down to how much work you're willing to do in the long run.

The embedded version requires a deep knowledge of AWS and specifically how CloudFormation works.  However, the compiled version requires running a client/server service that looks and acts much like chef.

If your team(s) are familiar with AWS and have a basic understanding of CF, then the embedded approach does make more sense.  However, for larger, more complex scenarios where many business groups are going to be using this system, a more modular approach with the compiled method could save time and headache in the long run.

Ping me for code snippets.