AWS Advice - Network Security

Much of the security of your AWS implementation is foundational.  Decisions that you make early on have the potential to impact the architecture of your system for a long time, often longer than you think.  And because these factors are so foundational, changing them has more impact, often in the form of downtime or a more complex migration.  It's important to carefully consider these up front and make your implementation as good as you can for your use case, but know that it will need to change at some point.

Automate Everything

This is an easy decision to make, but a hard one to honor.  It's vitally important to drive your infrastructure definitions through version control for traceability and to provision them only from those versioned changes for reproducibility.  Traceability becomes critical for audit and compliance.  Reproducibility is the bedrock for delivering high quality solutions.  Environments must be consistent to test potential release candidates and improve the system through testing cycles.  Automation also has the positive side effect of being quicker and easy to run.  In the cloud this means that resources can be built and torn down at will.  Treating your resources in this way, as cattle and not pets, can yield cost savings, scalability, improved security, and improved system testability.

VPC Network ACL

The VPC is the AWS resource that represents an isolated network.  When deciding how to scope your VPC(s) it's important to consider your options for network level security.  You'll have a stateless network ACL at the VPC level, giving you the ability to allow and deny CIDR ranges for all subnets to which the network ACL is attached.  Rules on network ACLs should be broad security definitions.  Here are some AWS recommended ACL rules for your VPC.  For instance, if you expect internet traffic into your VPC you may choose to allow port 443 globally.  You will likely need to create rules for intra-VPC, inter-subnet traffic too, as the network ACL operates at the subnet level.  That is, traffic leaving subnet-A destined for subnet-B, will be filtered by the attached network ACL, even though subnet-A and subnet-B are in the VPC.  These ACLs should be broader definitions, possibly port ranges for microservices, but should be secured to your VPC CIDR range.  An important sidenote - since the network ACLs are stateless they are unaware of connections sourced within your VPC.  You'll have to craft inbound rules that allow responses to traffic generated by your applications and services (and vice versa).  Overall, when building network ACL rules think broad, environment-wide rules.

Security Groups

The network filtering complement to the VPC network ACL is the security group.  Think of security groups as single purpose firewalls.  These are stateful and operate most effectively at the resource level rather than the overall network level.  Here are some important notes on security groups.

• Stateful - no need to worry about rules for response traffic. They're aware of active connections and allow accordingly. 
• Can be attached to a number of resources including EC2 instances, load balancers, and RDS instances, just to name a few.
• Take advantage of the fact that these can be applied granularly.  You can get a win-win in the form of security and documentation if you apply rules at the most granular level (i.e. the service or application).  You'll only be allowing what's absolutely necessary (security++).  And you will now have documented(++) the fact that X service requires this/these port(s).  If you want to take this a step further you can even write a test suite that checks your security groups, asserting these granular rules.  The unfortunate reality is that it's extremely easy to rely on too few security groups, apply broad rules across multiple servers / systems.  You end up with a confusing, coupled, security group spiderweb that's extremely challenging to untangle.  Check out aws-security-viz to see a visualization of your security groups.  Very enlightening.
• Limits - there are limits on the number of rules per security group, how many security groups per VPC, and how many security groups a resource can have.  The limits change, but start with:  VPC service limits

Subnet Accessibility

Most companies typically design their network with at least an internal network and an externally-accessible network, often called a DMZ.  You can achieve this same effect in at least 2 different ways with the VPC.

Your first option is what I'll call the all-in-one subnet option.  Create one subnet per AZ, as usual, and logically divide internal vs internet-facing resources servers by only allocating public IP address to internet-facing servers.  Servers that only get a private IP address will not be accessible on the internet, achieving the typical internal subnet effect.  Because public and private servers are colocated in the same subnet your network ACL(s) will have rules for internet traffic and inter-subnet traffic.  This isn't necessarily bad, just something to be aware of.  And you'll of course have security groups wrapping your servers with more specific rules.  With this approach you will need to decide whether your subnets default to assigning a public IP address, or not.  My preference is to avoid mistakes, default to private, and require deployed servers to explicitly assign a public IP.  Now, the most significant downside to this approach is you'll likely want at least some of your private servers to still access the internet for something - hit APIs, download packages, etc.   Internet-accessible servers require an internet gateway, whereas private servers will require a NAT to access the internet.  The route table destination for both of these would normally be 0.0.0.0/0.  So, unless you're willing to identify the destination CIDRs for the private servers you'll have to go with the second option.

The second option is what I'll call the DMZ subnet option. In this scenario you're not mixing internet-facing and internal servers in the same subnet.  You'll have public and private subnets.  Here are the basic steps:

• Create two subnets per AZ.  For the internal ones make them default to not assigning a public IP address.  For the internet-facing ones (DMZ) make them default to assigning a public IP address.
• Your internal subnets will likely require larger CIDR blocks.  The DMZ subnets should only really host your NAT Gateway and your internet-facing load balancers.  Unless you have a ton of ELBs, these subnets can likely be smaller, although there's no downside (other than available IPs) to making them large.
• Build two network ACLs.  One will be attached to the DMZ subnets.  For that one you should limit the traffic, ideally, to HTTPS (443) only.  If you can't, grant the minimum.  Remember, it's stateless, so you'll have to add rules for response traffic and traffic to / from your internal subnets.  The second ACL will be attached to the internal subnets.  This should be more restrictive, likely without any 0.0.0.0/0 rules.
• Create a NAT Gateway.  Attach this to your public subnets.  This is how your internal servers will still be able to reach outbound to the internet.
• Create an Internet Gateway.
• Create two route tables.  The internal route table should have a 0.0.0.0/0 rule pointing to the NAT gateway you just created.  The DMZ route table should have 0.0.0.0/0 pointing to the Internet Gateway.

An important caveat on security group limits

Most of these limits are in place to ensure that AWS can guarantee service levels.  Obviously, the more rules that must be evaluated to make a network filtering decision, the more time and resources required.  Behind the scenes your security group rules evaluate to jump rules.  If you specify a CIDR range in a rule, this evaluates to one jump rule.  A direct IP comparison can be done on the traffic.  If you reference another security group from which traffic should be allowed this can result in a large number of jump rules.  Essentially, the referenced security group is taken and the resources to which the security group is attached are resolved.  If the referenced security group was attached to 10 other AWS resources, then 10 rules would be created behind the scenes - one per attached resource, so that the underlying firewall can do the proper IP comparisons.  In essence, referencing security groups is very convenient, but suboptimal when applied broadly.

I recommend specifying CIDR ranges wherever possible.  It does, however, make perfect sense to reference security groups in an individual deployment stack.  For instance, if you have a typical stack with load balancer, app server, and database you can give each their own security group.  The load balancer might allow port 443 for HTTPS traffic, then be forwarding to port 8080 on the app server.  The server security group can allow port 8080, but reference the load balancer security group, effectively allowing traffic only from the load balancer.  Similarly, the database might allow traffic on port 3306, but reference the server security group, allowing traffic only from the app server.

If you craft rules that result in a large number of jump rules you will likely get restricted to a maximum of 100 security groups in your VPC.  Rearchitecting out of this design can be a significant undertaking.  I highly recommend that you instead design your AWS environment to have smaller VPCs in which you expect deployed applications / services to communicate with one another.  This will allow you to mostly rely on CIDR ranges.  If you instead deploy disparate applications to the same VPC and expect to filter their network traffic you'll be forced to create logical "environments" within your VPC using security groups.  This is where the jump rules start, and proliferate.  At a minimum, plan to deploy separate VPCs for each of your environments - test, prod, etc.  That way you can prevent test services from talking to production services with CIDR-based rules.  Then you can deploy individual stacks with security group references like in the example above.  You will have more security groups, but AWS will approve an increase from the initial 100 limit because your total number of jump rules will be very low.

What else?

There are a lot of considerations when you design your AWS environment.  In upcoming posts I'll talk about:

• s3
• iam - users, groups, roles, and policies
• logging

Hopefully this helps as you think about your VPC design.  What other considerations / implications have you come across that impacted your design choices?  Did you discover anything after the fact that caused a significant redesign?  I'd love to hear from you. 

Jurassic Delivery

Who didn't love when Jurassic Park hit in 1993?  Hot damn if I didn't own that on VHS as soon as it came out.  Velociraptors. T-Rexes. 3-D GUI Unix operating systems.  It was the shit.  There have been some downturns, but when Jurassic World came out with Chris Pratt in a leading role I couldn't pass it up.  Clearly things had been taken to the next level and I needed to submit and enjoy.  I saw it later than most and, despite reviews I heard, I thoroughly enjoyed it.

I love when humans try to control the uncontrollable and pay the price.  If you're going to build a dinosaur park... I don't care how much effort you put into trying to control the dinosaurs... you are going to lose.  There's something very gratifying for me watching these wannabe puppeteers suffer the T-Rex bite they genetically engineered, bred fierce, and sought to tame.

Software delivery is a vicious dinosaur.  You start with an idea.  So innocent, harmless, but requiring so much care and nurturing.  You grow it gradually.  It begins to require more and different kinds of care to keep progressing.  As your little dino grows you realize that he's getting unruly and you need to build some safeguards into your system and delivery process.  Maybe you need to add some tests, some automation.  Before you're done with any of that... BAM a major bug hits.  Your dino's fully formed teeth are capable of biting through the steel cable you engineered to keep him in.  We'll introduce a stronger, higher gauge cable, AND let's electrify it.

Lather. Rinse. Repeat.

We all face significant challenges that we need to solve on behalf of our customers.  There will be an endless string of problems that we run into along the way.  We'll solve some of those by leveraging any combination of libraries, frameworks, and solutions - some well-known, some well-understood, some cutting edge, and some not well-understood.

You have options.  More well-known technologies likely have a larger support base.  They probably also have more and better documentation, and a community that is seeking and solving its problems. On the flip-side, technologies that are more well-known and understood may not be solving the latest problems, and maybe not in the most effective way.

Cutting edge technologies are likely solving or streamlining more problems, or more significant problems.  They're a jump forward of sorts.  Maybe you can solve the problem in far fewer lines of code.  Maybe concurrency is simplified immensely. Maybe deployment and scalability become low-hanging fruit.  Being a newer technology though, it is likely not as well-understood, possibly not as well-documented.  Certainly the adoption level is lower, which means that community support is going to be lower.

The former might result in a more manageable, tamable stegosaurus.  It's less rapid than other approaches, but consistent.  Your stegosaurus is going to eat, sleep, and shit in a fairly predictable manner.  It's not going to bite you, and any fires it may start will be manageable.

The latter might result in an unmanageable, unpredictable T-Rex.  It's fast, vicious, and will bite your head off as soon as it gets the opportunity.

Remember when Dr. Grant and the kids were running through the field as a flock of dinos ran past them?

These guys seem reasonable.  Extremes are rarely the right answer.  Maybe a Gallimimus software delivery pipeline is a proper middle ground?

Whether you realize it or not, you own the characteristics of your delivery pipeline.  The series of choices you made since the inception of your idea created your pipeline.  It's not done though.  You are constantly adjusting and molding it.

Investing in change... in new and valuable technology is important.  After all, who doesn't want to avoid solving solved problems, and leap forward?  But, we must treat it as an investment.  If it's the right business decision do it, but do it intelligently.  Don't take on a conversion or technological change expecting stegosaurus-like outcomes.  You might have a raptor on your hands.  You have no business taking on a change like this without an investment in understanding the ways that it can bite you and accounting for those.  If you think you're going to convert from a COBOL stack to a Java stack, a .NET stack to a Scala stack, an on-prem stack to a cloud stack, or any other kind of major conversion, expect unpredictability.  Likewise, don't breed a T-Rex / velociraptor hybrid without expecting some casualties.

As you move forward make intentional decisions about the state of your pipeline.  Consider:

• Your current state
• The relative newness of the thing you're evaluating.  Is it well-known and understood?
• The learning level and talent of your engineering organization
• The trust level your management team has for your engineering organization

Maybe the right thing for your organization is to engineer your own dinosaur.  But mayyyybe not an Indominus Rex.  Maybe a stegoraptor though?

Thoughts on DevOps

DevOps, like most paradigm shifting buzz words, has become an overloaded, muddled term.  I've been thinking about this a lot lately and here are some uncategorized thoughts on this rapidly evolving area of software delivery.

(Most of these statements should probably start with.. "regardless of where you are today")

• Focus on customers.  The culture and organizational change associated with DevOps should make everyone involved in the delivery of a solution (including infrastructure roles) more aligned and accountable to the customer.
• I view this primarily as a movement to apply engineering practices to infrastructure and operations management.  Versioning, automation, testing.

• DevOps is about dependency removal, in much the same way that agile is.  There's no better way to remove dependencies than to add the function of that dependency to teams requiring it. Add people with the skills, or grow the skill set within the team.
• I believe one maximizes agility by removing all dependencies and allowing a team to create, manage, and run its entire stack.  
• Teams running their entire stack leaves the potential for similar, possibly duplicate effort across teams.  While duplication is evil, its tradeoff is agility.
• A centralized "DevOps" team is completely reasonable in my view, but it should not be how an organization starts.  The formation of a central DevOps team should be a conscious decision to follow the DRY principle - to remove duplication that has emerged organically.  As well, the team must not become a bottleneck as teams evolve / change. 
• Ownership boundaries are clearer with fewer dependencies.  If a dev teams owns the app code and ops owns the infrastructure, who addresses an unclear problem near the boundary?

Still thinking...

The Feature Toggle Antipattern

The move toward a more agile software delivery model requires the adoption of improved technical practices.  One of the first is generally the concept, and tooling associated with, continuous integration (CI).  The adoption of CI practices yields other challenges, one of which is partially complete features.  Many features take much longer to complete than a best practice integration cycle.

Feature toggles are a very useful way to solve this problem.  By employing this concept you can effectively decouple commits and code integration from the release of a feature in that code.  This is very powerful.  We can now get the benefits of continuous integration without the obvious issue of exposing a partially completed feature.

Like with all things we can take this concept too far.  Martin Fowler advocates for avoiding feature toggles to hide things in production:

Your first choice should be to break the feature down so you can safely introduce parts of the feature into the product. The advantages of doing this are the same ones as any strategy based on small, frequent releases. You reduce the risk of things going wrong and you get valuable feedback on how users actually use the feature that will improve the enhancements you make later.

He simply suggests embracing your agility.  The need for these toggles means you're already releasing to production more frequently than you can complete features.  Why not break the work down further, and learn from each release?  Here Martin is suggesting avoiding features that will take longer than your release cycle.  Instead, break them down.  But a frequent production release cycle is good.  Don't attempt to solve this by releasing less frequently...

Not everyone may be able to accomplish this easily though.  It's a worthy goal to improve to over time.  However, teams need to monitor for over reliance on these toggles.  So a couple things to watch out for:

1. Completed, but not released features.  If you've completed a feature you should be ready to release it.  Otherwise what could you have been working on instead that could be released today and have added customer value today?
2. The number of hidden features.  If #1 is a problem, this is likely also a problem.  However, this can also manifest if you have too much work in progress (WIP).  Reducing WIP can drive feature completion, therefore releasability, and therefore customer value.

If taken to an extreme the number and scope of features that are hidden in production can reach cumbersome levels.  I call this the Feature Toggle Antipattern.  In its worst form agile teams lose sight of their stockpile of not-yet-released features, even releasing (i.e. no longer hiding) features less frequently than in a typical waterfall project.

In a waterfall project there is a clear beginning and end, often to a fault.  That's one of the things that agile overcomes very successfully.  Aligning teams around a product and driving features through that team eliminates the on / off nature of waterfall projects - the eminent big bang.

With frequent releases (no big bang) it's easy to lose sight of the customer value that's hiding in your toggles.  When it's time to actually release those features you could end up with a big (detoggling) bang that dwarfs its waterfall alternative.  There will be other loss too aside from the customer value opportunity loss.  If you encounter any issues, or as you get feedback from customers you'll need to change and adapt.  Many of those features were developed a long time ago and you have all the cost of context switching and refamiliarization.

If you're going to use feature toggles to deliver your product make sure you avoid this antipattern.  You'll avoid many of the heartaches that drove you to embrace agile in the first place.

The Cloud Decision

For some there may not even be a debate when it comes to the cloud.  The flexibility and scalability that it offers small companies that don't want to build their own datacenter and are not sure about the size of their customer base (their viral coefficient even), is invaluable.  For larger companies the evaluation is generally more difficult.  The primary driver is usually cost, and that savings must be measured against all the other change that's necessary - in security, architecture, and governance to name just a few.  Any good startup (or smaller company) has a strong culture of innovation.  After all that's how startups start.  Innovation is a critical element of the cloud decision making process that too easily gets lost in the evaluation for larger companies.  This, and other value generating endeavors are amplified by a service enabled infrastructure.  Particularly one where self-service is encouraged.

Looking at the cloud decision from the CIO level is just too high.  Doing that is going to dismiss the most significant benefits.  I imagine the typical evaluation goes something like this:

• Cost.  In the cloud we can pay for what we use, and not worry about underutilized assets. Ok, that's a +.
• We'll really need to ramp up security.
• Let's do it!

It may be true that an organization will lower costs by doing just this.  However, there is an enormous amount of lost opportunity in making this move so naively.

If you have an on-premise datacenter you probably have dedicated infrastructure teams.  You've probably also built up processes for development teams to interact with those infrastructure teams. Focusing on cost and ignoring the self-service, service-enabled nature of cloud providers might cause you to reimplement your existing datacenter, architecture, and processes in the cloud, avoiding the majority of the benefits.

Let's take a simple example that I recently heard to illustrate a company that recognized the value of the self-service model and reaped the benefits.  A developer at a larger company supported an existing, cumbersome process to make regularly released files available to external parties.  That process was one where, once her releasable artifact was built, she sent and notified another team of its availability.  At that point the other team would "approve" its release and put it on an external facing FTP site.  The process took several days on average.

In their cloud migration / implementation this developer was empowered to use the available services.  She recognized that the cloud storage solution now available could easily replace the FTP site and the to-be-released artifact could be automatically sent to the storage solution directly from the build process. The net result was an automated, reliable process with immediate results rather than a multi-day lead time.

There are two reasons this succeeded.

1. The developer was intimately familiar with the process.  Enough so such that she could recognize and implement the improvement.
2. There was a conscious decision to empower her; to allow her access to cloud services.  Her organization could have easily restricted access the storage solution such that only the FTP team (or other infrastructure team) had access.

An enormous benefit of moving to the cloud is its service-enabled nature.  Organizations with manual processes and hand-offs have all the opportunity in the world to take advantage of this.  There is a key though.  The true benefit only occurs when there is a reduction in dependencies.  Reduction. In. Dependencies. This must mean that requests are not necessary; that teams can "request" infrastructure via a console or API, on-demand, and not depend on an external entity.  

The on-premise datacenter versus cloud provider decision is a difficult one.  It's one that I do not think should be made lightly, and one that should not be made for cost reasons alone.  Organizations need to make sure they recognize the real benefits and take advantage of them.  This can be a very large change.  In many cases it's a culture change, an architecture change, and a governance change.  Think through what is means to reduce dependencies.  Roles may change, and skill sets may be challenged.  This requires great leadership, trust, and maturity to accomplish successfully.  I'll end with a sobering statistic from VMWare:

63 percent of Amazon AWS projects are considered failed, compared to 57 percent of projects on Rackspace and 44 percent of Microsoft Azure projects.

Single Responsibility Principle 2.0

I see this concept coming back a lot as of late, at least in my ongoing learning and discovery of good design and architecture.  The Single Responsibility Principle is one of the SOLID principles for good object oriented design and development. I think the SRP, and likely many other principles, have become applicable at higher levels of abstraction.

Some amazing advances in approaches to infrastructure and configuration management, deployment, scaling have hit our industry hard in recent years. Historically, before virtual machine technology was really the norm there was still a need and desire to consolidate applications and services to run on a minimal physical footprint.  After all you want to use the hardware that you've purchased effectively. You don't want to run a beefy server at 10% utilization all day, so you load it up to more fully utilize it. This drives a lot of coupling at the infrastructure level of our architectures.

When VM technology became heavily adopted this started to become less of an issue.  You could use the same physical hardware, but host many logical VMs on it.  Thus, you've separated more of the concerns, thereby further decoupling co-hosted applications.  This improves architecture, but trades for increased complexity and load on infrastructure teams.  If we're going to build more single-purpose servers then we'll need more VMs.  This spawned the need for greater automation in the infrastructure space.  Along come tools like Puppet, Chef, SaltStack, and Ansible.  These tools have done an amazing job fulfilling exactly this need.  Write your infrastructure as code, version it, and leverage it for infrastructure, on demand.

We now have tools that enable us to rethink our approach to design and architecture in support of the single responsibility principle.  In the early 2000s when Uncle Bob wrote about the SRP he raised visibility to a concern that we should be asking ourselves as we design and change classes.  It's now 2015 and our tools have advanced incredibly.  We need to ask ourselves this same question as we design all layers of our systems.

How can we leverage the SRP to reduce coupling not just in our classes, but in our application / service design, and in our infrastructure design?  

Modern tools enable, and frankly necessitate, that we ask ourselves this question across our entire stack.  It's really this thinking that drives teams and organizations to microservice architectures.  A class that is designed with a single responsibility changes for one reason and one reason alone.  This keeps classes small, thus easier to change.  Services should be easy to change as well.  What better way than to enable that than to give each its own, single purpose and therefore small, changeable implementation.

I have been particularly interested in Docker lately.  I believe it adds a tremendous amount of value in this space.  Despite great tools like Puppet, Chef, etc., their approach is still bulkier.  With Docker's lightweight VMs the creation of individual service or application images is fast, and spawning them is even faster.  You can start a Docker container just as fast as your service itself can start.  Docker also really shines in its simplistic mechanism for linking together containers for interactions.  If you have a typical web server, app server, database architecture it does not take long to get those pieces Dockerized and running together, linked, in a Docker environment. I really like what Fig did to simplify this even further.

Many of us likely have some work to do to improve our applications and systems in consideration of the SRP.  There won't be any shortage of work there.  The good news is that there is little holding us back when it comes to available tooling.  It's there and most of it is free.  Despite the challenges it seems to me that it's worth giving some thought and moving in that direction.  More easily changed services will yield business agility, and therefore customer satisfaction.

Agility, Technical Leadership and the so-called Talent Shortage

My exposure to the agile community has not been super broad, but from what I have seen people tend to talk about agile in two forms.  Or at least I perceive two distinct discussions.  There is the "process" side of agile.  This includes things like planning, communication, team norms, managing deliverables, prioritization, etc.  Then there's what I call the technical practices side of agile.  This includes practices like continuous integration, test first development, automation, pairing, etc. - mostly those practices borne out of extreme programming (XP).

I'm a whole-hearted believer in these technical practices.  I believe it's these practices that form the foundation for agility.  You can do XP without the agile processes, but you can't be agile without XP.  Even doing XP in a pure waterfall world would yield huge productivity gains.  For that reason I mostly equate one's agility with one's strength in technical practices.  That's not to say there aren't significant gains to be had with process change.  You just can't get functionality in the hands of customers faster than you can build, test, and release it.  And the speed of your build, test, and release cycle is equivalent to how much of it you've automated.

A few years ago Andy Singleton posted Tech Leads Will Rule the World.  I've had it bookmarked ever since.  I liked what it had to say then.  I firmly believe it now.  Businesses are desperately fighting for agility as competition continues to increase and as software disrupts our world.  To achieve the kind of agility that is so critical now and for future business viability, technical practices could not be more important.

When it comes to adopting technical practices there is no one more important than tech leads.  These are the key influencers with the ability to set team norms, and most importantly, the ability to lead by example.  The only way to successful adoption is strong technical leadership.  The quickest way to peril is poor technical leadership.

But how do we all get there?  Especially with all this talk of the technical talent shortage.  Andrew Clay Shafer has taken a strong position on this topic.  He's well worth listening to.  Yes, we need to attract great talent.  First and foremost we need to look internally and treasure the strong technical leadership that we have today.  Your tech leads have a profound impact on the culture of the teams with which they work.

Do you know where your true tech leads are today?  If you're not sure look no further than your highest performing teams (they'll have the best technical practices).  And your "tech leads", well, you know how to identify them now too.

DevOps is Blue Ocean

I am fortunate to have recently attended the inaugural DevOps Enterprise Summit (DOES).  The event consisted of ~600 tech professionals from around the world.  It had an aspect to it that I really liked.  The event was not all about rockstar, industry-leading companies and professionals telling their stories.  Many presenters were and are change agents and leaders from larger companies with slower-to-change cultures.  This resulted in a unique, and fantastic conference.

On my way to the conference I decided to read a book that was given to me over a year ago – one that I had not yet prioritized high enough to read.  I’m glad I adjusted my backlog.  The book is called Blue Ocean Strategy and was published in 2005, so I’m a bit behind.  The premise of the book is basically this:

Companies spend most of their time competing in “red oceans”.  Red oceans are where the blood is.  The market is very competitive, profit margins have been driven down, and in order to remain competitive you have to scale.  The book advocates looking for and executing blue ocean strategies.  Blue oceans are where there is a gap in the competition.  There aren’t the same competitive challenges, and therefore the opportunity for massive growth still exists.  This may seem obvious, but the implementation is often not.

It’s easiest to illustrate with an example.  The first one in the book is about Cirque de Soleil.  Traditional circuses compete on a variety of characteristics that include (I’m sure I’m missing some):  price, use of animals, venue, aisle concessions, and number of simultaneous acts.  Price is low, animals are used frequently, venue is unimportant, aisle concessions help with revenue, and multiple simultaneous acts is considered important.  Cirque took these characteristics, eliminated some, reduced some, increased others, and added new ones, creating a whole new model.  They positioned their offering as a theater-going experience, allowing for a higher price.  They were able to do this by changing the venue, making it a higher end experience.  To align with the environment change they ditched the aisle concessions, but the revenue loss was more than offset by cost reductions.  Animal use in the circus makes some people uncomfortable anyway, and is often the most expensive part of the traditional circus.  They eliminated this all together.  And rather than put on multiple costly acts at once, which tends to overstimulate the audience and increase costs, they stuck to one.

You may have observed that with the changes Cirque made they actually increased revenue and reduced costs at the same time.  This is a primary goal in Blue Ocean Strategy.  And something we can and should endeavor to do outside of pure business ventures.  We should seek out win-win opportunities in the way we do our work too.

Typical infrastructure / operations management, over time, becomes a liability in most companies.  There is no better way to understand this than to read The Phoenix Project.  Or to ask someone who has worked in the pure Ops space (i.e. no DevOps).  At the conference Gene Kim shared the remarks of a colleague:

Support work is through the roof.  There is little (or no) time for long-term value-add work.  And the quality of life is terrible.  So from a business perspective you have high costs, low throughput, and low employee engagement. 

DevOps is the blue ocean.  By collaborating, and applying engineering practices to infrastructure management we can simultaneously achieve low costs, high throughput, and high employee engagement.  How often do businesses find opportunities like this?

So let’s get moving in the right direction.  Many companies are well on their way.  I was blown away to discover that one area within the Department of Homeland Security is deploying to the cloud with solid DevOps practices.  There is some great starter information in the 2014 State of DevOps Report, including business justifications and practices that result in better business outcomes.

Chaos Monkey 2.0... or 0.5?

Netflix began transitioning to host their services on AWS back in late 2009 / early 2010.  In 2010 they posted a very interesting article about their transition to the Amazon cloud.  A lot of interesting problems cropped up.  What they decided to do is something that much of the IT community is now familiar with - Chaos Monkey.  This service runs in the wild, randomly bringing down entire chunks of infrastructure without remorse.

Why did Netflix decide to do this?  They realized that these problems were going to happen, and happen constantly.  They asked themselves if they would be better off waiting for a failure to happen, then running around like chickens with their heads cut off, or would introducing failure, learning, and improving the architecture be a better approach?  The answer is unequivocally the latter.  The regular introduction of failures, when accompanied with learning and improvement, drastically improves quality.

This was pretty revolutionary in the tech space.  Why haven't we applied this more broadly in our organizations?  For instance, into organizational structures?  Organizational agility is, in large part, about the decentralization of organizational structures.  Check out Reuse Creates Bottlenecks for a great write-up on this topic.  In The New Gold Standard, Joseph Michelli writes about how the Ritz-Carlton has taken decentralization and empowerment to impressive extremes.  Each and every employee is empowered to spend up to $2000 per day per guest to improve their stay.  Do any of you work for very hierarchical organizations where the prevailing management style is command and control?  What tools do we have at our disposal for measuring our organization's ability to push decision making down to the lowest level possible?  Localized decision making is always significantly faster.  This is the heart of true agility.

Netflix's chaos monkey is really a disaster audit.  Why don't we introduce Chaos Monkey as an empowerment audit?  Imagine a common scenario in a typical organization.  You're getting ready to deploy a feature to production. As usual you e-mail all the managers necessary for production deployment approval, when you promptly receive out of office replies and realize that two of them are traveling on a business trip.  What do you do?  If you follow the process you delay releasing valuable customer features.  If you go ahead and deploy you risk the command and control powers that be coming down on you in force.

Here's where the beauty of injecting these type of faults into the system comes into play. What if those managers weren't truly out of the office? What if there was a way to inject this type of chaos into our daily routine?  Could we take all of the managers and senior leaders that are too involved in decisions that should be made at lower levels in the organization, and randomly sever their normal communication points? Email? Gone. Messaging? Gone. Desk phone? Gone. Company-issued cell? Gone.  How long would it take for command and control organizations to screech to a halt?  Would empowered teams hum along?

Leaders that embrace that concept can go on vacations for a week, two weeks, and things keep on humming.  They create a better work-life balance for both themselves and their employees.  If your organization has made a conscious effort to change culture, empower employees, and improve products and services, this could be a way to measure progress.  At the very least leaders approached to make a decision must constantly ask, "Could this decision have been made at a lower level in the organization?"

This concept could even be applied to knowledge management, as a "silo audit" of sorts.  Knowledge sharing within a team is critically important.  Ever heard of the bus number concept?  How many people on your team or in your org would have to get hit by a bus before progress would halt?*  I would guess that for a lot of teams and organizations that number is one.  What happens when that one person severs all communication ties for a period of time?  Better to practice and simulate this before it really happens!  Knowledge sharing, much like disaster recovery, must be automatic, and continuous.

We have a long way to go in this space, but maybe applying some of the same technical practices to our organizations will help us improve.  Or we could just buy a tool for that.....



* The bus number concept is obviously hyperbole (in most cases?), but this isn't that different from job transitions.  What happens when your bus #1 employee takes a different job.  Maybe it's within your company and that's a slight relief.  Or maybe they leave altogether.  Organization continuity planning is the knowledge and talent equivalent of disaster planning for critical IT systems.

Economies of Scale

Let's walk through the typical software development cycle for a new product.  Needs are identified, a team is put together, functionality built, and deployed.  As business needs evolve new features are developed, existing features are changed, and in general systems usually grow in size.  As the size of the system grows testing becomes a bottleneck for most companies.  In fact, companies that let this go on for too long end up with testing cycles that dwarf the development portion of the delivery process.  Why?  Testing needs to cover new features as well as existing features, so testing effort is theoretically equal to all past testing efforts plus the effort required to test whatever is new.  If manual, this obviously gets cumbersome quickly.

Enter test automation. The argument becomes clear to organizations that more of the testing process must be automated.  It's a process that is repeated over and over again.  Why would a company pay a tester to do perform the exact same task over and over again?  The logic is straightforward.  It seems to me that this is the reason that test automation practices are gaining a much larger foothold in companies in the last several years.

The Infrastructure as Code movement is much newer, however.  It seems to me that the argument for test automation (above) is not as applicable in the ops space.  If you consider the same product development cycle described above, but from an ops perspective, it might go something like this.  Needs are identified, teams request new infrastructure to host new solution(s), infrastructure is provisioned, new resources are folded into existing support processes.

"Infrastructure is provisioned".  Once.  Granted, in the case of Amazon, Netflix, Facebook, etc, this wouldn't hold true.  Everything they build is scaled massively.  That's not the case for a lot of companies, especially those in the Information Technology Dark Side.  Infrastructure is provisioned once and that's it.*  There is no economy of scale.

Now there is still an argument, it's just an argument that I think it harder to make and back up with concrete businessy goodness.  Testing?  Manual regression testing takes 1 month.  Our automation suite will take 4 minutes.  Boom.  Infrastructure?  Welllll, the automation is a repeatable, reliable process that will significantly improve our confidence in the provisioning and subsequent change process.  "That's all well and good, but you're telling me we need to write all this... what did you call it... infrastructure codeage?  That seems like a lot of work to stand up these 2 servers.  We'll just hammer them out."

I wholeheartedly believe that the real value in infrastructure as code + automation is the actual repeatable, reliable process that results in consistency.  The extremely beneficial by-product is that the result can scale across as many nodes as you want.  To me this is akin to the TDD argument.  A lot of very smart people pointed out that the real value in TDD is the improved code design.  The fact that the resulting tests form a regression suite is just icing on the cake.  But neither of those real reasons is an easy argument to make to management.

Cost-focused organizations and managers want economies of scale.  They affect the bottom line.  It seems to me that test automation is an easier argument to make.  What am I missing?  What's the business case for infrastructure as code / automation?  How do you frame it up in a way that connects to concrete business value?



* You could certainly make the argument that subsequent changes to the infrastructure should be vetted through an automated test suite process similar to the one I described for application code.  That's fair.  And I'm sure people do it.  That just feels even less tangible to me right now.

Do you have a cost-reduction, or value optimization mentality?

What's the difference?  A monetary cost-reduction mentality focuses on just that - reducing costs.  IT is a cost center.  It's a business enabler.  Period.  This mentality has little awareness or concern for the value function presented by a given decision.  On the other hand, a value optimization mentality focuses on optimizing the value function through careful consideration of trade-offs.  Every decision presents trade-offs.  While a cost-reducer looks at a decision and sees fluctuating costs that must be minimized, a value optimizer sees an outcome spectrum.  The optimal outcome is the one where known trade-offs are considered and optimized.  One trade-off is cost, but it's not the only one.  In my experience, organizations focus on reducing the cost of IT.

Let's take an easy example that seems to come up frequently.  One component of IT is employee computing assets.  That may include things like smartphones, but most important is their computer(s).  This is the primary mechanism through which employees add value to the organization.  Different users tax their computers differently.  One might run their email suite and spreadsheet app on a regular basis, while the other is running one or more development environments, supporting tools, and related automation.  The machine that does the job for the serial emailer is not going to cut it for the software developer.  One size does not fit all.

The cost-reducer either fails to recognize or does not care that one size does not fit all.  He would also advocate infrequent machine upgrades because of cost.  The value-optimizer sees trade-offs abound.  Reducing the cost of computing equipment is critical for obvious reasons.  Developer (and other power users') time is also a critical cost to the business.  Developers are expensive, and their skills always in high demand.  The machine that reduces costs leaves the developer twiddling her thumbs while routine tasks complete.  Even a modest developer salary will quickly justify a better machine that minimizes thumb-twiddling time.  Organizations that don't take this into account are throwing money away and hindering productivity.

A quick sign of an organization focused on cost-reduction rather than value-optimization is one where people and services are centralized.  Centralization yields economies of scale, but (you guessed it!) has trade-offs.  The primary trade-offs associated with centralization are flexibility, and speed.  Mike Cottmeyer wrote this very short, great article called Reuse Creates Bottlenecks, on the topic.  Centralization absolutely has its place, but when teams need to deliver, it creates bottlenecks.  It creates dependencies that are outside the control of teams trying to deliver, and requires them to attempt to manage them.  Outside dependency management can easily cripple a team focused on delivery.  In Donald Reinertsen's book called The Principles of Product Development Flow, he succinctly describes the considerations for centralization versus decentralization.  He makes several more, but here are two primary points:

• "Decentralize control for problems and opportunities that age poorly."
• "Centralize control for problems that are infrequent, large, or that have significant economies of scale."

I haven't had the chance to read this book yet, but I've talked with colleagues about The New Gold Standard by Joseph Michelli.  The focus of the book is the Ritz-Carlton Hotel Company's approach to creating an amazing customer experience.  My understanding is that employees are enabled to spend up to $2,000 to make any single guest satisfied with their stay.  This is empowerment and decentralized decision making authority at it's best.  Solve the problem the instant it's exposed.

Now, let's acknowledge that this is an extreme example.  Different organizations are going to have different value functions.  The Ritz-Carlton's differentiated market position is unparalleled luxury, quality, and customer experience.  That position likely does not mesh well with most cost reduction efforts.  They spend, a lot, for extraordinary quality.  But I digress...

The Ritz-Carlton is doing what we strive to do as agile teams.  As software developers we build quality in by writing tests that assert our expectations and make sure we pass those tests.  Code complete features are tested as soon as possible, and if issues are found, they are fixed immediately.  When a quality issue arises, nip it in the bud as quickly as possible.  In our case, and in the case of the Ritz-Carlton, the optimized approach is achieved through empowered employees with decentralized decision making authority.

Organizations absolutely should consider shared service (i.e. centralized) approaches for infrequent, non-urgent problems.  Often times I think of employee benefit services as ideal candidates - functions outside the business value chain.  But for business functions and customer solutions, organizations must be structured to optimize the delivery of customer value.  Organizations can't just look at the cost reduction because there are trade-offs present and one size does not fit all.  Localized empowerment through decentralized decision making achieves the right fit for the problem at hand, and with agility.

Yes, decentralization creates inefficiencies, but often times those inefficiencies are optimal when compared to the bottlenecks of a centralized approach.  Realizing this, we can make better decisions that result in better outcomes for organizations as a whole.

How to contain those nasty change agents

We've all been there at least once in our lives.  Things are humming along nicely.  You're in a really good routine, operating at status-quo productivity level.  There are plenty of challenges at work,  needing to balance meetings with time dedicated to 401k management.  Then, in the midst of everything already going on, change agent Chad rears his ugly head.  When you find yourself in this unfortunate circumstance, here are some steps you can take to contain the problem before it spreads like wildfire and your environment is forever disrupted.

Limit collaboration.  It is through collaboration that the change agent spreads toxic ideas.  Avoid cultivating an environment in which collaboration occurs freely.  This can be fuel to the change agent's fire.

Feign interest in change movements.  Change agents thrive on their ability to influence to achieve positive outcomes.  They may be able to successfully change perceptions at the grassroots level.  If this occurs, do not worry.  The organization that you've spent years controlling will look to you for direction.  When this happens, dismiss or feign interest in the movement.  Overtly condemning is an option, but this may create a greater ripple effect than desired.  Optimally, you can absorb, but silently dismiss any proposed changes, thereby preemptively smoothing out the rough spots.

Prevent expenditure.  When change comes along, there are likely costs associated.  You can manage this growing change movement by preventing expenditure on new tools, technologies, equipment and training materials.  Without these elements for success, any change is likely to fall short.

Stifle learning.  Learning is another opportunity for ideas to spread.  When maintaining the status quo is critical, at times like these, lead by example.  Promote and embody a culture where learning is difficult, overlooked, and if necessary, reprehensible.  Persistent change agents may attempt to circumvent your expenditure-limiting efforts through frugal or organic learning opportunities.  In dire situations like this, direct prohibition of the effort may be your only option.

As Bob Dylan noted, the times they are a-changin'.  Sometimes we need to take matters into our own hands to ensure this doesn't come true.  After all, our future mediocrity depends on it.

Do doers matter when getting it done?

It seems like large companies believe doers don't matter anymore.  A developer is a developer, a QA is a QA, hire them by the masses!  Any warm body will do!  How did it get to this point?  More importantly, how do we get out?  How do we reinstate development to its proper level of importance; to the level of craftsman?

It's really interesting when you talk to people doing traditional enterprise software development.  Many developed software in smaller companies, startups, or even on their own previously.  If you ask them to rewind and think about how they operated in that environment you find that they embodied many characteristics of an agile developer or agile team.  Collocation, constant communication, collaboration, small chunks, frequent releases, customer focus, getting the job done with great engagement and satisfaction.  Fast forward to today.  Now they're entirely enterprisified.  Lots of documentation to make up for the lack of communication and collaboration, doing an unfulfilling, small slice of the delivery pie, unsure who the customer is and not really understanding how they intend to use the system, and unable to step in when gaps arise because role boundaries are cemented into the culture.  Generally these people say that they were far more effective in their former role.  Why?  They were smaller.  They were focused on the product and the customer, not the process and their role.

Successful small teams start with talented contributors.  Inevitably, greed or business need or both prevail.  Must get more done.  Realistically, if the talented group is humming along, the only way to get more done is to add people to the effort.  That all seems fair to me.  What usually happens at this point though is that a bunch of sub-par people are added to the effort.  That's the worst thing for the team for multiple reasons.

1) Initial ramp-up for new people is costly.  Add talented people, not sub-par people.  Talented individuals will ramp up faster.

2) As Fred Brooks talks about in The Mythical Man Month, increasing people on an effort increases the amount of communication required by the team.  But wait Jason, you just said adding people to the effort was fair?  Absolutely.  Add talented people, not sub-par people.  In my experience, adding talented individuals to a project only marginally increases timeline due to communication, much less so than adding mediocre folk.  This is really the same point as #1, but directed more at the ongoing cost of communication rather than the upfront (ramp up) cost.

3) A diminishing talent pool begets quality policing.  Policing is ugly for obvious reasons.

The diminishing talent pool is the issue I really want to highlight.  As this scenario plays out, the talented folks become "leaders" of sorts where they begin to focus less on implementation and more on leading, managing and overall design.  The thinking here is that having a top tier person design the system and set standards will result in a great and healthy system regardless of who does the development.  Managers in organizations think this works because, as Uncle Bob wrote in The Clean Coder, "they don't see the God-awful code."

The fact is, developers have both immediate and long-lasting impacts that make the role absolutely vital, arguably the most vital.  The code is what users interact with and experience.  Not the system design, underlying components, and certainly not a backend DB.  Users interact with apps written by developers.  Therefore, developers have an immediate impact in the form of product quality.  They introduce, reduce, and prevent defects in the product.  Devs also have a more indirect and long term impact on  users in the form of a product's changeability, testability, automation, and technologies that make up the product.  The code that David the Developer's writes today determines releasability tomorrow.  The code that he writes today determines how hard it is to change tomorrow.  And the code that he writes today determines the customer's product quality experience tomorrow (or even today if you really rock!).  How many sub-par David's can you and your customers afford on your team?

Age-old clarity on the role of architecture

A little over two years ago I was afforded the opportunity to embrace an agile transformation.  I suspect my experience shares parallels with others' agile transformations, particularly when I say that the road was bumpy.  One of the recurring bumps was the role that architecture plays in an agile environment - an environment where delivery teams increasingly focus on the present and embrace the uncertainty of the future.

I've recently been reading a classic software book for the first time.  That book is none other than The Mythical Man Month.  The first few chapters reveal some great insights in the areas I was expecting.  More interestingly though, it lends some fantastic perspective in at least one completely unexpected area - architecture.

For years now I have struggled to understand the role of the architect.  I've worked with many, all with the same job title, but many with different "day to day titles".  Those have ranged from application architect, delivery architect, enterprise architect, security architect, to most recently, user experience architect.  Sometimes while working with the great people in these roles, we clashed.  I think I always knew why in the back of my mind, but the reason has become much more clear recently.  Put simply, decision boundaries were not, and are not clear.

Fred Brooks says that the most important aspect of system design is conceptual integrity.  Conceptual integrity simplified is the relative ease of use of a system on a 'functionality offered' basis.  Increasing the functions offered by a system increases its complexity.  In fact, he says that complexity can increase beyond expectations because "the things one wants to do often requires involuted and unexpected combinations of the basic facilities."  Said more simply, functions must be combined to achieve desired behavior.  The point that Brooks arrives at is that "ease of use, then, dictates unity of design, conceptual integrity."  Ease of use is paramount in system design.  Ease of use is the primary responsibility of architecture.  He elaborates, writing that, "The architect of a system, like the architect of a building, is the user's agent.  It is his job to bring professional and technical knowledge to bear in the unalloyed interest of the user."

Where I have struggled with decision boundaries the lines between the what and the how are blurred.  Brooks clearly highlights the need for distinct separation between architecture - the what, and implementation - the how.  A clear example with a clock is laid out in the book.  The architecture of a clock consists of the clock face and the hands.  Once a person learns this architecture, he can read the time from a church tower or a wristwatch equally easily.  The implementation, however, is invisible to the user.  It's everything behind the scenes that results in the user's collective experience.  But again, the conceptual integrity is paramount.  Consistency and the ability to uniformly interpret the design must be the highest priority.

Several months ago I had a light bulb moment.  It's really an unremarkable thought, but for me it was powerful.  Every architect should be a user experience architect.  Their mandate should be to create the ideal user experience.  I believe that is what Brooks is really saying.

Systems today are very complex.  More and more users want to combine functions from disparate systems to achieve greater efficiency and insights.  Where users desire combined functions, design integrity presents serious challenges.  Architects need to be out in front of these challenges designing the combined experience, and making sure it happens that way.  That's not the role I see most architects playing today.  Their focus is far more on implementation and lower level design.  And I think the reason for that is that architecture has become the "next logical step" for the talented implementer.  But what does the talented implementer excel at and enjoy?  And what does she focus on?  Boom, conflict.

In the world that Brooks describes in The Mythical Man Month architecture often requires fundamentally different sets of skills, and some collaborative overlap with implementation.  Granted, the role of the architect varies wildly based on the product.  For example, a user experience architect for a web-based fitness system might ideally be someone interested in fitness with expertise in web technologies and web design.  On the other hand, a user experience architect for a set of APIs for use by developers is likely far more technical, and would undoubtedly he himself be a seasoned developer.

Architecture and development require a rethink and a lot of clarification.  Blurring the lines between the two creates role conflict.  There are certainly cases where the lines are blurred and collaborative efforts between the roles result in excellent outcomes.  Even so, we must ask ourselves if we have properly separated architecture and implementation responsibilities.  Because if our architects are focusing on implementation, are we focusing enough on the user experience?  We must ensure we have the right focus on the overall user experience.  Conceptual integrity is paramount.