Race to the Bottom with Cloud Certifications

A colleague of mine wrote in an email today, “I am in a class, week long, with almost 20 folks to get AWS Dev cert, Unisys is taking the week not billing so they can get these certs on their profile for proposals.”

Another colleague, who owns an IT staffing firm, wrote last week, “…we have a large pool of AWS Certified and DevOps resources and we have invested significantly in training our teams and get them certified.”

This is how you turn junior people into senior people overnight — at least in the minds of government and large company procurement officers. Right now there is a race for everyone to get certified, in AWS or Azure. Pretty soon, if you don’t have the cert, you won’t be able to work. It will be like a tax: everyone will have it, and so it will not differentiate anyone, but you have to get it. I personally have a basic AWS Cloud Practitioner certification, but I plan to get the Solution Architect one, because it is becoming mandatory.

Certification is how you turn junior people into senior people overnight.

The AWS certs are pretty difficult: the material is not difficult — definitely not rocket science — but there is a-lot of material, and if the Cloud Practitioner exam is any indicator, the exam will want their answers. (In the exam I took, I counted nine questions that could have been answered differently and still been arguably correct, but not the answer that Amazon was looking for.) The exams are picky, and that means that even if you know all of the subjects that they test for, you will fail unless you read their material and learn their answers.

The AWS exams are picky, and that means that even if you know all of the subjects that they test for, you will fail unless you read their material and learn their answers.

It is good material though. A-lot of what is in the AWS Solutions Architect certification curriculum is what was covered in my book on secure and reliable architecture: things like designing for failure, designing for increments, defense in depth, caching and replication, designing for manageability, and so on. The patterns are timeless. The AWS program, however, is entirely in the context of AWS’s services, and so it is missing much of the conceptual foundation: you just use their IAM service and then you don’t have to think about or understand access control models. You just secure your VPC with two-factor auth, create a public subnet for your apps, secure your VMs with AWS Security Groups (a strange term that is not used by the industry at large), and use AWS Cognito for authenticating your users and you don’t really need to understand defense in depth, or do you?

I have found that it is not possible to give programmers “safe” tools: if they do not understand their tools, they will misuse them. I am very security conscious, but about a year ago my personal AWS account got hacked: I had a private key in a source code file that I had deleted, but I had since made the git repo public, and my git repo history still contained the deleted key, and someone scanned for that and found it. In the span of twelve hours overnight, a hacker had spun up several hundred large machines in four regions, doing what I do not know, and had run up $12,000 in charges — charges that took two months for Amazon to rescind. And I am really, really careful, and knowledgeable about security, and so if it can happen to me, in AWS, it can happen to anyone. AWS has tried to make things appliance-like, but it is not: there is brittle JSON everywhere, there are keys, there are configurations galore, and there is your application code, which can do anything you tell it to do. You have to know what you are doing. You have to be vigilant, and you have to understand things.

AWS has tried to make things appliance-like, but it is not: there is brittle JSON everywhere, there are keys, there are configurations galore, and there is your application code, which can do anything you tell it to do.

The certifications also do not cover software architecture. That is okay if you are building simple software services that respond to events and just make entries in a NoSQL database, but while a-lot of e-commerce sites are like that, many are a-lot more complex. Not everything is suitable for the event-oriented design that Amazon has been evangelizing. In fact, among the open source projects that I myself have created in recent years (here, here, here, here), none fit that pattern.

Today, a true software architect should probably know about most of these broad domains:

1. Cloud strategy
2. Agile and DevOps
3. Security
4. Microservices
5. Development (coding)
6. Increasingly: DevOps for machine learning

and they should probably be familiar with at least one of each of the following, a few examples provided for each:

1. Cloud platforms (AWS, Azure, GCP, others)
2. Languages (Java, Ruby, Python, Scala, Javascript, Go, Kotlin, Swift, some C, others)
3. SQL databases (Oracle, DB2, MySQL, MS-SQL, Postgres)
4. NoSQL databases (Dynamo, MongoDB, Cassandra, Elastic Search)
5. In-memory caches (Redis, memcached)
6. Containers and container platforms (Kubernetes, OpenShift, ECS, Docker using Alpine on Atomic or RHEL, and Centos)
7. Messaging (Azure Event Hubs, Kafka)
8. CI/CD orchestration (Jenkins, AWS CodePipeline, VSTS, Gitlab CI/CD)
9. Software provisioning (Chef, Puppet, Ansible)
10. Test automation (BDD: Cucumber, Robot Framework, JBehave, SpecFlow)
11. Large scale batch and stream processing (Hadoop, Spark, Storm, HDInsights, Java stream APIs)
12. Increasingly: Machine learning platforms — TensorFlow, Theano

The AWS technology stack covers pretty much all of the above items, but they have their own version of everything. For example, instead of Hadoop, you use AWS EMR. In Azure, you use HDInsights. Is this a good thing, being tied to these cloud platforms? There is little doubt they are here to stay for awhile, so that is not the issue. What is the issue? The issue is whose interests they have in mind.

I have some personal experience with Azure HDInsights. It takes about 30 minutes to start, and includes five VMs. It is very expensive to run. More importantly, if you want to enable your teams to test continually and in isolation, you will need to create multiple HDInsights clusters. You are talking about tens of thousands of dollars as month — maybe more! Yet, HDInsights is nothing but Hadoop, Spark, Storm, HBase, and a few other things — all free Apache software! — packaged with an Azure Resource Manager template so that it can all be deployed together. If you were to use the generic components, all downloadable from Apache, you could run it all locally, on on-premises blades or even on your laptops, saving a fortune.

The cloud providers want you to do everything in the cloud. They are even pushing for you to give up Jenkins and run their own cloud version of that, because that means more money for them. Their refrain is, “Don’t operate your own data center.” But sometimes it makes sense to have an on-premises footprint. For production stuff, it is probably sensible to use the cloud, because it scales and is cheaper at scale and they have built all the security controls; but for development and testing, it often makes more sense to do that on-premises: you don’t need high availability for testing — you can just have a few racks in any of your buildings. Set up your own Kubernetes cluster. Done. Cheap. Fixed cost.

The cloud providers want you to do everything in the cloud, but they do not have your interests at heart.

So remember that AWS, Azure, Google, and all of those do not have our interests in mind: they have theirs in mind. Their certifications are no exception.

Their architect certifications do not train you to be an architect. To be a competent architect, you need a decade or more of experience with a range of technology stacks: only then can you have perspective. You need to have done a-lot of reading, and a-lot of coding, and a-lot of deploying, and used multiple languages and approaches. If you have used only one suite of tools, and your knowledge was all force-fed in a short period of time, without a wide range of experience to back it up, then you are not an architect, no matter how high your certification is with those tools: you are a tool guru. You are, in effect, an independent technical support rep for that tool vendor — not an architect.

To be a competent architect, you need a decade or more of experience with a range of technology stacks: only then can you have perspective.

The fact that a company has gotten all of their staff Solution Architect certified in AWS should not be confused with all of their staff being architects. It is not the same thing.

Does an architect even need to be certified to work in AWS? I have been working in AWS since 2012, and I am an architect. I wrote Sun Microsystems’ original Enterprise Java book, and a book on secure and reliable architecture, and was a CTO with about a hundred programmers and architects, and have personally built things ranging from compilers to microservices. The AWS documentation is superb. I have not found that I need training to use it. I have used EC2, S3, Lambda, IAM, ECS, Route 53, Transcription, and other AWS services, have created VPCs and subnets and configured security groups. I have written to their REST APIs, and deployed to AWS using Vagrant and other orchestration frameworks. You don’t need a certification to know what you are doing with AWS. Not necessary.

Given a choice of someone who is a real architect but not AWS certified, versus someone who is less experienced but an AWS Solution Architect, I would pick the real architect any day, hands down, even if all the work will be done in AWS. The real value is the experience and intelligence of the person: the certification is of marginal value to me if the person is smart, and real architects are smart.

The recruiting firms are in a race to the bottom, and want to turn us all into checkbox commodities that they can screen for quickly. They don’t care what you really know or can do.

My concern is that recruiting firms do not understand these things. They go for the certifications. They will recruit the wrong people. They are in a race to the bottom, and taking us with them. They are trying to turn us into commodities that can be evaluated quickly by checkboxes: Jenkins? Check. Scrum Certified? Check. Python? Check. EC2? Check. AWS Certified — no? Sorry, move on — no matter what else you have done in your career, even if you friggin’ created a whole Web based product for a startup, or you are Azure certified instead of AWS (Azure has all the same kinds of things as AWS), or did some truly singular things that prove incredible capability. And commodities tend to be selected based on the lowest cost — not the best quality, effectively filtering out the best and the brightest.

We need to tell others about this, or our industry will drive out those who are really smart: they will not want to be treated like a commodity, and will leave the profession. A DevOps colleague of mine is even looking into buying a gas station.