Jack of all clouds
In an episode of Screaming in the Cloud podcast, Corey Quinn, a cloud services expert, mentioned a running prank that he sometimes pulls on Amazon engineers: Quinn inserts a fictional AWS service name into the conversation, with the AWS person not batting an eye. Even Amazon’s employees do not have the complete grasp of all the vast Cloud offerings their own company provides. And what chance do the rest of us have?
This is not 2002
What is a “full-stack engineer”, anyway? What do they do? The days when a site was just some HTML with a Perl CGI script as your “backend” are gone, and so are the “webmaster@domain.com” email addresses. Software development has become a sprawling field. Not only is there a head-spinning number of ways to solve the same problem, but we now have less time to focus on all them. A common software development team is no longer a handful of male nerds with bad hygiene, cranking out code with little supervision and little understanding of their work by anyone else. It is now a more diverse, more collaborative, and a more conventional place of business. This naturally comes with an overhead of inefficiency. There is way more complexity, there are more meetings and lines of communication, with fewer and fewer hours in the day for improving ourselves at one specific thing. Your precious “10x developer” is now probably answering Slack messages every 5 minutes.
Keeping up with all the new developments and “hot” new frameworks in the Javascript ecosystem alone can be draining, and dev/cloud ops is no different. Why, then, do we expect software engineers to be Cloud experts?
Let’s list just some of the things we expect a typical software engineer to do day to day:
- Develop new features
- Review code
- Stay on top of the latest “best practices”
- Be an expert on observability - or debugging complex distributed systems in production
- Write and maintain scripts that make local development possible
- Attend in-person and remote meetings
- Participate in a stream of work chat messages
- Compose and maintain code documentation, API documentation, WIKIs
- Onboarding
- Create presentations for knowledge transfer
- Mentor junior developers
- Be proficient with infrastructure-as-code and frameworks such as CloudFormation or Terraform
- Understand which architecture patterns match a certain scale or a use case
- Understand database engines and their applications
- And this: have in-depth knowledge of which Cloud solutions are appropriate, cheap, and secure
This is an insane list of expectations, where one single bullet point alone can easily be someone’s full-time job. Not only is it impossible to be fluent in all of this, this leads to critical developer blind spots. We are becoming satisfactory at everything and good at nothing.
Ken Kantzer, of the startup auditing firm PKC, mentioned in the Changelog podcast a common case of software developers not understanding that JWT tokens are not encrypted by default. Developers were shown plain-text version of their own JWT tokens, to their amazement - with sensitive information right there for anyone to read.
You expect the same engineers to also have the bandwidth to understand the intricacies of cost dynamics in a multi-region AWS setup? Which brings us to…
“Knows just enough to be dangerous”
A developer who knows zero of something is probably better for your infrastructure security than a developer who knows half (but thinks they know it all).
Software vendors go out of their way to make their products user-friendly, working out-of-the-box in order to push for fast ramp up and adoption. That often comes at the price of literally no security. “Look, ma! I installed Mongo and it just works. I have gone web scale, ma!”
What ma knew and what the developer didn’t was that Mongo did not set a password by default, so the giddy engineer who mirrored their local setup when going to production just exposed the production database to the world as a free-for-all. That ended up just about how you would expect.
MongoDB is not the only culprit, either. There have been spectacular security breaches stemming from Amazon AWS allowing wide-open S3 buckets with public access ON by default.
Once the technology in question gets more complicated, so are the ways in which it can be misconfigured. Making your engineers administer Kubernetes is probably not the greatest of ideas, as the “working knowledge” is just enough to get it working but not enough to be aware of all the security caveats and pitfalls (and not just security, really).
Still, even if you are lucky enough to not have been ransomwared into oblivion, there will be a price…
It will cost you
Your cloud bill is directly proportional to the size of your team. In part to Conway’s Law, and in part because many workplaces give teams autonomy, which manifests as those teams piling on their favorite Cloud toys without any regard for performance, cost, security, or overall complexity.
As your engineers add whatever they desire to your already extensive cloud infrastructure, who really gets down and dirty with a cloud cost calculator? Who analyzes the redundancies, or cheaper and simpler ways? Who gets to validate that rebuilding your entire application in Lambda is not just someone’s resume-driven development effort but a pragmatic decision, not resulting in nasty performance and cost surprises? Who tracks how much data is stored where, accumulating in cost every month? Can it be moved? Deleted? Compressed and archived? Give me a second - I will get right on that after I finish my standup, two Zoom meetings, prep to conduct an interview, attend to technical debt, and spin up another goddamned microservice.
With the Cloud, everything seems reasonably priced and simple - at first, until someone notices the bill. In these lean times, when tech companies are forced to be only reasonably frugal, everything seems fine - and then it’s a code red emergency with all hands on deck. “Our cloud bill is too high, fix it!”
A shallow, ocean-sized puddle of knowledge
Non-stop learning is part of this job, but the ever-growing amount of things to know is relentless and even paralyzing. Basic knowledge of things like database indexes and server-side rendering is passé.
In this environment, expecting software engineers to also be masters of Clouds is, as we say, an anti-goal.