Jack of all clouds
In an episode of Screaming in the Cloud podcast, Corey Quinn, a cloud services expert, mentioned a running prank that he sometimes pulls on Amazon engineers: Quinn inserts a fictional AWS service name into the conversation, with the AWS person not batting an eye. Even Amazon’s employees do not have the complete grasp of all the vast Cloud offerings their own company provides. And what chance do the rest of us have?
This is not 2002
Nowadays, the concept of a “full stack engineer” is largely a myth. The days of a simple HTML site with a Perl CGI script as your “backend” are gone, and so are the days of “email@example.com” email addresses. Software development has become a sprawling field. Not only did the ways of doing things have grown exponentially, but we now have less time to focus on them. A common software development team is no longer a handful of male nerds with bad hygiene, cranking out code with little supervision and little understanding of their work by anyone else. It is now a more diverse, more collaborative, and a more conventional place of business. This naturally comes with an overhead of inefficiency. There is way more complexity, there are more meetings and lines of communication, with fewer and fewer hours in the day for improving ourselves at one specific thing. Your precious “10x developer” is now probably answering Slack messages every 5 minutes.
Let’s list just some of the things we expect a typical software engineer to do day to day:
- Develop new features
- Review code
- Stay on top of the latest “best practices”
- Be an expert on observability - or debugging complex distributed systems in production
- Write and maintain scripts that make local development possible
- Attend in-person and remote meetings
- Participate in a stream of work chat messages
- Compose and maintain code documentation, API documentation, WIKIs
- Create presentations for knowledge transfer
- Mentor junior developers
- Be proficient with infrastructure-as-code and frameworks such as CloudFormation or Terraform
- Understand which architecture patterns match a certain scale or a use case
- Understand database engines and their applications
- And this: have in-depth knowledge of which Cloud solutions are appropriate, cheap, and secure
This is an insane list of expectations, where one single bullet point alone can easily be someone’s full-time job. Not only is it impossible to be fluent in all of this, this leads to critical developer blind spots. We are becoming satisfactory at everything and good at nothing.
Ken Kantzer, of the startup auditing firm PKC, mentioned in the Changelog podcast a common case of software developers not understanding that JWT tokens are not encrypted by default. Developers were shown plain-text version of their own JWT tokens, to their amazement - with sensitive information right there for anyone to read.
You expect the same engineers to also have the bandwidth to understand the intricacies of cost dynamics in a multi-region AWS setup? Which brings us to…
It will cost you
Your cloud bill is directly proportional to the size of your team. In part to Conway’s Law, and in part because many workplaces give teams autonomy, which manifests as those teams piling on their favorite Cloud toys without any regard for performance, cost, security, or overall complexity.
As your engineers add whatever they desire to your already extensive cloud infrastructure, who really gets down and dirty with a cloud cost calculator? Who analyzes the redundancies, or cheaper and simpler ways? Who gets to validate that rebuilding your entire application in Lambda is not just someone’s resume-driven development effort but a pragmatic decision, not resulting in nasty performance and cost surprises? Who tracks how much data is stored where, accumulating in cost every month? Can it be moved? Deleted? Compressed and archived? Give me a second - I will get right on that after I finish my standup, two Zoom meetings, prep to conduct an interview, attend to technical debt, and spin up another goddamned microservice.
With the Cloud, everything seems reasonably priced and simple - at first, until someone notices the bill. In these lean times, when tech companies are forced to be only reasonably frugal, everything seems fine - and then it’s a code red emergency with all hands on deck. “Our cloud bill is too high, fix it!”
A shallow, ocean-sized puddle of knowledge
Non-stop learning is part of this job, but the ever-growing amount of things to know is relentless and even paralyzing. Basic knowledge of things like database indexes and sending plain HTML over HTTP (!) is passé.
In this environment, expecting software engineers to also be masters of Clouds is, as we say, an anti-goal.