Self-Service Platform Development Made Easy with Pulumi
Over the last five years I’ve worked on and consulted with nearly a dozen internal platform teams. In almost every case, the goal was to provide self-service infrastructure capabilities to development teams within the organization.
This is a desirable pattern because it improves the efficiency of development teams by reducing context switching and the need to understand infrastructure in a deep way. It’s also useful for governance and ensuring that infrastructure is properly architected for security across an organization.
Puppet recently released the State of DevOps Report 2020 and, not surprisingly, there is a whole section dedicated to “Scaling DevOps practices with internal platforms.”
The report claims that “High DevOps evolution correlates strongly with high use of internal platforms. Highly evolved firms are six times as likely to report high use of internal platforms as firms at a low level of DevOps evolution.”
The report also notes that common challenges to building an internal platform are “lack of time”, “lack of standardization” and “lack of technical skills within the team”.
This is right in line with what I’ve seen. It’s hard for platform teams to properly build and manage an internal platform.
In an attempt to surmount these challenges, it’s common to see teams cobbling together bash scripts with ticketing systems like ServiceNow or Jira Service Desk. Platform teams are usually comprised of former system administrators so it makes sense that they would use tools they are familiar with.
Unfortunately this is not how developers want to interact with infrastructure. Developers want to interact with APIs and code. Another point in the report from Puppet: “When we compare low-evolution companies to those at a high level of DevOps evolution, we see expanded use of raw APIs, GitOps, ChatOps and command line interfaces (CLIs)”.
That said, I’ve also worked with some teams that have software engineering capabilities and the problem isn’t what they are capable of building, it’s a lack of focus on their customers and product.
Because of this it’s not uncommon to see these teams spend months if not years building without working with a specific “customer”. So they end up building the wrong product or they spend so much time building that the underlying patterns and assumptions they had when they started are no longer relevant and the product ends up getting scrapped.
Again, a relevant point in the report: “Focus on developer experience and flow. We can’t stress enough that empathy is a critical skill set. Empathy means understanding someone’s position, and it’s impossible to build a good product without having empathy for your user.”
So how can a recent offering from Pulumi make building an internal platform easier?
Well, that offering is called “Automation API” and basically it’s a way to programmatically create, run and manage Pulumi programs.
It doesn’t solve all the aforementioned challenges but it does provide building blocks that make the actual development of an internal self-service cloud platform much easier.
And to prove it, I built a simple prototype of what this would look like called “cloud platyform”.
So what does it do?
Cloud platyform provides an easy way for a platform team to create self-service.. services.. for developers or other users to consume. Services are authored as Pulumi programs in a Github repo, so they are really simple to create and manage.
And to provision those defined services, users can click a few buttons and get a service that includes all of the best practices for that organization while hiding any unnecessary complexity. See the demo videos below.
Service creation:
Service instance creation:
It’s basically like your own AWS, GCP, Azure, etc.
It’s important to mention that beyond interacting with the cloud platyform console, users could also interact with the API that backs it. So it’s easy to use for developers and creating automation.
To take that a step further, you could create a Pulumi provider for this API so developers have a really slick way to consume services put together from a platform team, next to other services that they may be using now or in the future.
Of course, there are a whole host of features that could be added to a platform like this. Here are a few ideas:
- Ability to provision cloud credentials in a self-service way that can be used to connect and provision services.
- A way to group a number of different service instances into an “application” so there can be a higher level view and increased visibility for developers.
- Role-based access control for everything. Role to create and manage services, access to different instances, etc.
That said, I had fun putting together this prototype. It took minimal effort considering what it can do. I can only imagine what a properly staffed product team could do with this.
Pulumi has done a great job getting the ball rolling and I’d love to see a solution like this from them or from the community as an open-source project.
Here’s a link to a blog post that introduces the Automation API from Pulumi: https://www.pulumi.com/blog/automation-api/.
I’ve open-sourced the code and can be found here: https://github.com/danfhernandez/cloud-platyform
(The project is a quick prototype and was only deployed locally. If there’s enough interest I’d consider working on it further.)
