I have the pleasure of working with a lot of very smart people who challenge me and help me grow every day. Recently we have been talking a lot about moving to a model where we have an API for our CTO organization rather than so many carefully hand crafted integrations. If a tool or solution we consume has five different patterns for integration we will happily use them all. It creates fragility and complicates our operators lives, not to mention it makes an upgrade of said solution arduous. The concept of interfacing rather than integrating has been around for more than thirty years and I dare say we are still trying to get good at it across all of IT. I posit that we need a Systems Programming Interface approach in the spirit of W. Richard Stevens and Stephan A. Rago’s Advanced Programming in the Unix Environment which was ubiquitous in the libraries of all my mentors early in my career. That book and the related series on Network Programming set out how to reliably plumb everything you needed to make Unix systems and their software hum efficiently. I differentiate between a System and Application Programming Interface because I believe one supports the other, the system focused on the environment we run high quality applications in.
There are already many approaches out there to help us with this as it is the basis of Infrastructure and Platform as a Service capabilities. We use BOSH, Puppet, Terraform, vRealize, Concourse, and ServiceNow to name a few to build high quality, repeatable, and auditable operating environments. We need to extend that approach further, and I have seen some great examples of how companies are tackling the problem. I used to think it was clever to say, “The bad news is everyone has this problem, the good news is everyone has this problem”. There is nothing new under the sun really. And while I think we have many solved problems we can draw upon I have not yet seen much by way of collating all the goodness in this space into a body of knowledge.
I have seen some approaches that unfortunately build fragility into the solution. Hand coding integrations between key components via a home grown master of masters, any time a component’s API changes you have to make updates to the master. This certainly solves many problems, but it creates this one new big system that starts to resemble the Borg. Soon, in order to use the new system, you must assimilate: it starts with needing to use the system’s artifact repository, then the source code repository, then the build engine, and it goes on from there. Eventually you end up with a developer and operator community that is chafing at the choices they have been forced into, or a system that is not able to keep up with the workload thrown at it, causing disruptions for teams across the enterprise. Neither is really all that fun. Ooops, we have built a monolith for systems integration, oh the irony.
So the question becomes how do we apply a microservices type approach to the problem? There are some really good resources on microservice architectures, the one that we have been using is Sam Newman’s Building Microservices: Designing Fine-Grained Systems. It contains a lot of practical advice on how to approach the disaggregation of a monolith system, or how to start the design of a new system and build it to scale. To apply this advice and approach we need some new roles or skills in IT, architects who think in terms of interfaces rather than integrations, people who can take service definitions as a concept from catalog to code. Service delivery is going to have to move from a click in a catalog to a YAML file (for instance) defining what is required and being built in real time.
We will also need to work with our vendors and partners differently. A frequent frustration is just how willing a Software as a Service provider might be to alter or customize their offering for a customer. It all seems well and good until the first time you have to do all the integration testing required for a SaaS capability upgrade. Changing a SaaS data model because we want our thing to be called a widget rather than a blodget might seem like a good idea until you realize you have just caused someone else to fork their monolith and now you are dependent on the fragility you have just introduced. We are going to need to consume services and capabilities in as vanilla a fashion as possible. If we want to do some customization let’s do it outside the system through a microservice taking advantage of the provided interfaces. This means our traditional IT organization is going to need some developers! And an approach to service discovery. And code reviews. And an enablement mindset. I could go on but you get the point. Now I have got to get back to my Golang Udemy class, it is time for change as Chad Sakac likes to say.