Part 2: Asking Other Teams to Do Something Really Hard: Platform Software

Background

In Part 1: Asking Other Teams to Do Something Really Hard, we explored two simple examples of when asking a person or team something really simple turns into something really inefficient. In that first article, we used simple examples, like new users asking simple questions over and over on Reddit or a mailing list. The effect is quite negative on the skilled users who answer the questions, don’t have a network effects, like we’ll describe in this article. In this article, we’ll explore more complex examples with more nuance which also have powerful positive and negative consequences, especially in the context of large organizations or software platforms. Sometimes it makes sense to do hard work that other teams ask you, sometimes it doesn’t. In this article we’ll try to provide examples of both.

Platform Example

Say that you work in a large software company that builds and delivers a large piece of platform software used by thousands of customers, partners, and even other business units within your company. Now, also imagine that your company has decided to build and deliver a new software product on top of this platform. The new product team needs a specific library or tool, and they’re hoping to put it in the platform as opposed to support it themselves. These reliance on another piece of software is called a dependency and it’s a major point of friction for any team building on top of a platform, whether a customer, partner, or another business unit. To make this example easier to understand, we’re going to name each team:

Product Team P builds the platform

Product Team Y builds yet another software product on top of the platform

It would be very natural for Team Y to ask Team P to “just add” the dependency to the platform, right? Then, Team Y can just consume it, instead of maintaining it themselves. Here’s where the problem starts. Team P serves thousands of customers, and partners which is a massive amount of work. To do all of this work, Team P is composed of thousands of staff engineers, distinguished engineers, quality engineers, documentation people, support people, product managers, product marketers, sales people who can explain it to customers, and partner managers who can explain it to partners.

Here is some important information about the dependency that Team Y is asking Team P to add to the platform:

• Can do all kinds of fancy character and text processing
• Is an open source library written and supported by a single developer upstream
• This developer does not work for your company
• Team Y only uses less than 3% of the functionality in this library

Here is some information about what would be necessary to add this dependency to the platform, all of which are quite expensive:

• It would require require much greater than 3% test coverage
• Documentation would need to be improved
• Maintenance work for the life cycle of the platform would need resourced
• Security analysis coverage and patching of CVEs
• Performance might need to be tested with the set of hardware that your platform supports
• Your company would likely need to hire the upstream developer, or put a core contributor on the upstream team
• Partners and customers may adopt this library pushing the requirements even further

Here’s some extra meta-work that you may not have even thought about. Whenever a request like this is made, somebody has to:

• Try to understand what the library does
• Try to understand who maintains it and how healthy the community is
• Gather requirements to understand how long the package needs supported for.
• That’s a ton of work. The team from Product B probably can’t even answer all of the questions about requirements for security and life cycle because their product is new and may not even succeed in the marketplace.

It feels really easy for Team Y to ask Team P to do this work, but it’s actually quite difficult. There’s a lot of initial work, just to gather requirements as well as maintenance, support, and perhaps even marketing and sales work down the road. This pattern will feel familiar to anyone who works at a platform software company, as well as any large bank, manufacturer, telecom, or transportation company – really any global Fortune 1000 company. If you’re an Enterprise Architect, Systems Administrator, Site Reliability Engineer, or even a developer working on a platform team, you’ve seen requests like this.

Please support my thing, they’ll say. It’ll be easy, they’ll say! 🤣

 

Drawings and Math

Since we’re talking about software here, let me reinforce this using some math. If you’re familiar with Big O notation, it represents the amount of work done by an algorithm, but I’m going to abuse it a bit to represent the amount of work done by different teams. And, as one of my beloved Computer Science teachers used to say, “if it’s between a better algorithm or more resources (CPU, Memory, etc), I’ll take a better algorithm every time!” I believe this applies to dependencies and platforms as well! 😀

In our above example, Team Y is asking Team P to do some work of adding a dependency. These two teams have two main options. Team Y can do the work of packaging and maintaining the library they need for the life cycle of their specific product, which is likely shorter than the platform requirements. Team Y can also build test coverage for the 3% of the dependency that they use. The other option is Team P accepts the library into the platform with all of the associated requirements. Let’s try to put some made-up numbers on that work, which are useful enough to do some logic:

• Team Y: O(1): just use the library like any other package
• Team P: O(n^2): with all of the QE, Docs, and Engineering work necessary to add it to the platform

Here’s a drawing to represent the request. Note, this could be a rational and reasonable request, but we’ll get to that in a minute:

 

Now, let’s do a thought experiment and scale this out over several products, customers and partners. If each of the products, customers and partners have completely different dependencies, and Team P does all of the work, it will look something like this. This won’t scale, as Team P will run out of resources very quickly. This is bad for partners, customers and the company building the platform. Again, these numbers are arbitrary, and made-up, with just enough accuracy to demonstrate the concepts:

 

But, there are times when the platform should take-on the dependency, when it’s actually more efficient. Imagine another scenario where Team Y, Partner A and Customer B all share the same dependency. This is the magic of platforms. When a they can satisfy the needs of multiple users, there are economies of scale to be had!

To make things even more fun, I gave Team C an O(n^3) because perhaps the dependency they’re trying to use is something for which they have no expertise which would make it very difficult to learn and maintain. Be careful tough, this is what almost every application feels like when they ask, like they have no expertise, and it would just be easier for the platform team to help them (standard Part 1: Asking Other Teams To Do Something Really Hard stuff).

The following drawing shows that the Team P taking on the this shared dependency is orders of magnitude less work. This math is why platforms work, it’s why products in general work. Products and platforms give you approximately what you want. Though Team Y, Partner A and Customer A may have to compromise on the version of the dependency that is supported by the platform, it’s worth it for the cost savings:

Conclusion

As stated in the first article, respect is the first step. Respect Team Y and Team P have to respect each other and have an honest conversation about who does the work. But also, respect product marketing, quality engineering, technical writers, program/project managers, support engineers, product managers, etc. When it comes to a platform, each of these disciplines is a business partner. The impact on each of these teams should be considered and respected!

The platform example used in this article demonstrated that the initial cost of adding a dependency to the platform is quite expensive. However, as more and more customers, partners, and internal teams express a need for the same dependency, there comes a point where the math flips. At scale, it’s much cheaper to have the platform team do the work instead of each individual team duplicating the work for their specific needs. This is true with internal and external users. If you’re part of a platform think about paying customers, partners who build your ecosystem, as well as the needs of your company as a whole, which will likely need to build a portfolio of products on top of your platform.

Of course, there are caveats. The platform example above is extremely simplified. Different customers, partners and engineering teams may need different versions of a particular dependency, with different life cycle requirements, and different security requirements, and different performance requirements, etc. The never-ending stream of slightly different requirements can be quite challenging for platform teams. Sometimes, it’s necessary to remind users that to be more efficient, we all need to compromise!