Designing a Successful Microservices Engineering Culture
Tags: microservices, leadership
Published: 2018-07-17 19:35:44

---
title: Designing a Successful Microservices Engineering Culture
published: true
description: Learn how to ensure your team culture meshes well with a microservices architecture
tags: microservices, leadership
---
<span style="font-weight: 400;">Back when I was working with Java developers, I remember there being a source of tension within the camp around who got to work on the newest and meatiest features. Our engineering leadership had decided that we would exclusively use Java to build all new microservices.</span>

<span style="font-weight: 400;">There were great reasons for this decision, but — as I will explain later in this chapter — such a restrictive decision come with some repercussions. Note that to the extent possible, communicating the</span> _<span style="font-weight: 400;">why</span> _<span style="font-weight: 400;">of a technical decision to the team can go a long way towards creating a culture where people are kept in the loop.</span>

<span style="font-weight: 400;">When it comes to organizing and managing a team around microservices, it’s always challenging to balance the mood, morale and overall culture of your team. In most cases, the leadership needs to balance the risk of team members using new technology against the needs of the client and the business itself.</span>

<span style="font-weight: 400;">This dilemma, and many others like it, has led CTOs to ask themselves questions such as: How much freedom should I give my team when it comes to adopting new technologies? And perhaps even more importantly, how can I manage the overarching culture within my camp?</span>

<span style="font-weight: 400;">As in previous chapters, we’ve turned to the experts for some guidance.</span>

## <span style="font-weight: 400;">Stability vs flexibility</span>

<span style="font-weight: 400;">Let’s say you hire a new junior developer. They may be excited about some brand new fresh off the press</span> [<span style="font-weight: 400;">JavaScript framework</span>](https://en.wikipedia.org/wiki/List_of_JavaScript_libraries)<span style="font-weight: 400;">. Almost</span> _<span style="font-weight: 400;">too</span>_ <span style="font-weight: 400;">new.</span>

<span style="font-weight: 400;">That framework, while sporting some technical breakthroughs, may not have proven itself in production environments, and it most probably doesn’t have great support available. CTOs have the hard choice between okaying that move for the good of morale and excitement within the camp — or declining it to safeguard the needs of the company, to protect the company’s bottom line, and to keep the project stable as the deadline approaches.</span>

<span style="font-weight: 400;">The right answer depends on a lot of different factors, which also means there</span> _<span style="font-weight: 400;">is</span>_ <span style="font-weight: 400;">no right answer.</span>

<span style="font-weight: 400;">“We give our team and ourselves 100% freedom in considering technology choices. We eventually identified two or three technologies not to use in the end, primarily due to not wanting to complicate our deployment story,” said Benjamin Curtis, Co-founder of Honey Badger during</span> [<span style="font-weight: 400;">our interview</span>](https://buttercms.com/blog/microservices-for-startups-an-interview-with-ben-curtis-of-honeybadger)<span style="font-weight: 400;">.</span>

<span style="font-weight: 400;">“In other words, we considered introducing new languages and new approaches into our tech stack when creating our microservices, and we actually did deploy a production microservice on a different stack at one point. [While we do generally] stick with technologies that we know in order to simply our ops stack, we periodically revisit that decision to see if potential performance or reliability benefits would be gained by adopting a new technology, but so far we haven't made a change,” Curtis continued.</span>

<span style="font-weight: 400;">When I</span> [<span style="font-weight: 400;">spoke with Stephen Blum, CTO at PubNub</span>](https://buttercms.com/blog/microservices-for-startups-an-interview-with-stephen-blum-of-pubnub)<span style="font-weight: 400;">, he too expressed a similar view in favor of welcoming pretty much any technology that cuts the mustard:</span>

<span style="font-weight: 400;">“We're totally open with it. We want to continue to push forward with new open source technologies that are available and we only have a couple constraints with the team that are very fair: must run in container environment and it has to be cost-effective,” Blum said.</span>

<span style="font-weight: 400;">On the other hand, Darby Frey, Co-founder of Lead Honestly,</span> [<span style="font-weight: 400;">takes a more restrictive approach</span>](https://buttercms.com/blog/microservices-for-startups-an-interview-with-darby-frey-of-gamut) <span style="font-weight: 400;">to technology selection.</span>

<span style="font-weight: 400;">“At my last company we had a lot of services and a fairly small team, and one of the main things that made it work, especially for the team size that we had, was that every app was the same. Every backend service was a Ruby app,” he explained.</span>

<span style="font-weight: 400;">Frey told me how this helped simplify the life of his team, as every service has, “the same testing framework, the same database backend, the same background job processing tool, et cetera. Everything was the same.”</span>

<span style="font-weight: 400;">“That meant that when an engineer would jump around between apps, they're weren’t having to learn a new pattern or learn a different language each time. So we're very aware and very strict about keeping that commonality,” Frey said.</span>

<span style="font-weight: 400;">At the same time, Frey is sympathetic to the concept of developers wanting to introduce a new language, admitting that he “loves the idea of trying new things”. However, he feels that the cons outweigh the pros.</span>

<span style="font-weight: 400;">“Having a polyglot architecture can increase the development and maintenance costs. If it's just all the same, you can focus on business value and business features and not have to be super siloed in how your services operate. I don't think everybody loves that decision, but at the end of the day when they have to fix something on a weekend or in the middle of the night, they appreciate it,” said Frey.</span>

## <span style="font-weight: 400;">Give every team member a chance to thrive</span>

<span style="font-weight: 400;">Let’s revisit my story from the start of this chapter. If you can recall, engineering leaders decided that Java was the best technology to use when building microservices. Java is performant and many of the senior people on the team we well versed with it so that’s why we went with Java.  However, not everyone on the team had experience with Java.</span>

<span style="font-weight: 400;">The problem was, our team split into two camps: the Java guys, and the JavaScript guys. As time went by, new and exciting projects came up, and we’d always reach for Java to get the job done. Before long, there was some annoyance within the JavaScript camp that crept up.</span>

<span style="font-weight: 400;">“Why do the Java guys always get to work on the exciting new projects, while we’re left to do the mundane front-end tasks like implementing third party analytics tools? We want a big, exciting project to work on, too!”</span>

<span style="font-weight: 400;">Like most rifts, it started out small but grew worse over time.</span>

<span style="font-weight: 400;">The lesson I took from that experience was to take your team’s expertise and favored technologies into account when choosing a de facto tech stack for your microservices, as well as when it comes to adjusting the level of freedom you give your team to pick and choose their tools.</span>

<span style="font-weight: 400;">Sure, you need to have some structure, but if you’re too restrictive — or worse yet, blind to the desire of people in your team to innovate with different technologies — you may have a rift of your own to manage.</span>

<span style="font-weight: 400;">So, evaluate your team closely, and come up with a plan that empowers everyone in your team. That way, every section of your team can get involved in major projects, without anyone feeling like they’re being left on the bench, so to speak.</span>

## <span style="font-weight: 400;">The question of team organization</span>

<span style="font-weight: 400;">The way your team is structured is also going to impact your microservices engineering culture — for better, or worse.</span>

<span style="font-weight: 400;">For example, it’s common for software engineers to write the code before shipping it off to the operations team, who in turn deploy it to the servers. But when things break (and things</span> _<span style="font-weight: 400;">always</span>_ <span style="font-weight: 400;">break!), an internal conflict occurs.</span>

<span style="font-weight: 400;">Because operation engineers don’t write the code themselves, they rarely understand problems when they first arise. As a result, they have to get in touch with those who did code it — the software engineers. So, right from the get-go, you’ve got a middleman relaying a message between the problem and the team that can fix that problem.</span>

<span style="font-weight: 400;">To add an extra level of complexity, because software engineers aren’t involved with operations, they often can’t fully appreciate how their code affects the overall operation of the platform. They only come to know of any issue when operations engineers complain about it. As you can see, this is a relationship that’s destined for constant conflict.</span>

<span style="font-weight: 400;">One way to attack this problem is by following the lead of Netflix and Amazon, both of which favor decentralized governance. James Lewis and Martin Fowler, two software development thought leaders, laid out their thoughts</span> [<span style="font-weight: 400;">via a lengthy blog post</span>](https://martinfowler.com/articles/microservices.html#ProductsNotProjects)<span style="font-weight: 400;">. According to them, decentralized governance is the way to go when it comes to microservice team organization.</span>

<span style="font-weight: 400;">“One of the consequences of centralized governance is the tendency to standardize on single technology platforms. Experience shows that this approach is constricting - not every problem is a nail and not every solution a hammer,” the article reads.</span>

<span style="font-weight: 400;">“Perhaps the apogee of decentralized governance is the ‘build it, run it’ ethos popularized by Amazon. Teams are responsible for all aspects of the software they build including operating the software 24/7,” it continues.</span>

<span style="font-weight: 400;">Netflix, Lewis and Fowler write, is another company pushing higher levels of responsibility on development teams. They hypothesize that, because they’ll be responsible and called upon should anything go wrong later down the line, more care will be taken during the development and testing stages to ensure each microservice is in ship shape.</span>

<span style="font-weight: 400;">“These ideas are about as far away from the traditional centralized governance model as it is possible to be,” they conclude.</span>

### <span style="font-weight: 400;">The Concept of High-freedom, High-responsibility</span>

<span style="font-weight: 400;">During my interview with Sumo Logic CTO Christian Beedgen and Chief Architect Stefan Zier,  they expanded on this topic, adding clout to the position that, if you’re going to give developers freedom to choose their technology, it has to come with a high level of responsibility attached.</span>

<span style="font-weight: 400;">“It’s really important that [whoever builds] the software takes full ownership for it. In other words, they not only build software, but they also run the software and remain responsible for the whole lifecycle,” they said.</span>

<span style="font-weight: 400;">They continued by explaining that a system that resembles a federal government system should be put into place to help keep those freedoms in check by heightening responsibility.</span>

<span style="font-weight: 400;">“[You need] a federal culture really. You've got to have a system where multiple, independent teams can come together towards the greater goal. That limits the independence of the units to some degree, as they have to agree that there is potentially a federal government of some sort. But within those smaller groups, they can make as many decisions on their own as they like within guidelines established on a higher level,” he said.</span>

<span style="font-weight: 400;">Decentralized, federal, or however you wish to frame it, certainly seems to be an attractive approach to structuring microservice teams. It’s an approach that gives each team and each team member the freedom they want — without giving anyone free reign to pull the entire project apart at the seams.</span>

<span style="font-weight: 400;">But not everyone agrees.</span>

### <span style="font-weight: 400;">Who gets paged on the weekend?</span>

<span style="font-weight: 400;">You see, a decentralized system implies that each decentralized team takes responsibility for one service, or one set of services. But that too creates a problem; silos.</span>

<span style="font-weight: 400;">That’s one reason why Darby Frey, the Co-founder of Lead Honestly, isn’t a proponent of the concept of decentralized governance.</span>

<span style="font-weight: 400;">“The pattern of ‘a single team is responsible for a particular service’ is something you see a lot in microservice architectures. We don't do that, for a couple of reasons. The primary business reason is that we want teams that are responsible not for specific code but for customer-facing features. A team might be responsible for order processing, so that will touch multiple code bases but the end result for the business is that there is one team that owns the whole thing end to end so there are fewer cracks for things to fall through,” Frey explained.</span>

<span style="font-weight: 400;">The other main reason, he continued, is that developers can take more ownership of the overall project:</span>

<span style="font-weight: 400;">“They can actually think about [the project] holistically,” said Frey.</span>

<span style="font-weight: 400;">Nathan Peck, Developer Advocate for Container Services at Amazon Web Services,</span> [<span style="font-weight: 400;">explained this problem in more depth</span>](https://medium.com/@nathankpeck/microservice-principles-decentralized-governance-4cdbde2ff6ca)<span style="font-weight: 400;">. In essence, when you separate the software engineers and the operations engineers, you make life harder for your team whenever an issue arises with the code — which is bad news for end users, too.</span>

<span style="font-weight: 400;">But here’s what I’m thinking; does decentralization need to lead to separation and siloization?</span>

<span style="font-weight: 400;">Peck goes on to explain that his solution lies in DevOps, a model aimed at tightening the feedback loop by bringing these two teams closer together, strengthening team culture and communication in the process. Peck describes this as the, “you build it, you run it” approach.</span>

<span style="font-weight: 400;">However, that doesn’t mean teams have to get siloed or distanced away from partaking in certain tasks, as Frey suggests might happen.</span>

<span style="font-weight: 400;">“One of the most powerful approaches to decentralized governance is to build a mindset of “DevOps”,” Peck wrote. “[With this approach], engineers are involved in all parts of the software pipeline: writing code, building it, deploying the resulting product, and operating and monitoring it in production. The DevOps way contrasts with the older model of separating development teams from operations teams by having development teams ship code ‘over the wall’ to operations teams who were then responsible to run it and maintain it.”</span>

<span style="font-weight: 400;">DevOps, as</span> [<span style="font-weight: 400;">Armory CTO Isaac Mosquera explained</span>](https://buttercms.com/blog/microservices-for-startups-an-interview-with-isaac-mosquera-of-armory)<span style="font-weight: 400;">, is an agile software development framework and culture that’s gaining traction thanks to — well, pretty much everything that Peck said.</span>

<span style="font-weight: 400;">Interestingly, Mosquera feels that this approach actually flies in the face of</span> [<span style="font-weight: 400;">Conway’s Law</span>](https://en.wikipedia.org/wiki/Conway%27s_law)<span style="font-weight: 400;">:</span>

<span style="font-weight: 400;">"Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations." — M. Conway</span>

<span style="font-weight: 400;">“Instead of communication driving software design, now software architecture drives communication. Not only do teams operate and organize differently, but it requires a new set of tooling and process to support this type of architecture, i.e. DevOps,” Mosquera explained.</span>

<span style="font-weight: 400;">Chris McFadden, VP of Engineering SparkPost, has an interesting example that might be worth following. At SparkPost, you’ll find decentralized governance — but you won’t find a one-team-per-service culture.</span>

<span style="font-weight: 400;">“The team that is developing these microservices started off as one team, but they’re now split up into three teams under the same larger group. Each team has some level of responsibility around certain domains and certain expertise, but the ownership of these services is not restricted to any one of these teams,” said McFadden.</span>

<span style="font-weight: 400;">This approach, McFadden explained, allows for any team to work on anything from new features to bug fixes to production issues relating to any of those services. There’s total flexibility, and not a silo in sight.</span>

<span style="font-weight: 400;">“It allows [the teams to be] a little more flexible both in terms of new product development as well, just because you're not getting too restricted and that's based on our size as a company and as an engineering team. We really need to retain some flexibility,” he continued.</span>

<span style="font-weight: 400;">However, size might matter here, as McFadden admitted that if SparkPost was a lot larger, “then it would make more sense to have a single, larger team own one of those microservices.”</span>

<span style="font-weight: 400;">“[It's] better, I think, to have a little bit more broad responsibility for these services and it gives you a little more flexibility. At least that works for us at this time, where we are as an organization,” he said.</span>

<span style="font-weight: 400;">On the other hand, Sam Stagg, the VP of Engineering at Pusher, takes Conway’s approach — choosing to keep teams working on separate areas of the project:</span>

<span style="font-weight: 400;">“We have tried to organize our team to exploit Conway’s Law. Thus, we have a small team who builds core shared services for the platform, and services that support specific products are built by the teams who build the products,” said Stagg.</span>

## <span style="font-weight: 400;">A successful microservices engineering culture requires a delicate balancing act</span>

<span style="font-weight: 400;">When it comes to technology, freedom — with responsibility — looks to be the most rewarding path. Team members with differing technological preferences will come and go, while new challenges may require you to ditch the technologies that have previously served you well. Software development is constantly in flux, and so you’ll need to continually balance the needs of your team are new devices, technologies and clients emerge.</span>

<span style="font-weight: 400;">As for structuring your teams, a decentralized, yet un-siloed approach that leverages DevOps and instills a “you build it, you run it” mentality seems to be popular, although other schools of thought do exist. As usual, you’re going to have to experiment to see what suits your team best.</span>

<span style="font-weight: 400;">Here’s a quick recap on how to ensure your team culture meshes well with a microservices architecture:</span>

*   **Try to be Sustainable, Yet Flexible:** <span style="font-weight: 400;">Balance sustainability without forgetting about flexibility and the need for your team to be innovative when the right opportunity comes along. However, there’s a distinct difference of opinion over how you should achieve that balance.  </span>
*   **Give Equal Opportunities:** <span style="font-weight: 400;">Don’t favor one section of your team over another. If you’re going to impose restrictions, make sure it’s not going to fundamentally alienate team members from the get-go. Think about how your product roadmap is shaping up and forecast how it will be built and who’s going to do the work.</span>
*   **Structure Your Team to be Agile, Yet Responsible:** <span>Decentralized governance and agile development is the flavor of the day for a good reason, but don’t forget to install a sense of responsibility within each team.</span>

Web Components are the central concept of Micro Frontends: a new way to design complex front-end application similar to the Microservices architecture. Using microservices architecture, we can use  [Web Components to create micro applications](https://buttercms.com/blog/using-web-components-with-buttercms), each with a specific mission. Check out  [our post](https://buttercms.com/blog/using-web-components-with-buttercms) for more

_Thanks to the following individuals for their contributions to this chapter:_ _[Ben Curtis](https://twitter.com/stympy?lang=en), [Stephen Blum](https://twitter.com/stephenlb?lang=en), [Darby Frey](https://twitter.com/darbyfrey), [Christian Beedgen](https://twitter.com/raychaser), [Stefan Zier](https://twitter.com/stefanzier), [Isaac Mosquera](https://twitter.com/imosquera), [Chris McFadden](https://twitter.com/cristoirmac), [Sam Stagg](https://twitter.com/swstagg), and [Sierra Sterling](http://www.linkedin.com/in/sierrasterling). _

<div class="chapter-cta well">

This is part of [_Microservices for Startups_](/books/microservices-for-startups). If you've enjoyed, please share!

### Enjoy this chapter? Don’t miss the next one.

<div id="mc_embed_signup">

<form action="//buttercms.us11.list-manage.com/subscribe/post?u=fbebec17d74d7b0e85da677a3&amp;id=6f5fadd58c" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate form-inline" target="_blank" novalidate="">

<div id="mc_embed_signup_scroll">

<div class="mc-field-group"><input type="email" value="" name="EMAIL" class="form-control input-lg required email" placeholder="Email..."> <input type="submit" value="Send me new chapters" name="subscribe" id="mc-embedded-subscribe" class="button green"></div>

<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_fbebec17d74d7b0e85da677a3_bde6aa4670" tabindex="-1" value=""></div>

</div>

</form>

</div>

</div>