Value Types In Kotlin

With the recent release of Kotlin version 1.5 the value types have exited the experimental stage. This means we can now use type driven development approach without the fear of the overhead that wrapping in a custom classes caused. The value class works by inlining the wrapped value during compilation. From now one we will be able to safety pass values around without the overhead of new instance creation. Example below:

@JvmInline
value class UserId(val value: String)

data class User(val userId: UserId)

Unfortunately we still need to use an annotation and probably will needed it for the time being as stated:

In Kotlin/Native and Kotlin/JS, because of the closed-world model, value-based classes with single read-only property are inline classes. In Kotlin/JVM we require the annotation for inline classes, since we are going to support value-based classes, which are a superset of inline classes, and they are binary incompatible with inline classes. Thus, adding and removing the annotation will be a breaking change.

KEEP

There are also some problems with language interoperability. For example, Groovy. The language does not see those types. In Spock tests we will have to use raw types:

def user = new User("someUserId")

This will cause a lot of headaches during refactors (as is often the case with Groovy, but that is a different topic).

From Java perspective we will (hopefully) have value types from Project Valhalla but since there is no known release date as of today, Kotlin release is very welcome. Especially for someone that firmly believes in type driven development.

Value Types In Kotlin

Scrum only works with independent CI pipelines

Photo by Christophe Dion on Unsplash

The title is a bit clickbaity, I know. I couldn’t figure out a better one, so there it is.

What is agile development? This is a very controversial topic. There is the Agile Manifesto, there are strict “Agile” frameworks like Scrum and methodologies like Kanban. I like the simple definition where a team of senior engineers is put into the conference room with a direct line to the client and are asked to deliver a releasable program increment every X period of time. Scrum has a lot of additional elements, one of which is the requirement that the program increment should have prior agreed set of features. One could argue that this limits the development flexibility but allows for the development team to focus on the delivery and make predictions on where the product could go in the future. You can change the scope of the sprint during its run but this is frown upon and with good reason. In order for the team to be able to estimate with any certainty and agree to the sprint scope with the PO during the planning, it needs to have a certain level of predictability of the environment in which they operate. Continuous Integration (CI) pipeline plays a huge role in that predictability.

CI pipeline is an automated process that takes our new code change and runs it through a series of steps to verify its correctness, like running tests, static analysis tools, etc. Generally any modern development team/s will have a CI pipeline as they help with making sure our change is correct. The problem arises when the same pipeline is used by multiple teams.

What happens when the teams share the CI pipeline?

The severity of the problems outlined here, depend on the pipeline implementation, however all of them eventually hit these issues:

Queues

When many engineers try to merge their changes, there will be times when the CI will become overloaded and the engineers will have to wait a long time for their change to be verified. The more engineers, the longer the wait will be. This can be mitigated with automated scaling but this will become either expensive or unstable (if low quality servers are used or something like AWS Spot instances).

Pipeline failures

Having one pipeline makes changes to it difficult and dangerous. Just like you would be cautious with making changes to your product on all client instances simultaneously. One wrong change can cripple the pipeline and force all your developers to be unable to work, potentially costing the company millions. Having many deployment pipelines lets you do independent changes, e.g. through canary releases.

Bloat

This will be more pravailaint with large monorepos, but having one deployment pipeline makes it very tempting to basically build everything everytime. As the code base grows, so will the build times, making the developers again wait longer and longer. This can be mitigated with appropriate build tools like Bazel (never used it personally, but I heard good things) but it can be costly for the company to change their processes once they reach the point where this becomes a problem.

Instability

Once in a while some team will try and push something through the CI something that will cause it to die. Maybe they will deploy some docker image that blocks a vital port (seen that happen, done it myself once or twice) or run a script that kills everything on the given machine (also seen it). Such mistakes happen. They should not block merges of other teams.

You might ask, so what? Just take that into account when estimating the time needed to deliver the feature. After all the estimates are made based on previous estimates, when those incidents also occurred. The problem is those issues are unpredictable. One sprint they will not happen, the second they will, derailing the whole sprint. We want to limit uncertainty to increase predictability.

So what to do with it?

It would be perfect if the team was the one maintaining and taking the responsibility for their CI pipeline. This will give them high control of their environment, significantly lowering uncertainty (of course there can still be unforeseen circumstances like a network failure, but there is little one can do with that). This approach can be unrealistic (and unnecessary in a company with high DevOps culture) in large corporations, for various reasons like security, costs, etc. However if the company does not have deployment properly figured out, they should let the teams set up their own pipelines and manage them independently. Once the company matures then it can try and unify the processes, keeping the pipelines separate. Just like in a microservice architecture. This approach works for any project setup. Regardless if it is a monolith or microservice, monorepo or multi repo (although it is easiest in a multi repo microservice architecture).

Final Thoughts

Think about this in terms of encapsulation. You try to enforce encapsulation within your services/modules in order to make them loosely coupled, stable and easy to change. The same applies to CI pipelines. Having a separate CI pipeline will allow the team to make better estimations as there are fewer variants.

That said, in some products having independent pipelines will be infeasible. It can be too expensive/time consuming, there might be a lack of skill necessary within the teams, etc. There can be many reasons. This, however, means it is also infeasible to expect development teams to deliver releasable features every sprint. And that is OK, alabait painful. In such cases maybe use something closer to Kanban than Scrum.

Scrum only works with independent CI pipelines

The Mythical Modular Monolith

What is it and how to build one?

Photo by Raphael Koh on Unsplash

We all know the Microservice trend that has been around for years now. Recently voices started to rise that maybe it is not a solve-all solution and there are other, better suited approaches. Even if, eventually, by evolution, we end up with a Microservice architecture, then there are other intermediate steps to safely get there. Most prominent, a bit controversial in some circles, is the modular monolith. If you follow tech trends you would have already seen a chart like this:

I will not get into details of this diagram, since there are other great resources that talk about it. The main idea is that if are at the bottom left (Big Ball of Mud) then we want to move up and right through the modular monolith instead of the distributed ball of mud. Ideally we want to start with modular monolith and if our product is successful enough potentially move towards Microservices.

The problem I found with those articles/talks is that they talk about the modular monolith but rarely go into details as to what that actually means, as if that was self-explanatory. In this piece I will outline some patterns that can help when building a modular monolith.


What is a modular monolith?

Before we can talk about how to build a modular monolith we need to answer this question. After all, what makes it different from a regular monolith? Is it just a “correctly” written monolith? That is correct but too vague. We need a better definition.

The key is in the word modular. What is a module? It is a collection of functionalities that have high cohesion in an isolated environment (lowly coupled with other functionality). There are various techniques that can be used to collect such functionalities and build a boundary around them, e.g. Domain Driven Design (DDD).

Breaking it down:

  • Clear responsibility / high cohesion: Each module has a clearly defined business responsibility and handles its implementation from top to bottom. From DDD perspective: a single domain.
  • Loosely coupled: there should be little to any coupling between modules. If I change something in one module it should affect other modules in a minimal way (or even better not at all).
  • Encapsulation: the business logic and domain model should not be visible from outside of the module (linked with loose coupling).

A good rule of thumb to check if a module is well written is to analyse how difficult it would be for it to be extracted into a separate microservice. If it’s easy then the module is well written. If you need to make changes in multiple modules to do it, then it needs some work. A typical ball of mud might also have modules but they will break aforementioned guidelines.

Having defined what a module is, defining a Modular Monolith is straightforward. A Modular Monolith is a collection of modules that adhere to those rules. It differs from a Microservice architecture that all the modules are deployed in one deployment unit and often reside in one repository (aka mono-repo).


Integration Patterns

There are a number of integration patterns one can employ when building a modular monolith.

Each one has its strengths and weaknesses and should be used depending on the need. I have ranked them in the level of maturity.

Level 1: One compilation unit/module

The codebase has one compilation unit. The modules communicate between each other using exposed services or internal events. This approach is fastest to implement initially, however as the product grows it will become progressively harder to add new functionality as the coupling tends to be high in such systems. Same applies to ease of reasoning. At first it will be very easy to “understand” the system. As the time flows the number of cases needed to keep in mind will grow as it is very hard to determine what are the relations between domains. Benefit from this approach is that we can quickly deliver initial value while refining our development practices (especially new teams). From a practical point of view the build/test times of such a system will rise exponentially, slowing down development.

Recommendation: Use with a single small team (2–3 people). Ideal for Proof of Concept and MVPs.

Level 2: Multiple compilation units/modules

The codebase has multiple compilation units, e.g. multiple maven modules, one per domain. Each module exposes a clearly defined API. This approach allows for better encapsulation as there is a clear boundary between the modules. You can even split the team and distribute responsibility for each module, allowing for independent development. The readability also benefits from this approach since it is easy to determine what dependencies are between the modules. In addition we can only build and test the module that has been changed. This speeds up development significantly. Requires a little bit more fiddling with build tools but nothing a regular developer couldn’t handle.

Recommendation: Good for a typical product team. Team members can work fairly independently. Could work with 2–3 small teams. This approach will take you far as the implementation overhead is small, while it is very easy to maintain consistency through the code.

Level 3: Multiple compilation module groups

Each domain is split into 2+ modules. This is an expansion on the previous approach. This way we can extract an API module that other domains will be dependent on. This will further enforce encapsulation. You can even employ static analysis tools that ban other modules from being dependent on anything but the API modules. This approach could benefit from the Java Jigsaw Project.

Recommendation: This is ideal when moving from a medium sized product to a large product where 2+ full size product teams are needed. Each team will expose their API module that the others can ingest.

Level 4: Going web: Communicating through network

Same as Level 3 but modules are totally independent (no shared API module) and communicate using the network (REST/SOAP, queues, etc). This is an extreme step. One that should not be taken lightly. You lose compile time checking on the APIs and gain multiple problems related to networking. This allows a very high decoupling of the modules as there is no shared code (apart from some utils, etc). When deciding to take this step it means that we are nearing a Microservice architecture.

Option A: Having a single deployment unit. It might seem weird to call a REST API when everything is deployed in one unit but this approach does allow for better load distribution. This is especially possible when using a queue like kafka for communication. However I agree that in most cases this is a redundant approach. It is a good stepping stone when moving to Option B.

Option B: Separate deployment units. This is pretty much the final move from a modular monolith to microservice architecture.

Level 4.5: using separate repositories (aka moving away from monorepo), CI/CD pipelines, etc

Moving to full blown Microservice approach. Not going into detail as this article is not about Microservices.

Recommendation: Large/Multiple Products, many development teams, efficient DevOps culture, mature company.

Summary:

You might have noticed that Cost of Maintenance is never low. That is correct.

You can not hide complexity. You can only change its location.


Contracts and Testing

A very important aspect of modular monoliths is to treat the APIs or domain boundaries (call it as you wish) as Contracts between domains/modules that need to be respected. This might seem obvious but it is easy to fall into a trap in a monolith where the module APIs get treated as second class citizens. When designing and maintaining them we should think of them as if designing a REST API. Changes to them should be done carefully. They should have a proper set of tests. This is key when there are multiple teams cooperating.

One of the more common issue is no clear distinction between what team is responsible for which module. Responsibility for APIs becomes blurred and their quality drops rapidly. Each module should have one team responsible for it. Ownership is key. The API of this module should be a contract between that team and the teams that use it. That API MUST BE covered by tests. Only the team responsible for that module can introduce changes to that API. This does increase communication overhead and extends the time of introducing changes but keeps those factors constant instead of spinning out of control.


Conclusion

I hope those make the Modular Monolith a tiny bit less Mythical. I think we developers like to over-complicate things while in truth most software engineering comes down to a few basic principles. There are a few more topics I think would be worth discussing regarding the Modular Monolith (tooling, architecture as code) but I think this gives a good starting point. The most important takeaways are:

  • Encapsulated modules with high cohesion and low coupling
  • Ownership is key

Keep those in mind and you will get far.

The Mythical Modular Monolith

Testing

What is the point of having tests? Why do we write them? To prove that the code is correct (in some cases, yes)? To fill out the test coverage requirements? Or maybe we don’t write them at all? If this is a single use program or a proof of concept then the last approach is usually best. Sounds heretical but hear me out. 

Where tests shine is in code maintainability. Having tests is indispensable when writing a piece of code that is intended to live. Code that lives is code that changes. Code that has multiple people working on it. Tests give you semblance of security as to the correctness of your changes during refactors/modifications.

I once heard a description of legacy code. Code that reaches a point where no one dares to change it, becomes legacy. I think it is spot-on. Having tests extends code lifespan (all code eventually becomes legacy). Being less scared of introducing breaking changes and having runnable use cases gives people courage. You will need it when asked to modify that piece of code that haven’t been touched for the last 10 years.

Tests are also a great insight into what is the purpose of the particular code. A runnable documentation of sorts. Documentation that has to be updated as functionality changes. When going into a new code base my first reaction is usually to start my reading from the tests. 

However, it’s not all sunshine and roses.

Note: I don’t want to get into discussion as to what is a unit test, what is an integration test, etc. I have seen people fight to the death over nomenclature whose sole purpose is to ease the communication. As long as both parties understand what they are both referring to then you can call it whatever your heart desires. My preferred approach is not to call them anything specific and just state exactly what I am testing. Single method, single class, collection of classes, connection with database, rest endpoint etc.

The problem with having tests, as with all things, is that there is a way to overdo them. Many developers (myself included) would jump on the testing train (and rightly so) and would write a test for every class, every method.  This doesn’t sound so bad. After all, you have a really well tested code, right? In theory, yes. The problem is that the tests cost. Cost time. To write them, to review them, to fix them, to run them. Eventually you will reach a point where adding yet another test will bring very little additional value. This is called the law of diminishing returns. The trick is to strike the balance. There are a number of techniques that try to remedy this like the test pyramids. My experience with those techniques is that often time trying to adhere to their rules can cause us to write tests that we usually would not write, but have to, in order to fit the framework. I have experimented with multiple approaches and I want to share what has worked best for me.

I like the idea of thinking about the tests as documentation. You know, that thing that we always say that we need but no one ever writes and even if there is one then it immediately stops being up to date. Approaching tests as documentation means that they need to cover your business use cases. In my experience, when taking the DDD approach, use cases are usually scenarios that describe how a given bounded context interacts with other contexts. This means that our tests should reside on that boundary. They should run against boundary entry points. For a Microservice this would mean testing the REST/SOAP/whatever endpoints. For a modular monolith, it would mean testing against modules external APIs. 

Does that mean that there should never be tests for individual classes/methods? No. There are bound to be cases where you are delivering a functionality that has a huge number of variations, e.g. a credit score calculator. Testing such a functionality might be impractical through the module API. In such a case a test for an individual class is highly recommended, with a caveat. This class should be leaf. What does that mean? 

This is not my idea but I really like this analogy. When we look at our class dependency graph there are usually classes that are roots, i.e. nothing depends on them. Those are usually our module APIs, aka boundary entry point. This is what we test. The other distinctive classes are those that depend on no other class in our module. Those are usually our repositories, utilities, etc., aka, leaves. Why do we want our class that needs individual testing as a leaf? Exactly because it has no dependencies. We do not have to mock any other service and therefore assume how it behaves. If we had mocks and the behavior changes then we need to remember to fix them all. Without mocks we can truly test this class as a standalone functionality. In order to make our class a leaf we would usually need to refactor it out from the middle of the dependency graph, since, in my experience, such functionalities end up inside domain service classes. As an added benefit such operation usually improves our design.

In summary, we want to test our bounded context APIs, aka module entry points, aka dependency tree roots. For special functionality that needs an individual suite of tests, we extract such functionality to a leaf class and test it in an isolated manner. In my experience, this approach strikes the best cost-benefit balance. This rule of thumb is also quite simple to follow. No need for arguments regarding, if we need to test those getters with five other services mocked. As a matter of fact I have become a firm believer that the existence of a mock is a code smell and probably should be refactored.

I think any text about tests would be incomplete without the mention of test coverage and its measurement. I will not get into much detail here apart from that I am against measuring line coverage and instead much prefer measuring branch coverage. I think it allows for better test optimization. Unfortunately I have not found any good tools for measuring it, especially for large, multi module code bases. Maybe I will eventually write my own or try to fix an existing one.

As a last note, I would like to share my approach to writing tests in Java. Again, this is my preferred way. I really like how Spock (a Groovy test framework), forces the developer to use the BDD approach. I try to write all my tests using it, however there is a huge down side to it, Groovy. As mentioned before, we write tests for code that lives, changes, that will be refactored. Groovy, being a dynamically typed language, is abysmal during refactors. Due to this, I write the tests (Specifications) in Spock, however all the fixture setup is done in plain Java. This way we can get the best of both worlds. Nice, readable tests, and fixtures that can react to refactors.

There is also the subject of context bootstrapping. The most common way in Java by far is by wiring all your applications using Spring and, during tests, starting the whole context. I find this approach to be OK in a Microservice world, but still much prefer bootstrapping the dependencies manually. First of all the tests will run much faster, second I can clearly see how my dependencies interact and therefore check for smells and warning signs. Remember: Dependency Injection is not equivalent to IoC containers. This approach is pretty much the only sensible way in a large monolithic application.

I have not mentioned so called End-to-End tests, aka testing through e.g. clicking on the front-end application that is deployed on a close resemblance of production environment. This is a topic for a whole other post.

Testing

Builder Anti-Pattern

Since I am professionally a Java developer, this will primarily focus on Java code.

I feel there is a misunderstanding, and in effect, overuse of the Builder pattern in Java codebases. The purpose of Builders is to allow creation of complex objects that have different representations (i.e. have optional fields). First and foremost I consider Builders a code smell (as to why, I will get to later). In my experience developers use Builders not as a way to create objects that can have multiple representations but because it is convenient to construct big objects using Builders. I was one of them. On the surface this does not seem to be a bad practice. It allows you to create an object with less chance of assigning an incorrect parameter to a field since you can see what field you are setting. This approach became notorious with the appearance of Lombok (note: this text is not a Lombok hate text, I actually really like some aspects of it). The problem with this approach is that it is not really a Builder pattern. Instead it is a substitution for a lack of language feature called: named parameter (actually most design patterns are ways of working around language deficiencies). When faced with an object that has a huge number of fields typed String, int and boolean (we all saw those), developers cower in fear on a thought of building such an object using a constructor. Anyone with a bit of experience faced hell searching for a bug that was caused by a wrong argument order in a constructor. If Builders solve this issue then what is the point of this post? Lets just use them… except lets not. I say use constructors (i.e. construct the whole object in one method call) or in most extreme cases, factories.

I mentioned earlier that I consider Builders a code smell. Why? Above all Single Responsibility Principle. The clue is in the description of the pattern: construction of different representations. In essence if you need a Builder then there is a huge chance that your object is doing too much. It has too many dependencies and too many properties. Split it. Actually, I have almost never seen a Builder used to construct an object that could have many representations (e.i. all fields were always set regardless of context), meaning that by definition it was used incorrectly.  If an object has one representation but still a large number of dependencies, split it further. If there are many implementations of a single interface that you do not want to expose, use Factory Pattern. An exception could be a large DTO that we have to handle due to some higher forces e.g. integration with external system. Even then I would be very careful due to reasons stated further down. 

Many of us use Domain Driven Design, or at least say they are using. It introduces a concept of Value Objects. Use them. This will solve the problem of having parameters of the same type. Instead of three String typed variables you will be passing three domain specific typed variables. Let the compiler help you. The Project Valhalla will make this even easier. Digression: I will take this opportunity to shamelessly plug something I really believe in, which is Type (not Test) Driven Development. Look it up. Unfortunately Java type system just barely qualifies as a statically typed language (and does not have type aliases) so it makes it hard to benefit form it. Maybe our industry will eventually move to better typed systems.

Next issue I take with Builders is that we forgo the help of the best checking tool available to us during refactors, the aforementioned compiler. When adding fields or dependencies to a class, if you use Builders for construction, you will have to manually check all the places it was used. If you use a constructor, then the compiler will automatically find it for you. The code will not run until this is fixed. I have seen countless bugs caused by not passing all arguments to the builder. This can also happen during initial development. It is easy to accidentally forget a field or set the same one twice (and just compare the number of setters to the number of fields). You might argue that this could be solved using tests (hopefully you have those…). Which is true. But so could be said about the incorrect order of parameters in a constructor. Unless you use e.g. Spock (by the way, love it), and Groovy to construct your Fixtures. God help you. Construct your objects in Java/Kotlin so you have that sweet compiler verification. In addition, compile time checking is faster and… can’t be accidentally deleted. Bottom line: Builders are easier to write initially (in theory) but harder to maintain. Note: I find modifying/removing fields to have a similar refactoring cost in both constructors and Builders.

This is mostly a problem with Lombok Builders but they leak implementation details. They expose all internal fields (unless there is a feature I am not aware of). Not a huge problem for anemic objects, especially if the Functional Programming approach is used (but then you face other problems, see below). Not acceptable when used with OO related encapsulation.

Builders used with a Functional Programming approach should be immutable. Lombok Builders are not. So are most implementations I have seen. So now you have an object (Builders are objects too), that is mutable and, depending on use, possibly stateful. One of useful things that Lombok introduced to Java is the toBuilder() method. It allows you to create a shallow copy of an object while modifying fields. Reminds me of functional lenses.

So this is it. If you actually must use Builders then please take the Effective Java approach: The mandatory fields are initialized in the Builder constructor, while optional fields through setters. This allows us to make sure necessary fields are always passed since we have compile time checking, while still allowing us to have multiple representations.

Builder Anti-Pattern

F*** Microservices

This is a rant. If you have some experience with microservice architecture then you will probably find nothing new here. I focus here on an architectural choice where a single team works on multiple deployment units especially where there are more deployment units than developers on the team. For me a team consists of up to 6 developers. If you have more, then your organization has other, more pressing, problems.

First of all, if you do not know what your business process is going to look like, especially if your product is not in production yet, do not make it a distributed process. I am not talking about orchestration vs choreography here. I am all for orchestration, and a clearly visible process that is defined in one place, however that discussion is for another post. I take issue with a business process that has to communicate with multiple deployment units in order to perform its task, regardless of the communication protocol (REST, event driven, message driven, etc). Apparently some believe that developing a process that changes weekly (or even daily), is easier to do it when it spans multiple services/repositories. 

– We are not in production, we can break our platform for a small period of time to save time. No worries. 

– Sure, but then why are you applying microservices architecture right now in the first place? 

– So that the team can learn how to work with microservices before we are in production.

– That is great, but if you are deploying services with breaking changes then how are they supposed to learn?

– Ok, let’s do it properly then. With backward compatibility and versioning.

And then you fail to meet deadlines arbitrarily set by business. 

Here is a simple calculation. We have two aggregates that are in two different services and a business process that interacts with them. New requirements require us to change a single field in each of the aggregates. If the process and aggregates were in a single deployment unit this would probably result in a single 20 – 30 line PR (this includes business logic, storage changes, unit and integration tests, etc). Consider a case where the aggregates and the process were in separate deployment units. If we were to introduce breaking changes then we would need 3 PRs of 20 – 30 lines of code. At least 3 times as much work plus higher chance to get it wrong on the service boundaries. If we were to adhere to backward compatibility then this would result in at least 5 PRs. As to why I will leave it to the reader.

Alternatively, although some consider it a heresy, you could keep all your code in a single repository. This would limit the amount of work needed, while providing an additional benefit of keeping the code in a consistent state. The trade off is the swelling of the repository (think indexing in IDE).

I once heard that no breaking changes will be introduced since only additive changes will be done to the integration model. This was for a system that was not even live yet. I have no comment. Even for a system that would be live this seems highly irresponsible. Will it result in a huge mess? Probably.

Next, continuous delivery. By continuous delivery I understand that, after a PR is merged to master, it gets built (e.g. as a docker image), then goes through automated testing and finishes in a state that it could be deployed to production (e.g. canary deployment) at any time, without the need for further testing. If your organisation does not have a reliable system for continuous delivery you should not do microservices. This guideline is not something I came up with but I wholeheartedly support. To be honest I would go as far as to say that if you have more than one deployment unit per team then you should have continuous deployment. Continuous integration is not enough. If for some reason you can’t, be it due to some weird regulations, or whatever, then do not do microservices.

Continuing, the size of the microservices. This has been debated to death. If you follow DDD approach then this should be straightforward and you should not go wrong with bounded context as a boundary. However. If you think that DDD is not for you or your product does not have a domain (do not ask me how), you might end up with one service per database table, or smaller. I draw the line where half of the microservice code is infrastructure. My preference is one service per team of 6 developers (this includes QAs, since I treat them as developers).  If service is too big for them to handle, split it. If there is not enough work to go around, reduce the team.

Microservices are not an excuse for poor design. Neither strategic (e.i. spanning the whole application, multiple services), or tactical (within the bounds of a single service). I have heard Greg Young once talk about treating services as small classes that could be easily written in a week. If you can write it in a week then it shouldn’t be a problem to rewrite it from scratch if needed. This is an awesome concept but this does not mean that the services are supposed to be unreadable. Small classes still are a subject to clean code principles. When time comes to rewrite it in 6 months time you have to be able to understand what this service does. After all the code is the documentation, isn’t it?

Monitoring. This is another thing that has been talked about multiple times. When applying microservice architecture your monitoring has to be top class. You will not be able to test all the integration variations that happen between services. Neither in an automated or manual manner. If your organisation cannot supply metrics in an easy to access way (and searching though individual AWS ECS instances does not count), then get this straightened out first before diving with multiple deployment units. 

IMPORTANT NOTE: I believe determining what should be monitored is the first thing that should be done when starting working on a new product. Both from a technical and business view. Reason for this is that if you know what you need to measure then you know what your product should deliver. Treat it as a specification of the fitness function for your product.

Whenever you start working on a new service, always run it with two nodes from the start. Regardless of the size of the service. If you cannot develop multi-node service, then your service is not horizontally scalable, you cannot provide at least some sensible measure of availability, doing zero down or canary deployments becomes very hard, etc. If I see such a service in development, that is a huge red flag.

Regarding performance. I am not that sold on the benefits of microservices regarding performance. Sure you can scale horizontally the services that are under heavy use (although I am yet to see a spring application that has a small startup time), or make sure that a service that needs low response time doesn’t get killed by that report generation. When you reach this point and these become true problems that can’t be fixed with just adding another machine, it will most likely mean that you succeeded in making a valid product and you have traffic that is probably in top 1%. Rejoice.

As a closing remark, people that often warn against microservices mention that you are not like Netflix. I say the opposite. You are like Netflix and you should behave like them. They had a monolithic application until well over 100 developers were working on it. That is when they decided to split the application. So be like Netflix.

F*** Microservices

On Product Development Team

Disclaimer: I am writing only my opinions that in no way reflect my previous, current or future employers. They are based on my experiences and reflections.

The purpose of this piece is mostly to organise my thoughts and ideas regarding the functioning of teams. If anyone finds something interesting or something they totally disagree with, then great!

I have been pondering upon the idea of a product development team and how they function for a while now. I have been a part of a few and had seen a few, be it in the workplace or spoken about in conferences. This lead me to a number of conclusions as to what, according to me, works well and what does not.

Team size

This feels fairly straight forward. I have worked in a full spectrum when it comes to a number of people on the team. I was the only team member and I was part of 20+ people. In my experience Scrum (not saying this is the origin of the idea), regardless if you like it or not, got this right. Four to six team members is more than enough to get the work done while not too many for the communication overhead to get too large. This, however, comes with a few caveats.

Team member competencies

The first are the team member competencies. The most performant teams I have seen were those where each member was capable of working on every area of the product. What do I mean by this? Let’s take a look at a typical web application that has both a frontend and a backend. Often times the product development team will naturally split into those that work only on the front side and those that work on the back side of the App. Of course there can be other splits, however I find this to be the most common one. My opinion is that allowing such divisions is highly irresponsible.

Let’s start with the ability to replace. Eventually, only people that know only one part of the application will be available. Be it due to holidays or sick leave, or because some people will change teams or even employers. The development, bug fixes, the planning will effectively halt. Some work will still be possible but this makes the employer vulnerable to unexpected circumstances. Its surprising that these situations are still permitted to occur. You could argue that for products that are not yet on production this shouldn’t be such a problem. On the contrary. This cripples the development and the ability plan further work which in effect drains the product funds.

We can go even further. Consider a modern continuous deployment pipeline. I would say that having only one person doing code reviews in such a case is unwise. Now to be able to deploy new code you need a person that publishes a PR and two people that review the code. In a team of six this means that you need to have half of the people working front and half working the back. This might not be optimal regarding characteristics of the project. Maybe UI is very simple and one person would normally suffice? But you need three. What happens if one leaves on a two week vacation? Deployments stop. No hot fixes. No bug fixes. You have two week sprints? Yeah, no new features. Even if you dial down to one reviewer you still need to have two team members in each area present at all times. 

Another huge problem with this approach is that such a split effectively creates two or more sub teams that need to communicate and integrate with each other. Each additional point of integration adds to complexity. This is costly. Instead of limiting the need for communication and overhead, we’re increasing it.

If developers know only a part of the product and how it works then the solutions they come up with will be sub-par. They will be operating with a limited knowledge when designing the product. This is unavoidable in a larger corporation where multiple teams have to cooperate but on a single product level this is unacceptable.

You might be tempted to split the team into e.g. two teams, front and back. Ignoring the fact that this goes against Scrum (if you are using it) idea of a team being able to deliver the whole business functionality from start to finish, you will now need to deal with the overhead associated with having two teams. Double number of meetings, double the amount of work for the business side, etc. In my opinion it is far more efficient to have a single team of six, where.each member can handle every area of the product at least in a basic capacity, rather than to have two, or even three, teams that focus on separate parts.

That is not to say that each member can not have a preference as to what they like to work on. Some might prefer to work more on optimising the database indexes while others configuring the network routing. This is OK. As long as each member has a working knowledge and has done some work in every area. The team should be, then, resistant to team member changes, while limiting the communication overhead and cost of development.

There might be a rare case where you will need someone with highly specialised knowledge, e.g. some detailed behaviour of Oracle Database. To handle this you do not swap out an existing member for one that only works with Oracle Databases. Instead use an external expert as either your teams consultant or add him to the team as a temporary member. We will talk more about this later.

The only reason that I see where such a team could exist is when the budget is lacking. Excuses like we can’t find the right people means that you can’t find the right people at this salary level. If such is the case then maybe you should consider the notion that maybe you can’t afford to build such a product.

Team roles

Generally speaking, each team member should be considered an equal in terms of competences and simply referred to as a developer. However, I would distinguish two other roles that I think are critical for efficient performance.

Side note: Regarding team leads. I have once heard that their role is to make the team better. The team makes the team better. Not one person. There should be no need for a team lead.

Architect

Team should have a dedicated Architect that is part of the development team and counted as a developer. I don’t want to get into minutia of titles (whether this is a system, enterprise, solution, etc) so for the sake of the argument I will refer to the role as simply the Architect. The purpose of the Architect in a development team is to be responsible for (obviously) the whole architecture of the product and its documentation. What I mean by architecture is very broad. It includes not only what you would get from the C4 diagrams (or whatever) but also package structures, design patterns, etc. Of course the Architect is not the only one responsible for it and the decisions should be made by the complete team as they are responsible for the product as a whole. It is however his main purpose.

Another responsibility of the Architect is for the teams best practices. Developing them and making sure they are followed. Actually, the whole team should be responsible for that but the Architect should lead by example.

The Architect must also write code on day to day bases. Not only the fun, cool one that flexes the brain matter (or even worse just POC) but also the boring and mundane one. This is so they can truly understand the ins and outs of their system and get a feel for which architectural/technical decisions were correct and which were not. The Architect is a technological leader and should be the most experienced developer on the team but he is not a team leader and not the boss of the other developers.

There is one more reason for having an Architect on the team. It is hard to swallow but is the reality of our industry. In a corporation, a person with an Architect in his email signature will have more sway and be able to get things that the team needs that other team members would have much bigger difficulty getting (or not at all).

The role of an Architect is a difficult one. It requires from them to be humble enough to allow the whole team to make decisions while being assertive enough to convince them to go another way if needed. They need to be able to understand how the product works as whole, from a greater perspective. How it interacts with the other systems. But, be also able to work on the detail implementation.

Specialists

We have touched upon specialists earlier. Specialists are people outside your core product development team that bring with them a very narrow but deep knowledge in some area. This can range from databases, through networks, security or even UX design. Core team members can work on all of these by themselves but as the saying goes “Jack of all Trades, Master of None”. In the vast majority of projects this is OK. Core team members do not need to be technological experts. They need to be product experts. When the situation comes where you do need that additional knowledge you can hire one for the team. Those specialists can take two forms. As mentioned before, either a consultant (not meaning a type of contract they signed, they can be salary workers from within the company) where you ask for help with a specific problem or as a temporary “visiting” team member. For example a DevOps joins a team for a month when work on new product starts and all the pipelines and environments need to be setup. He joins all the meetings, sits with them and for all intents and purposes is a part of the team. Until his services are no longer needed.

Regardless of which form the cooperation will take, they add up to the team member count and communication overhead. This is why teams of six should be a max. Add two experts like DevOps and UX designer and now there are eight.

Their role is not only to support but also to teach the core team members. Each member has all the same accesses and privileges. They only lack the knowledge. Once they pass that knowledge they shouldn’t be needed to access any resources. There might be some derogation from this rule for security reasons. Care must be taken that those are truly security reasons and not some imaginary managerial hogwash.

In larger organisations those specialists can form their own teams. Just to make sure this is clear, those teams are not product teams. They might even be developing some tools (be it internal or, better, open source) for their work but their main job is to provide specialised support to the product teams.

Team structure

Teams should have a good mix of experience. Of course this is very dependent on the project difficulty (although in my experience most of the complexity in projects comes from interactions and lack of communication than actual technical difficulty). Please do not get hung up on the titles that follow. I use them only to distinguish relative experience levels.

Ideally a team of six would consist of an Architect, Senior Developer, two Regular developers and two Junior developers. This is beneficial for both the employer and employees. The employer will get enough work power from a team to get the job done while not spending a fortune on only experienced developers. The less experienced developers will get a chance to learn while the more experienced developers will be able to delegate most of the mundane work. I know that this seems harsh but that is the truth. Someone has to write those mappers :).

Scaling Down

When scaling the teams down I would first remove one Junior, then one Regular. Making the team even smaller might make it too small to be effective except for the most unique cases. If you must, I would remove the last Junior in order to have a very small but highly performant expert team. Any less is not a team. It’s a pair. If you face a situation where you need to develop a small piece of software where one or two people should be sufficient then consider doing the development as part of one of the existing products. This will give the piece of software an immediate client that will validate (or not) its purpose. There is no need to create an artificial team. If other products could use it as well then make it available to them in any form you seem fit.

Scaling Up

This has been repeated multiple times but let’s repeat it again. You shouldn’t need to scale the team up because the communication overhead will become significant. If you must do so, I would only add Regular developers. You will need someone that can get quickly up to speed and this disqualifies less experienced developers. On the other hand, by the law of diminishing returns, adding a Senior will not get you the bang for your buck. If you need to implement something difficult then it would probably work better to assign an existing team member to it than a new one, even if they are very experienced.

Conclusion

These are my thoughts. You could say my dream team. It would seem that this would be easy to achieve. There really aren’t any hard elements to this. They seem common sense to me but my view is obviously subjective. I once had a chance to work in an environment that somewhat resembled this but only for a short time. And it was glorious.

On Product Development Team