January 26, 2020

Integration

people over process in modern software development

One year ago, in January 2019, Chris Coyier released the article The Great Divide. He shows that we are facing some challenges by introducing new disciplines and sub-disciplines to the field of software development.
What do you need to learn, to call yourself a frontend developer? What even is frontend-development?
The difference between "front of the frontend" (UX, UI-Design, a11y) and "back of the frontend" (programming) shows a divide - that is not without cost.

A cost that needs to be addressed: with every specialization comes the need for integration.
I would argue, that this is mostly overlooked and it creates one of the biggest problems in modern software development.

In this article, i will define the meaning of integration and why we need to address it.
I will describe a systemic problem at the very core of how we work,
and show a solution to resolve this problem - something you can use today.

This is all in good spirit, as this article is about improvement and a positive outlook on what is possible.

I hope that you will, more often than not, find a way to discover that integrated teams can be much bigger than the sum of their parts.

System And Parts

A way to look at our situation, is through the lens of Systems Thinking.

Systems Thinking was pioneered by the work of Russell Ackoff. In this video, he talks about improvement. Something that you may see as fitting for this article.

play the youtube video

I will summarize some of his points, for the following explanation.

We are dividing the system of software development into parts (specializations). The assumption is, that if we improve every part separately, the system as a whole will improve.
But this is very misleading.
A system is a whole, that can't be divided into independent parts.
The parts are all interconnected. Improving a part will not necessarily improve the system as a whole.
Most properties of a system will derive from the interaction of its parts - not from the parts taken separately.

Example
When an architect designs a house, he can't design the rooms taken separately.
The rooms are all interconnected, and if he wants to make the bathroom bigger, he knows that this has an impact on other rooms and the structure of the house itself. For the best possible house, no room might be the best possible room.
Everything needs to be evaluated against the improvement of the whole.

This is directly reflected in how we develop the software itself.
We often act and think in silos, and our focus easily shifts from improving the software as a whole to improving specific elements we see as important.
UI-redesigns, performance optimizations and better code abstraction layers - there is always something we can make better.
At the end of the day it comes down to what we have done with our limited time, money and information.
If we only improve the parts without the focus on the improvement of the system as a whole, we will not only be wasting a lot of opportunities to deliver the best possible result to our customers, we will make the system worse.

Hidden Cost

We assume that the right process (for example an agile framework) will bring everything together, but the truth is, that any process is limited to means of coordination.
We need a good process, but it is no substitute for integrative methods.

Think-Make-Check circle: coordination VS integration

Integration (on the right) can be shown as an overlap of multiple disciplines. It is more than bridging different disciplines. It is a common ground.
Without it, we are only able to throw our work over the fence for the next team to continue.

There is something called a "Developer Handoff".
A process, where a designer will hand the results over to the development team.
If the designer made a good job, the developers are able to understand enough to implement the necessary code.
The work is divided, and for a time this seems to be a great idea - but over time, we can see what we have missed.
The necessary interactions between designers and developers.
Developers can get angry and frustrated. "What are the designers doing? This is way to complicated!".

This is why we need coordination and integration. They are a complementary pair.
On a coordination level, we answer the questions on how to do things (the process).
Integration is not about the process, it is about a common ground that enables us to share an understanding.
TODO: WHY DO WE NEED INTEGRATION? Integrated teams are able to share an understanding to add value efficiently.

Information -> Knowledge -> Understanding

An ounce of information is worth a pound of data.
An ounce of knowledge is worth a pound of information.
An ounce of understanding is worth a pound of knowledge.

Can we make choices to enable better integration?
I think that we have to address a situation at the very core of how we work.
If we would resolve this issue, we would create opportunities to unify different disciplines.

Duality

You are in sprint 32.
Someone shows you a database schema.
Do we need field "x"?
You could say: it depends.
It depends on what?
It depends on the context

Your answer could be: Yes, we need field "x" because it is required for the user registration process and for...
This is an answer someone might expect, but to be able to give it, a developer needs to know all the scenarios that need field "x" and this information might not be easily found, as it got complected into the structure.

the divide

Design and Testing will describe the software in terms of scenarios (seen on the left).
On the right side, developers will describe the software in terms of an executable software construct.

This is the duality. On one hand, we have a scenario based thinking that lets us define in a very human way how things should work. This is in fact how we -as humans- try to make sense of the word that surrounds us.
Like: "the user is able to register by providing the needed information".

On the other hand, we write software, because we need something that is machine-executable. We add methods, classes, components, ..., to the software construct, so that it is capable of running our scenario. "we need to implement a new service, the form-component needs to include field x, ..."

drifting apart

This is the divide at the very core of how we work.

Problems

From a developer perspective, this duality creates a world of problems. Here are my top 3.

1. complecting scenarios

After adding functionality to the same part of our construct, we might fail to reason about it.
What scenarios have shaped this component?
Why is this if/else here?
If we are not able to reason about our code, how can we make confident changes?
Complecting scenarios will result in complexity.

2. enabling unwanted scenarios

When we implement a new feature, we add new functionality to parts of our construct - but what unwanted scenarios we have we accidentally enabled, too?
Bob Martin tells me that i need "negative tests".
It is like the advice to "expect the unexpected" - those tests are not easy to find.

3. change of mindset

We want to increase the power of our software.
The wish is, that the software will be so well constructed, that it would be easy to enable possible future scenarios, or even possible future projects.
We focus on our powerful software - and less on the needs of our users.


Resolving

Different specializations not only need to be coordinated - they need to be integrated as well.
To enable integration between two disciplines, we need something that serves as a common ground.
In other words, it would be beneficial to talk the same language when we talk about software.

The goal is to keep an artifact during the whole process. From design over to development and testing.

Modelling languages for reactive systems, like Statecharts come to mind.

Statecharts

TODO: short introduction on statecharts.
They serve as an integrative tool, because the artifact can be used across different disciplines, as long at the participants understand the basic building blocks. - todo: make clear that people need to learn the modelling language.
People from different disciplines are enabled to communicate with the help of the statechart graph.

An excerpt from Statecharts in the Making by Prof. David Harel:

I recall an anecdote from somewhere in late 1983, in which in the midst of one of the sessions at the IAI ... There was a knock on the door and in came one of the air force pilots from the headquarters of the project. He was a member of the "customer" requirements team ... he had never seen a state machine or a state diagram before, not to mention a statechart.
He stared for a moment at this picture on the blackboard, with its complicated mess of blobs, blobs inside other blobs, colored arrows splitting and merging, etc., and asked "What's that?"
One of the members of the team said "Oh, that's the behavior of the so-and-so part of the system, and, by the way, these rounded rectangles are states, and the arrows are transitions between states". And that was all that was said.
The pilot stood there studying the blackboard for a minute or two, and then said, "I think you have a mistake down here, this arrow should go over here and not over there"; and he was right.

Todo: The statechart approach is intra-object, in that ultimately the recommendation is to prepare a statechart for each object of the system (or for each task, function, component, etc., whatever artifacts your system will be composed of). Of course, the statecharts are to also contain information about the communication between the objects, and one could build special controlling statecharts to concentrate on these aspects; however, by and large, the idea of finite-state machines in general, and statecharts in particular, is to provide a way for specifying the behavior of the system per object in an intra-object fashion. The more recent work has to do with scenario-based, inter-object specification.

Inter-Object Modelling

The idea is to concentrate on specifying the behavior between and among the objects (or tasks, functions, components, etc.). The main lingua franca for describing the behavior of the system would have to be a language for specifying communication and collaboration between the objects

scenarios are like threads that use different parts of the construct.

For example: A button to send a request, or the service that makes an api call, can be part of the same scenario, even if they are in different parts of the construct.

The big difference between Scenario Based Programming is that scenarios are intra-object. Statecharts are intra-Object: https://arxiv.org/pdf/1911.10691.pdf

Scenario Based Programming

TODO: show a user-flow example that shows the inter-object behaviour of SBP.

  • show user flow
  • add scenario based programming elements
  • describe interactions between UX and DEV
  • describe wait, block and request

What Is Next?

In the next article i go into code details.
I will show the benefits for testing and describe a full circle with this integrative core.
It turns out that scenarios really shine, when we look at intra-component functionality.
Use this RSS-Feed to be notified.
Or follow me at twitter @thomas_deutsch.

Thanks for reading this article.
If you would like to say hello, please do so.
Do you have other ideas on how to enable integration as a central part of your process?

If you want to know more...