As a continuous delivery engineer I really want software teams to work fast and safe, with ease and having fun. Happy teams are productive, learning and innovative teams. Authoring pipelines is an important aspect of a safe and smooth development cycle. Every new piece of code should be tried, tested, reworked and eventually (potentially) promoted for real use. However, there are challenges…
Introduction, code chaos – reuse is needed!
In the beginning, pipeline as code seemed like a blessing. No more manual job editing, pipeline scripts included with project source code provide full control over CI/CD tools.
But as complexity grew and more and more teams and projects had started, the situation changed.
Problems emerged with diverging configurations and patterns. Getting infrastructure changes reflected across pipelines was taking forever. Teams were losing time maintaining and supporting pipeline code. Pipeline code bugs roamed freely.
Then Jenkins’ ‘Shared Libraries‘ arrived. They provide the needed code reuse and solve most of the issues. They are unit-testable and offer a single place to provide adapters to infrastructural systems.
But then at some point different teams were ending up with quite different shared libraries, because there’s always room for another standard. Code was still being copied: across libraries. Infrastructure changes still need to be applied to all shared libraries.
At that point the question arises: how to apply structure to this ever growing pipeline spaghetti?
We looked at MPL and JTE but they seem complex and introduce even more magic than Jenkins already imposes on developers.
Instead, our solution involves a minimum of just 2 shared libraries and as little magic as possible.
An implicit shared library
For Jenkinsfile pipelines to work it needs to be in the project repository branch that needs building. From the pipeline script (Jenkinsfile) you then either directly write your code or access a script from somewhere else.
The conventional way of doing this is using a shared library. Shared libraries can be ‘imported’ explicitly by annotating an import statement.
Any shared library may also be instead imported implicitly by Jenkins configuration, meaning that every pipeline will always get it imported.
A stack based shared library
In many organizations teams have some room for making their own choices concerning their development stack. Alignment is encouraged, but differences allow for innovation and often a better fit to a specific problem.
This means that code, shared between projects, is likely to fit a specific stack. For example java+tomcat+linux, windows+dotnet or go+linux. Of course code could be written to fit ‘any’ stack, but that would quickly lead to complexity which we would like to avoid.
Another option is to ‘configure’ the differences, but that would lead to lots of (configuration) code in all projects, again something to avoid.
Combining shared libraries
Libraries can be combined. Classes and resources, such as pipeline scripts, can be imported/used from different libraries by just adding more. Either implicitly by configuration or explicitly by naming multiple, using the annotation.
However, Jenkins does not have any dependency mechanism for libraries. One library cannot state that it needs another one to work. Client scripts need to know how to combine them. Furthermore, how resources may overwrite each other or in what order colliding class path entries are chosen is as far as I know not defined and deemed unreliable.
So with all this said. Our solution is to avoid the problem and use one very well known library that’s just always there implicitly, and then allow teams to add their own (usually just one).
A minimal Jenkinsfile
Teams want to get developing meaningful code as quickly as possible to get early feedback. It helps when pipeline code ‘just works’. Having as little code as possible in the Jenkinsfile is ideal.
The full setup
If this seems simple, then good! that was the intention.
The ‘always there’ library provides:
1. convenience ‘var’ scripts (not full workflows).
2. infrastructure related classes (more info here)
The stack/team specific library provides:
1. full pipeline scripts or ‘flows’
2. convenience classes for use in the scripts
An example Jenkinsfile
Let’s say a team named ‘awesome squad’ has their own specific git work flow and setup their own ‘squad-awesome’ shared library.
The only thing in the Jenkinsfile is a call to a ‘var’ script named ‘awesomeSquadSpecialGitFlowPipeline.groovy’ that holds the complete flow specific to the team’s way of working and their stack. Configuration is passed to it using a simple map.
A note on imports
There is a slight issue with the imports required for shared libraries.
Since there is no dependency mechanism for shared libraries, imports between libraries are not checked. Additionally, it seems that Jenkins is only lazy loading libraries if it encounters any relevant imports…
This means that when the Jenkinsfile contains only imports to the ‘stack’ lib, any other library may not actually get loaded! (at least the classes are not put on the classpath).
The line ‘import nl.first8.jenkins.All’ is very important for this reason. The class ‘All’ (arbitrary name) in turn imports some classes (any really) from the implicit library. Those imports then trigger the actual lazy loading by Jenkins of the indirectly accessed library. This prevents the Jenkinsfile from having to specify imports to classes it does not directly use in the script.
Are Jenkins pipelines scalable?
Perhaps. For some organizations, say 10 teams, this setup will be good enough with the benefit of low maintenance due to few moving parts and low complexity (relatively speaking).
For larger organizations that need to scale even more, this setup is probably not going to provide enough help. Whether MPL or JTE are any better, I would really want to know.
One question from me remains though. Is having more than 10 teams on the same ‘platform’ really effective? I expect there may be quite a few hidden downsides in such a situation. If you have any experience in this, I’d love to hear it. Ping me at @bwijsmuller