Cleanup Your Life, Cleanup Your CI/CD Scripts

DevOp’s guide to a long and happy life.

Image for post
Image for post
A beach, fast wifi and a troublefree build, secret to a happy life. Photo by Peggy Anke on Unsplash

How it all began

In the good old days of monolithic architecture, teams would write all their features, APIs and millions and millions lines of code onto a single repository. There was one piece of code to rule them all, the buildscript. Life was good. The DevOps (or their early equivalent) made infrequent changes, sipping coffee and discussing episodes of Dallas while watching the code slowly build on the server.

How did we get here

We’ve all been here. It is a Friday afternoon. Several large features from two teams must be deployed before the weekend, next week’s large marketing promo depends on it. And so your life depends on it. Toni, who regularly does deployments has conveniently taken leave today. So the honor has gone to you. You got this! There is a little catch, the CTO’s golf buddy has offered free space on their new cloudspace.com. So you need to modify the scripts, you look at one and panic begins to seep in.

Alone and forgotten

How’d it get so messy, when all the rest of the code (let’s just assume, but I hear you readers laughing) is so pretty and beautifully formatted? Well, build scripts often lack developer attention and linter love. They are quickly copied from some template at the beginning so that projects can be built and spun up as fast as possible. As the project grows, environments are added and removed, build stages and triggers get tacked on, modified, copied and pasted on multiple times. The result is a Frankensteinian script that most devs stay away from not wanting to risk breaking builds.

Figure 1: A typical script
  • Get dependencies
  • Do some linting/formatting
  • Run unit tests
  • Run the project build
  • Push image to repository
  • Done and Reporting

Validation Tools

Fortunately, there are a few tools available to help check our bloated scripts. Or perhap, in unspoken acknowledgement that we need help, these tools were created.

Figure 2: Validation tools for your scripts.

There is a better way

Is there a better way? The answer is a loud and emphatic yes! As shown in Figure 1, a lot of the steps are repeated for each environment or triggers. It seems logical to group everything together into separate scripts.

Build/
├── doBuildDocker.yml
├── doLoginDocker.yml
├── doBuild.yml
├── doTests.yml
├── getDependencies.yml
Figure 3: Generalized main pipeline using templates.
- template: azure/go/doBuild.yml
parameters:
imgName: $(imageName)
buildPath: $(buildPath)

An even better way

What if you can remove the build scripts completely from the main project repository? That way, any changes to the build steps will not require changing and committing to the project repository. Also this removes the build scripts from unintentional modifications that could cause breakage.

azure/
├── README.md
├── azure-pipelines.yml
├── docker
│ ├── doBuildDocker.yml
│ └── doLoginDocker.yml
├── go
│ ├── doBuild.yml
│ ├── doGenericScript.yml
│ ├── doTests.yml
│ └── getDependencies.yml
├── node
│ ├── doBuild.yml
│ ├── doGenericScript.yml
│ └── doTest.yml
resources:
repositories:
- repository: repositoryname
name: yourcompany/pipeline
type: github
endpoint: endpointname
Figure 4: Build script calling templates from an external resource.
Image for post
Image for post
Figure 5: How to mirror an external repository into GitLab

Setup External Template

How do you set up an external resource for use in an azure pipeline? The steps are simple but the documents are a bit lacking here.

  1. Click on the service connection
  2. On the New Service dialog (see Figure 7), click which repository your external templates is located. For instance click GitHub if it is on Github.
  3. For GitHub, you need to generate a Personal Access Token to allow Azure CI pipelines to access the repository.
  4. Finally, note the endpoint name assigned to your new service. This name will be used in the pipeline declaration under endpointname field.
Image for post
Image for post
Figure 6: Create a new service connection in the Project Settings.
Image for post
Image for post
Figure 7: Specify the external resource type

The last word

By removing all scripts related to build to an external repository, and keeping a single generic script with the project, you have a single point to manage the build for all the services. You can guarantee no changes to the original repository because all changes are done elsewhere. A single update also updates all your services down the line. Time to brew that coffee and sit back.

References

  1. https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/overview
  2. https://github.com/settings/tokens
  3. https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html

Written by

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store