Wednesday, 23 September 2020

Piping Software for Less: Why, What & How (Part 1)

Developing software is hard and all good developers are lazy. This is one of the reasons we have tools which automate practices like continuous integration, static analysis and measuring test coverage. The practices help us to measure quality and find problems with code early. When you measure something you can make it better. Automation makes it easy to perform the practices and means that lazy developers are likely to perform them more often, especially if they’re automatically performed every time the developer checks code in.

This is old news. These practices have been around for more than twenty years. They have become industry standards and not using them is, quite rightly, frowned upon. What is relatively new is the introduction of cloud based services such as BitBucket Pipelines, CircleCI and SonarCloud which allow you to set up these practices in minutes, however with this flexibility and efficiency comes a cost.

Why

While BitBucket Pipelines, CircleCI and SonarCloud have free tiers there are limits.

With BitBucket Pipelines you only get 50 build minutes a month on the free tier. The next step up is $15/month and then you get 2500 build minutes.

On the free CircleCI tier you get 2500 free credits per week, but you can only use public repositories, which means anyone and everyone can see your code. The use of private repositories starts at $15 per month.

With SonarCloud you can analyse as many lines of code as you like, but again you have to have your code in a public repository or pay $10 per month for the first 100,000 lines of code.

If you want continuous integration and a static analysis repository which includes test coverage and you need to keep your source code private, you’re looking at a minimum of $15 per month for these cloud based solutions and that’s if you can manage with only 50 build minutes per month. If you can’t it’s more likely to be $30 per month, that’s $360 per year.

That’s not a lot of money for a large software company or even a well funded startup or SME, though as the number of users goes up so does that price. For a personal project it’s a lot of money. 

Cost isn’t the only drawback, with these approaches you can lose some flexibility as well. 

The alternative is to build your own development pipelines. 

I bet you’re thinking that setting up these tools from scratch is a royal pain in the arse and will take hours; when the cloud solutions can be set up in minutes. Not to mention running and managing your own pipeline on your personal machine and don’t they suck resources when they’re running in the background all the time? And shouldn’t they be set up on isolated machines? What if I told you, you could set all of this up in about an hour and turn it all on and off as necessary with a single command? And if you wanted to, you could run it all on a DigitalOcean Droplet for around $20 per month. 

Interested? Read on.

What

When you know how, setting up a continuous integration server such as Jenkins and a static analysis repository such as SonarQube in a Docker container is relatively straightforward. As is starting and stopping them altogether using Docker Compose. As I said, the key is knowing how; and what I explain in the rest of this article is the product of around twenty development hours, a lot of which was banging my head against a number of individual issues which turned out to have really simple solutions.

Docker

Docker is a way of encapsulating software in a container. Anything from an entire operating system such as Ubuntu to a simple tool such as the scanner for SonarQube. The configuration of the container is detailed in a Dockerfile and Docker uses Dockerfiles to build, start and stop containers. Jenkins and SonarQube all have publically available Docker images, which we’ll use with a few relatively minor modifications, to build a development pipeline.

Docker Compose

Docker Compose is a tool which orchestrates Docker containers. Via a simple YML file it is possible to start and stop multiple Docker containers with a single command. This means that once configured we can start and stop the entire development pipeline so that it is only running when we need it or, via a tool such as Terraform, construct and provision a DigitalOcean droplet (or AWS service, etc.) with a few simple commands and tear it down again just as easily so that it only incurs cost when we’re actually developing. Terraform and DigitalOcean are beyond the scope of this article, but I plan to cover them in the near future. 

See the Docker and Docker Compose websites for instructions on how to install them for your operating system.

How

In order to focus on the development pipeline configuration, Over this and a few other posts I’ll describe how to create an extremely simple Dotnet Core class library with a very basic test and describe in more detail how to configure and run Jenkins and SonarQube Docker containers and setup simple projects in both to demonstrate the pipeline. I’ll also describe how to orchestrate the containers with Docker Compose. 

I’m using Dotnet Core because that’s what I’m working with on a daily basis. The development pipeline can also be used with Java, Node, TypeScript or any other of the supported languages. Dotnet Core is also free to install and use on Windows, Linux and Mac which means that anyone can follow along.

A Simple Dotnet Core Class Library Project

I’ve chosen to use a class library project as an example for two reasons. It means that I can easily use a separate project for the tests, which allows me to describe the development pipeline more iteratively. It also means that I can use it as the groundwork for a future article which introduces the NuGet server Baget to the development pipeline.

Open a command prompt and start off by creating an empty directory and moving into it.

mkdir messagelib
cd messagelib

Then open the directory in your favorite IDE, I like VSCode for this sort of project. Add a Dotnet Core appropriate .gitignore file and then create a solution and a class library project and add it to the solution:

dotnet new sln
dotnet new classLib --name Messagelib
dotnet sln add Messagelib/Messagelib.csproj

Delete MessageLib/class1.cs and create a new class file and class called Message:

using System;

namespace Messagelib
{
    public class Message
    {
        public string Deliver()
        {
            return "Hello, World!";
        }
    }
}

Make sure it builds with:

dotnet build

Commit the solution to a public git repository or you can use the existing one in my bitbucket account here: https://bitbucket.org/findmytea/messagelib

A public repository keeps this example simple and although I won’t cover it here, it’s quite straightforward to add a key to a BitBucket or GitHub private repository and to Jenkins so that it can access them.

Remember that one of the main driving forces for setting up the development pipeline is to allow the use of private repositories without having to incur unnecessary cost.


Read the next parts here:




Sidebar 1


Continuous Integration 

Continuous Integration (CI) is a development practice where developers integrate code into a shared repository frequently, preferably several times a day. Each integration can then be verified by an automated build and automated tests. While automated testing is not strictly part of CI it is typically implied.


Static Analysis

Static (code) analysis is a method of debugging by examining source code before a program is run. It’s done by analyzing a set of code against a set (or multiple sets) of coding rules.


Measuring Code Coverage

Code coverage is a metric that can help you understand how much of your source is tested. It's a very useful metric that can help you assess the quality of your test suite.



Sidebar 2: CircleCI Credits

Credits are used to pay for your team’s usage based on machine type and size, and premium features like Docker layer caching.



Sidebar 3: What is Terraform?

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.

Configuration files describe to Terraform the components needed to run a single application or your entire datacenter. Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.

The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc.


Piping Software for Less: Jenkins (Part 2)



Run Jenkins in Docker with Docker Compose

Why use Jenkins I hear you ask? Well, for me the answers are simple: familiarity and the availability of an existing tested and officially supported Docker image. I have been using Jenkins for as long as I can remember. 

The official image is here: https://hub.docker.com/r/jenkins/jenkins

After getting Jenkins up and running in the container we’ll look at creating a ‘Pipeline’ with the Docker Pipeline plugin. Jenkins supports lots of different ‘Items’, which used to be called ‘Jobs’, but Docker can be used to encapsulate build and test environments as well. In fact this is what BitBucket Pipelines and CircleCI also do.

To run Jenkins Pipeline we need a Jenkins installation with Docker installed. The easiest way to do this is to use the existing Jenkins Docker image from Docker Hub. Open a new command prompt and create a new directory for the development pipeline configuration and a sub directory called Jenkins with the following Dockerfile in it: 

FROM jenkins/jenkins:lts

USER root
RUN apt-get update
RUN apt-get -y install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -

RUN add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/debian \
   $(lsb_release -cs) \
   stable"

RUN apt-get update
RUN apt-get install -y docker-ce docker-ce-cli containerd.io
RUN service docker start

# drop back to the regular jenkins user - good practice
USER jenkins

You can see that our Dockerfile imports the existing Jenkins Docker image and then installs Docker for Linux. The Jenkins image, like most Docker images, is based on a Linux base image.

To get Docker Compose to build and run the image, we need a simple docker-compose.yml file in the root of the development pipeline directory with the details of the Jenkins service:

version: '3'
services:
  jenkins:
    container_name: jenkins
    build: ./jenkins/
    ports:
      - "8080:8080"
      - "5000:5000"
    volumes:
        - ~/.jenkins:/var/jenkins_home
        - /var/run/docker.sock:/var/run/docker.sock

Note the build parameter which references a sub directory where the Jenkins Dockerfile should be located. Also note the volumes. We want the builds to persist even if the container does not, so create a .jenkins directory in your home directory:

mkdir ~/.jenkins

Specifying it as a volume in docker-compse.yml tells the Docker image to write anything which Jenkins writes to /var/jenkins_home in the container to ~/.jenkins on the host - your local machine. If the development pipeline is running on a DigitalOcean droplet, DigitalOcean Volumes can be used to persist the volumes even after the droplet is torn down.

As well as running Jenkins in a Docker container we’ll also be doing our build and running our tests in a Docker container. Docker doesn’t generally like being run in a Docker container itself, so by specifying /var/run/docker.sock as a volume, the Jenkins container and the test container can be run on the same Docker instance.

To run Jenkins, simply bring it up with Docker compose:

docker-compose up

(To stop it again just use ctrl+c)

Make sure the first time you note down the default password. It will appear in the log like this:

Jenkins initial setup is required. An admin user has been created and a password generated.

Please use the following password to proceed to installation:

<password>

This may also be found at: /var/jenkins_home/secrets/initialAdminPasswor

To configure Jenkins for the first time open a browser and navigate to:

http://localhost:8080

Then:

  1. Paste in the default password and click continue.
  2. Install the recommended plugins. This will take a few minutes. There is another plugin we need too which can be installed afterwards.
  3. Create the first admin user and click Save & Continue.
  4. Confirm the Jenkins url and click Save & Finish.
  5. Click Start Jenkins to start Jenkins.

You now have Jenkins up and running locally in a Docker container! 

  1. To use Docker pipelines in Jenkins we need to install the plugin. To do this:
  2. Select Manage Jenkins from the left hand menu, followed by Manage Plugins.
  3. Select the ‘Available’ tab, search for ‘Docker Pipeline’ and select it,
  4. Click ‘Download now and install after restart’. 
  5. On the next page put a tick in the ‘restart after download’ check box and wait for the installation and for Jenkins to restart. Then log in again.

Next we need to create the Docker Pipeline for the Messagelib solution. 

  1. Select ‘New Item’ from the left hand menu, enter ‘Messagelib’ as the name, select ‘Pipeline’ and click ok.
  2. Scroll to the ‘Pipeline’ section and select ‘Pipeline script from SCM’ from the ‘Definition’ dropdown. This is because we’re going to define our pipeline in a file in the Messagelib solution. 
  3. From the ‘SCM’ dropdown, select ‘Git’ and enter the repository URL of the Messagelib solution. 
  4. Then click Save.


Jenkins is now configured to run the Messagelib pipeline, but we need to tell it what to do by adding a text file called Jenkinsfile to the root of the Messagelib solution.

/* groovylint-disable CompileStatic, GStringExpressionWithinString, LineLength */

pipeline
{
    agent
    {
        docker { image 'pjgrenyer/dotnet-build-sonarscanner:latest' }
    }
    stages
    {
        stage('Build & Test')
        {
            steps
            {
                sh 'dotnet clean'
                sh 'dotnet restore'
                sh 'dotnet build'
            }
        }
    }
}

This very simple Groovy script tells the Jenkins pipeline to get the latest ‘dotnet-build-sonarscanner’ Docker image and then use it to clean, restore and build the dotnet project. ‘dotnet-build-sonarscanner’ is a Docker image I built and pushed to Docker Hub using the following Dockerfile:

FROM mcr.microsoft.com/dotnet/core/sdk:latest AS build-env
WORKDIR /
RUN apt update
RUN apt install -y default-jre
ARG dotnet_cli_home_arg=/tmp
ENV DOTNET_CLI_HOME=$dotnet_cli_home_arg
ENV DOTNET_CLI_TELEMETRY_OPTOUT=1
ENV PATH="${DOTNET_CLI_HOME}/.dotnet/tools:${PATH}"
ENV HOME=${DOTNET_CLI_HOME}
RUN dotnet tool install --global dotnet-sonarscanner
RUN chmod 777 -R ${dotnet_cli_home_arg}

This creates and configures a development environment for Dotnet Core and Sonar Scanner, which requires Java. 

There is a way to use the Dockerfile directly, rather than getting it from Docker Hub, described here: https://www.jenkins.io/doc/book/pipeline/docker/

Once the Jenkins file is added to the project and committed, set the build off by clicking ‘Build now’ from the left hand menu of the MessageLib item. The first run will take a little while as the Docker image is pulled (or built). Future runs won’t have to do that and will be quicker. You should find that once the image is downloaded, the project is built quickly and Jenkins shows success.


Read the next parts here:


Part 2: Piping Software for Less: Jenkins (Part 2)
Part 3: Piping Software for Less: SonarQube (Part 2)