Linked-in profile Github profile Twitter profile Curriculum vitae
Categories: Automation Drupal
Created
30th May 2019
Modified
11th July 2020

Continuous Delivery, with Jenkins declarative pipelines, Composer, and Drupal 8+

WordPress, and Drupal 7 support coming at some point…

Nobody wants to deploy manually anymore, besides, why would you want to with the plethora of tools available to us?

Recently I was given the opportunity to move several Drupal projects into automation. Each environment is a little different, be it infrastructure, or operating system. But at least the CMS is the same, along with my options to interface with the sites, a command line.

I also need to be able to track project specific things to setup and do, like tool chains for style compilation, dependencies, and tests.

Anyway, long story short, I settled with Jenkins and it’s declarative pipelines system to carry out the heavy-lifting. With Jenkins, I can keep job pipeline steps in VCS. Now I just need something to orchestrate this process. Alas, I couldn’t find what I was looking for! So I decided to just throw together the exact same manual commands I use every day, and fine tune them for speed and fail-over.

Fine-tuning

I wanted to keep the deployments as short as possible, and reduce down-times to an absolute minimum. Not only was I able to reduce the down time, I found a way of avoiding maintenance mode altogether, and instead use the read-only module, along with a Blue-Green software-based deployment process, where the new build was setup along side the active build, and simplying updating symlinks.

Security

There is no requirement for passwords to be stored within the JWSA codebase for your projects. Only the path to a GPG key for use with SSH authentication, and the reliance on a deployment user environment that provides credentials for the scripts to connect to databases, such as a .my.cnf file for the database, in the user’s workspace – remember not to dial those permissions over 600 😉.

JWSA

With all this on my mind, I couldn’t be bothered with a fancy name, so Jenkins Web Scripts by Alex (JWSA) it is!

The scripts are available on GitHub for you to use, these are being actively developed, so keeps eyes open for more features. Or feel free to contribute https://github.com/AlexanderGW/jwsa

Note: The README on the repository may be more upto date than this post!

The build script

The build script intends to setup and bootstrap the entire Drupal site workspace, on the Jenkins server. With an exisiting database, either copied from the destination server, or default to using whatever is available, as defined in the environment file.

# These environment variables are the format used in Jenkins Groovy pipelines
build.sh ${env.JOB_NAME} ${env.WORKSPACE} /path/to/${env.JOB_NAME}/.env

Input

Output

The deploy script

The deploy script will copy the entire Jenkins workspace to the specified paths, on the destination server. Deployments are over SCP or RSYNC, so there is no need to install a VCS, Node.js, Composer etc on the destination server, just a LAMP stack, PHP extensions required by Drupal 8, and credentials for the deployment user, see Project files.

# These environment variables are the format used in Jenkins Groovy pipelines
deploy.sh ${env.JOB_NAME} ${env.WORKSPACE} ${env.BUILD_ID} /path/to/${env.JOB_NAME}/.env

Input

Output

The JWSA .env

# Get last successful Jenkins build number for the project
LAST_BUILD_ID=`curl http://jenkins.test:8080/job/$PROJECT_NAME/lastSuccessfulBuild/buildNumber`

Project files

An example of a JWSA project, can be found at ~/project/example-test

A project variable.sh file

Here are all possible variables you can define for a JWSA project environment.

# CMS type (drupal8 only. Coming soon; drupal7, wordpress)
TYPE='drupal8'
# Webserver environment (apache, nginx)
WEBSERVER='nginx'
# Destination host
DEST_HOST='deploy.test';
# REMEMBER: SSH keys need to be manually installed on the Jenkins deployment server.
DEST_IDENTITY='/var/lib/jenkins/.ssh/deploy.test.pem';
# Base deployment path
DEST_PATH="/var/deploy/$PROJECT_NAME"
# Product database dump location for reversion
DEST_DUMP_PATH="$DEST_PATH/backup"
# Project site assets (such as Drupal public files)
DEST_ASSET_PATH="$DEST_PATH/files"
# Location of the deployed builds
DEST_BUILDS_PATH="$DEST_PATH/build"
# Location of this build
DEST_BUILD_PATH="$DEST_BUILDS_PATH/$BUILD_ID"
# Drupal's Twig storage path
DEST_STORAGE_PATH="$DEST_PATH/storage"
# Location of this build
DEST_PRIVATE_PATH="$DEST_PATH/private"
# The SSH user to connect to the deploy environment
DEST_SSH_USER='root'
# The web server user for project ownership & permissions
DEST_WEB_USER='www-data'
# Local database
SRC_DATABASE_NAME=$PROJECT_NAME
# Deployment database (this is suffixed in 'prod' environments. with incremental job numbers "__n")
DEST_DATABASE_NAME=$PROJECT_NAME
# Webroot symlink, that will link to the active build
DEST_WEBROOT_PATH="/var/www/$PROJECT_NAME"
# Drupal public files path in the build, this is linked to the project's main files directory
DEST_BUILD_ASSETS_PATH="$DEST_BUILD_PATH/webroot/sites/default/files"
# Drupal settings location
DEST_BUILD_SETTINGS_PATH="$DEST_BUILD_PATH/webroot/sites/default/settings.php"
# CLI tool (Drush, WP-CLI)
CLI_PHAR="vendor/bin/drush"
# Services to restart (used at various steps in the deployment process)
declare -a DEST_SERVICES_RESTART=("php-fpm")
# Services to reload (used at various steps in the deployment process)
declare -a DEST_SERVICES_RELOAD=("nginx")
# Database table data to ignore, on data dumps.
declare -a DATABASE_TABLE_NO_DATA=(
"cache"
"cache_bootstrap"
"cache_block"
"cache_config"
"cache_field"
"cache_menu"
"cache_tags"
"cache_toolbar"
"cache_views_info"
"watchdog"
)
# Commands to execute before the platform step (The Drupal specific parts) of the build process.
declare -a BUILD_CMDS_PRE_PLATFORM=(
"echo \"hi\""
)
# Rsync flags and parameters
# REMEMBER: Exclude .env, assets, cache, test, and tool directories to speed up transfers
# MUST: Suffix with the with "-e" flag, to allow succeeding text to be executed remotely.
RSYNC_FLAGS="-al --stats --delete-before --exclude=.env --exclude=.git --exclude=.sass-cache --exclude=node_modules --exclude=simpletest --exclude=tests --exclude=/storage --exclude=/private --exclude=/web/sites/default/files -e"
# SSH connection string
SSH_CONN="ssh -i $DEST_IDENTITY $DEST_SSH_USER@$DEST_HOST"

A project .env file

Variables injected into the Drupal environment, specifically the settings.php file

# The type of environment we're working with (local, dev, test, prod. etc..)
JWSA_APP_ENV=dev

# Database credentials
JWSA_SQL_HOSTNAME=localhost
JWSA_SQL_DATABASE=dbname
JWSA_SQL_PASSWORD=123
JWSA_SQL_USER=dbuser
JWSA_SQL_PORT=3306

# Drupal bits
JWSA_HASH_SALT=your-salt-here
JWSA_CONFIG_SYNC_PATH=../config/sync
JWSA_PRIVATE_PATH=../private
JWSA_TWIG_PHP_STORAGE_PATH=../storage

A Drupal 8 project settings.php

<?php

// We have a JWSA environment.
if (($env = getenv('JWSA_APP_ENV')) !== FALSE) {

	// Drupal hash salt.
	if (($val = getenv('JWSA_HASH_SALT')) !== FALSE) {
		$settings['hash_salt'] = $val;
	}

	// Drupal YAML config sync path.
	if (($val = getenv('JWSA_CONFIG_SYNC_PATH')) !== FALSE) {
		$config_directories['sync'] = $val;

		// Or maybe something like this... (if you're looking for an alternative to 'configuration split')
//		$config_directories['sync'] = '../config/' . $env . '/sync';

		// Or stick to using ../config/sync with production configs. Overwriting config variables using environment settings files below.
	}

	// Drupal private assets path.
	if (($val = getenv('JWSA_PRIVATE_PATH')) !== FALSE) {
		$settings['file_private_path'] = $val;
	}

	// Drupal TWIG template storage (good for AWS S3 bucket use-cases for public and/or private assets)
	if (($val = getenv('JWSA_TWIG_PHP_STORAGE_PATH')) !== FALSE) {
		$settings['php_storage']['twig']['directory'] = $val;
	}

	// Drupal default database.
	$databases['default']['default'] = [
		'database' => getenv('JWSA_SQL_DATABASE'),
		'driver' => 'mysql',
		'host' => getenv('JWSA_SQL_HOSTNAME'),
		'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
		'password' => getenv('JWSA_SQL_PASSWORD'),
		'port' => getenv('JWSA_SQL_PORT'),
		'prefix' => '',
		'username' => getenv('JWSA_SQL_USER'),
	];

	// Get any environment specific settings.
	$file = $app_root . '/' . $site_path . '/settings.' . $env . '.php';
	if (file_exists($file)) {
		include $file;
	}
}

A project Jenkinsfile

Here is a really basic pipeline for a project. for the build and deploy scripts

#!/usr/bin/env groovy

pipeline {
	agent any

	stages {
		stage('Composer') {
			steps {
				sh 'composer install --no-interaction'
			}
		}

		stage('NPM') {
			steps {
			    sh 'npm install'
			}
		}

		stage('Grunt') {
			steps {
			    sh 'grunt foobar'
			}
		}

		stage('Build') {
			steps {
				sh "/path/to/jwsa/build.sh ${env.JOB_NAME} ${env.WORKSPACE} /path/to/env/${env.JOB_NAME}/.env"
			}
		}

		stage('Test') {
			steps {
				sh 'vendor/bin/phpunit foobar'
			}
		}

		stage("Deploy") {
			steps {
				sh "/path/to/jwsa/deploy.sh ${env.JOB_NAME} ${env.WORKSPACE} ${env.BUILD_ID} /path/to/envs/${env.JOB_NAME}/.env"
			}
		}
	}
}

Leave a Reply

Your email address will not be published. Required fields are marked *