Salesforce CI-CD: Building Blocks

At our Company, we use both Gitlab and Jenkins using docker images for workers. We use package based development process for Salesforce development.

In this article, I will not discuss our company's implementation of CI-CD.

I will instead discuss about the steps - the building blocks you could use to build pipeline according to your need. Gitlab and Jenkins can invoke shell scripts - and we used this in our pipeline. So I am sure you could too.

As a bonus, I would also share at the end

our project's definition file to show how to enable OmniChannel
docker image used by our Gitlab Runner and Jenkins Slave.

If you need a ready made pipeline, you could use https://gitlab.com/sfdx/sfdx-cicd-template.

If you need to build your own pipeline according to your requirements - you can carry on reading.

Warning: It is a long and boring read.

I will be using bash shell on mac OS. But I am pretty sure sfdx is shell neutral. So commands will work for Windows, linux etc too.

Prerequisite

Knowledge of Salesforce
Knowledge of shell script.
Knowledge of Gitlab is useful, but not required.
Able to connect sfdx with your Org
This is a follow up of our implementation of Salesforce CICD.
Know how to setup gitlab with a docker runner.

What is discussed here

Setup a connected app for CI-CD to work
Setup docker image with SFDX and required tools installed
Connect to Salesforce Org from inside docker container for testing
Decide on Salesforce CI-CD steps and stages
Discuss about commands

Some points to note

sfdx is command line to use Salesforce. Just like "git" is command line to use ... git.
DevHub Org is main account for your company. You need this to create ScratchOrg or Sandbox
Sandbox is a long live Org. It doesn't die. It can be derived from DevHub ... i.e. is a replica of DevHub when it is created
ScratchOrg is a short live Org. It has a deadline ... max is 30 days. Default is 3 days.
Username is used by sfdx to identify a connection to an Org. We can also set Alias for a connection to an Org.
sfdx command use connection whose username matches set 'defaultusername' property when it needs Org.
sfdx command use connection whose username matches set 'defaultdevhubusername' property when it needs devhub.
Commands I use have variables declared with $. e.g. $variable. In shell script, these are environment variables to configure CI-CD scripts.

How I proceed

This is not an example of pipeline. I just want to mention the steps you might

Setup a Salesforce Connected app for CI-CD

Salesforce is a cloud based software. Your account on a Salesforce Org is a DevHub account.

Connect by web

So you would connect your sfdx to Salesforce account through command

sfdx force:auth:web:login -r <your salesforce Org URL>

This opens up login page. Enter your ID and Password. And your sfdx is connected.

Simple.

Connect by connected-app

But CI-CD is an automated process and cannot enter it's ID and Password on login page. (you can do that by using automation like selenium, but we are not talking about that here). So we need a way for CI-CD to connect to Salesforce Org without a manual intervention. We need to bypass login.

I know you are thinking about OAuth. Yes. We need to use OAuth to connect to Salesforce.

Salesforce connected app to rescue. Salesforce Trailhead beautifully explains how to use connected-app to connect to an Org.

Please follow these steps:

Note: Do all this on Production Org. Then you can refresh it into other environments/Orgs.

Create a CI-CD user. Create a CI-CD permission set and assign to CI-CD user.
Enable Dev Hub and Create a Connected App.

Name this Connected App as CI-CD so that everyone knows.
In section "Create a Salesforce Connected App", Step 10, where you are required to manage profiles. You have ability to select Permission Set. Please assign CI-CD permission set here. This will allow CI-CD user to use this connected app we are making.

Create Private Key and Digital Certificate.

Please follow this upto (including) "Add Digital Certificate to Your Connected App" section.

Out of this, we need these items

CI-CD user ID/username (looks like an email - from Step 1)
Consumer Key (From Step 2)
Public Key as a counterpart of Certificate used in connected app (From second trailhead article)

Now let us test connectivity to connected app. This is important. If this doesn't work, we can't proceed.

sfdx force:auth:jwt:grant --setdefaultdevhubusername --clientid $CONSUMER_KEY --jwtkeyfile $PATH_TO_PUBLIC_KEY --username $USERNAME

This should work successfully - no errors. Exit code 0.

If you are not sure that your script exits when error, you could test exit code 0.

I use this bash function to test exit code 0.

_validate_exit_status() {
exit_status=$?
if [ ${exit_status} -ne 0 ]; then
echo "We have error when executing previous command. Status code: ${exit_status}."
exit "${exit_status}"
fi
}

Double connection always work

You connected to an Org through sfdx force:auth:web:login.
You try to connect through consumer key

This will cause second connection to pass.

So you need to disconnect first, and then try command above.

How do you disconnect?

I don't have a good answer. There is a folder at ~/.sfdx. In this folder are JSON files for each sfdx connection. Pick your file with mentioned username, and move it out. This disconnects.

I would suggest take a backup, so that you can swap it back after testing connection.

Building Blocks

Connect to Org

sfdx force:auth:jwt:grant --setdefaultdevhubusername --clientid $CONSUMER_KEY --jwtkeyfile $PATH_TO_PUBLIC_KEY --username $USERNAME

Check limits

Salesforce is a cloud software. To avoid over runs, Salesforce applies limits to every Org. Our company's Org has limits of 100 active ScratchOrg, 200 daily scratch org, 500 package creations etc.

So it is important to know the limits before every CI-CD run.

sfdx force:limits:api:display -u $SF_USERNAME | grep 'NAME\|────\|ActiveScratchOrgs\|DailyScratchOrgs\|Package2VersionCreates'

List connected environments

This command lists all connected Orgs.

sfdx force:org:list

To know in detail about an Org

sfdx force:org:display

Create ScratchOrg

and connect to it.

sfdx force:org:create --setdefaultusername --definitionfile $DEFINITION_FILE_PATH --wait $SFDX_TIMEOUT --durationdays 1 --setalias $SCRATCH_ORG_NAME

--setdefaultusername --> makes this scratch org as default org for sfdx. all subsequent sfdx commands will run for this org
--definitionsfile --> definitions file to be used when creating ScratchOrg. I shall discuss about it at end.
--wait --> sfdx wait timeout
--durationdays --> number of days this scratch org shall live. Default is 3. In CI-CD we create scratch org, use it, and destroy it. But in case of failure, we do not want it to linger around eating limits. So we mark it for autodelete in 1 day.
--setalias --> set an alias for this scratch org ... just for identification

Connect to Sandbox

We can connect to Sandbox only after connecting to DevHub. I am not sure if direct connection works.

sfdx force:auth:jwt:grant --setdefaultdevhubusername --setdefaultusername --clientid $CONSUMER_KEY --jwtkeyfile $PATH_TO_PUBLIC_KEY --username $USERNAME --instanceurl https://test.salesforce.com --setalias SIT

--setdefaultdevhubusername --> Mark it as default devhub for subsequent sfdx commands

--setdefaultusername --> Mark is as default username for subsequent sfdx commands

-- clientid --> This is connected app's consumer key. This is same as DevHub org if Sandbox was derived from DevHub after connected app creation. But if connected app was created after Sandbox was created, then connected app doesnt exist in Sandbox. So you might need to create one. And then use it's consumer key here. My suggestion is to create Sandbox only after creating connected app in DevHub.

--jwtkeyfile --> Path of public key

--username --> CI-CD user's name to use to connect to Sandbox. Make sure this user exists in Sandbox, and has permission set properly assigned to connect to connected app. If not, then create a new user.

--instanceurl --> Just copy-paste it. I had thought that this is just a placeholder, and used my sandbox's url, DevHub's url, but it didn't work.

--setalias --> An alias for Sandbox

Run LWC Tests

Before running LWC tests, we need to setup. On your local machine, this is one time step. But depending on your CI-CD setup, you may need to do it everytime.

In our case, for every run, a docker image is generated. Hence we do it every time.

sfdx force:lightning:lwc:test:setup

And to run actual tests

sfdx force:lightning:lwc:test:run

Deploy Code

sfdx force:source:push

This will deploy code to current default username. To select a specific org where you want to deploy code; use -u flag and mention usesrname/alias.

Run Apex Tests

You need to connect to DevHub Org first.

Note: Test is run on Apex code that is deployed on target Org. And not the code in codebase. Hence before running a test, code must be deployed in target Org - either as code deploy, or as package deploy.

You could either run all the tests in org. Or run only specific suite.

If you are running test in a common sandbox that has code for multiple projects, you might not want to test all the unit tests - of other teams.

Run all tests

sfdx force:apex:test:run --wait 10--resultformat human --codecoverage --testlevel RunLocalTests

Run only your suite

sfdx force:apex:test:run --wait 10--resultformat human --codecoverage --suitenames $TEST_SUITE_NAME

--wait --> sfdx timeout

--resultformat --> I am a human, and I want results in my format. So 'human'.

-- codecoverage --> Calculate code coverage of tests. If it falls below a certain threshold, tests are considered to fail.

-- testlevel --> test level to run

-- suitenames --> list of suites to be run

Test suites are defined in meta xml as

<?xml version="1.0" encoding="UTF-8"?>
<ApexTestSuite xmlns="http://soap.sforce.com/2006/04/metadata">
<testClassName>MyTestClass</testClassName>
</ApexTestSuite>

This file takes multiple test classes that form part of suite. Suite name is file name.

Packaging

In Salesforce, package is a deployment artifact. e.g. a jar file, or an exe file.

Package development process is suggested by Salesforce. So every team creates code and bundles them in a package. This package is tested, promoted and deployed on production. It can be backed out in case of issues.

Only promoted package can be deployed on production. Normal package is like beta version. Promoted package is released version.

Package is identified by PackageId that looks like '0Ho'.

Package version creation results in a PackageVersionID that looks like '04t'. This package version id is used in CI-CD pipeline.

First package

sfdx-project.json

This is an important file. It is unique for every project.

When a package is created, this file is updated with package information. When a new package version is created, this file is updated.

sfdx force:package:create --name $PACKAGE_NAME --packagetype Unlocked --path \"force-app\" --targetdevhubusername $SF_USERNAME

This command must be run before we begin package development.

Create package version

This command increments package version by 1.

sfdx force:package:version:create --package $PACKAGE_NAME -d force-app -k $PACKAGE_KEY --wait $SFDX_TIMEOUT --json -f config/project-scratch-cicd-def.json --tag $GIT_SHA

Package created is in beta stage. Salesforce maintains versioning.

It has provision to mention a package version - but in this case, you need to maintain versioning. i.e. increasing versions.

Also note that we are tagging this version with Git SHA. This is because pipeline is always run on a git commit. And this commit is identified by Git SHA. So we may not want to create multiple packages for same Git SHA.

sfdx force:package:version:create:list --createdlastdays 7 --json | jq ".result[] | select(.Tag==\"$GIT_SHA\") | .SubscriberPackageVersionId" -r

Note: I am using jq to parse and get a field from JSON.

Using created last days as we identified that in our project we may not need to look for versions more than 7 days ago in our pipeline.

Display package version

During pipeline, we may want to print package version in human readable format.

Pipeline itself uses PackageVersionId which is non human readable.

Note: package1 in command below is not a typo.

sfdx force:package1:version:display -i $PACKAGE_VERSION_ID -u $SF_USERNAME

Deploy package version on an Org

sfdx force:package:install --package $PACKAGE_VERSION_ID --wait $SFDX_TIMEOUT --publishwait $SFDX_TIMEOUT -k $PACKAGE_KEY --noprompt

This will deploy given package version id to defaultusername org. Or use a -u to specify a target org.

Promote package

sfdx force:package:version:promote -p $PACKAGE_VERSION_ID -n

This promotes package so that it can be installed on production org.

Definitions file

This file is used to create a ScratchOrg. You may create different ScratchOrg for developers, and different one for CI-CD.

Also in this you can tell Salesforce what features you want in your ORG.

e.g. in our ORG we want Omni channel enabled.

Our project's definition file looks like

{
  "orgName": "MyCompany CI-CD",
  "edition": "Enterprise",
  "features": [],
  "settings": {
    "lightningExperienceSettings": {
        "enableS1DesktopEnabled": true
    },
    "securitySettings": {
        "passwordPolicies": {
            "enableSetPasswordInApi": true
        }
    },
    "mobileSettings": {
        "enableS1EncryptedStoragePref2": false
    },
    "omniChannelSettings": {
      "enableOmniChannel": true,
      "enableOmniSkillsRouting":true
    }
  }
}

Docker Image

Dockerfile:

FROM artifactory.mycompany.com:8443/node-base:1.1.0
USER root
RUN mkdir -m 777 /assets /src
COPY config/* /assets/
COPY bin/* /usr/local/bin/
RUN apk add --update jq git openssl
RUN npm install sfdx-cli --global
ENV SFDX_AUTOUPDATE_DISABLE=false \
SFDX_USE_GENERIC_UNIX_KEYCHAIN=true \
SFDX_DOMAIN_RETRY=600 \
SFDX_LOG_LEVEL=DEBUG

USER www
CMD ["bash"]

We faced a lot of permission issues which we resolved. but I would suggest you to remove 'USER www' if you face permission issue. Since these images will be created and destroyed, we should be okay.

We placed our bash scripts in bin folder.

We placed public certificate to connect to Salesforce connected-app in assets folder.

Our base image is just latest nodejs. We use npm to install sfdx.

Harsh's Notes

Search This Blog