Jenkins Configuration

Jenkins Standardisation: This is the root for all documentation relating to the standardisation of Jenkins build server.

Jenkins Load Testing

Jenkins Master Setup

Jenkins Packaging

Jobs Standardisation

Plugins Standardisation

Slave Setup

User Experience Standardisation

Jenkins Load Testing

Section for all Jenkins Load Testing Contents.

Work In Progress

This section is under active development.

Content

Benchmarking servers

Load Test Strategy, Scenarios and Results

LikeBe the first to like this

jenkins load testing infrastructure requirements scenarios solution development stress benchmark 

Jenkins Master Setup

Hardware Requirements

TBC after load testing

Software Requirements

Sun Java 6

Latest released Jenkins packaged by Abccompany Build Team 

Setup Process

Install latest ss-fat-jenkins-gen & ss-fat-jenkins-plugins-gen RPMs from RHN/Spacewalk:

yum install ss-fat-jenkins-gen ss-fat-jenkins-plugins-gen

Upgrade Process

Upgrade to latest ss-fat-jenkins-gen & ss-fat-jenkins-plugins-gen RPMs from RHN/Spacewalk:

yum update ss-fat-jenkins-gen ss-fat-jenkins-plugins-gen

Securing the Jenkins server

Abccompany Internal CA

The security standard is to use Specsaver internal CA for securing internal resources (cost effectiveness). However, on Linux these CA certificates are not distributed & installed in browsers. In order to stop getting certificate warnings, please follow the instructions here.

As passwords are used to log in to the Jenkins server (authentication with Active Directory as per standard detailed below), the Jenkins master must only be accessible over SSL. In order to do this in an efficient & easy to configure way, we standardise on the use of Apache httpd to provide the SSL configuration & proxy requests to the Jenkins master. Apache httpd must do the following:

Bind to port 80 & 443

Send a client side redirect (permanent) all requests sent to port 80 to port 443

Send a client side redirect to an FQDN if the master is requested using only a hostname (e.g. redirect all requests to https://buildmanager to https://buildmanager.uk.abccompany.com)

Be configured to serve the full certificate path all the way back to the root CA

The Jenkins manager certificate should be created with both the short name & the FQDN as CN & subjectAltName to allow both to be used without certificate errors in the browser.

Apache config can be based on the following (should go in /etc/httpd/conf.d/buildmanager.conf). Remember to change the hostnames if required:

<VirtualHost *:80>

    ServerAdmin uk.buildnotifications@abccompany.com

    ServerName buildmanager.uk.abccompany.com

    ServerAlias buildmanager

    ErrorLog logs/buildmanager.uk.abccompany.com-error_log

    CustomLog logs/buildmanager.uk.abccompany.com-access_log common

    RewriteEngine on

    RewriteCond %{HTTPS} off

    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

</VirtualHost>

<VirtualHost *:443>

    ServerAdmin uk.buildnotifications@abccompany.com

    ServerName buildmanager.uk.abccompany.com

    ServerAlias buildmanager

    ErrorLog logs/buildmanager.uk.abccompany.com-error_log

    CustomLog logs/buildmanager.uk.abccompany.com-access_log common

    RewriteEngine on

    ProxyPass         / http://localhost:8080/

    ProxyPassReverse  / http://localhost:8080/

    ProxyRequests     Off

    SSLEngine on

    SSLCertificateFile /etc/httpd/conf/ssl.crt/buildmanager.cert.pem

    SSLCertificateKeyFile /etc/httpd/conf/ssl.key/buildmanager.key.pem.nopass

    SSLCertificateChainFile /etc/httpd/conf/ca-bundle/Specs-CA-Web-chain.cert.pem

</VirtualHost>

Standard Configuration

All configuration must be done after installation. This allows for multiple masters with different configurations. Configurations are checked into SVN after any change (see more details below) so if a new master has to be built as failover, it is simply a case of checking out configurations.

Number of executors

Default setting: 0

No builds should run on master in order to distribute builds to slaves. This keeps the master from being under load & accessible by both users & slaves to keep uptime of this critical piece of the build infrastructure as high as possible.

Default quiet period

Default setting: 5 seconds

Best practice is to ensure that every check-in does not break a build & contains all changes related to changed functionality. A very small quiet period, such as 5 seconds, encourages the practice of a build for each commit. This practice helps quickly identify commits that break builds or fail tests, allowing for quicker turnaround in solving problems.

Define default log rotator value

Days to keep builds: 90

Max # of builds to keep: -1

Days to keep artifacts: -1

Max # of builds to keep with artifacts: 5

These settings keep build logs on Jenkins for 90 days, but only keep artifacts of those builds on the master for 5 days. Logs will be accessible for 90 days. The build retention policy of the Framework Architecture Team is 60 days - this should be catered for by the actual build configuration (following standards) copying artifacts to the disk pool. The disk pool cleanup will cater for the build retention policy. If this setting is set too high, we are likely to hit disk space issues regularly as all builds archive back to the master. This is exactly what the disk pool is for (cheaper storage).

SCM Sync configuration

Never bother me with commit messages: false

Display SCM Sync Status: true

Commit message pattern: [message]

Source control type: Subversion

*Repository URL: http://svn/buildprojects/CI_Tools/Jenkins/trunk/jenkins-config*

This ensures that every change to any configuration file is checked in to Subversion. Displaying the SCM sync status shows an icon at the bottom of the screen showing the status of the SCM sync - whether it is up to date or not. The repository URL will need to be created & only used for one Jenkins instance (obviously) so it doesn't get overridden by another Jenkins instance. Credentials will need to be entered just as with any other SVN build.

Screenshot

The standard configuration is as below in a screenshot (copied from test Jenkins master http://10.4.23.113:8081:

Jenkins Packaging

Jenkins packaging is done via Abccompany strategic packaging methodology (i.e. RPM). The package delivery structure is based on the below concept:

Jenkins Installer

This is the base installer provided by Jenkins. This install contains the actual Jenkins software and it is published in various formats to support industry wide acceptance.

To effectively utilise our current Development & Build Technology stack, leverage the auto-retrieval capabilities of Jenkins installer from a remote repository and deliver Abccompany customised Jenkins solution, we use Maven builds. This provides the ability to enforce standards and uniformity in Jenkins installation on target platforms within Abccompany.

There are several dependencies which the installer require, and using Maven build helps to integrate such dependencies seamlessly.

Jenkins Plugins

This is a collection of approved and supported Jenkins plugins within Abccompany. The list of plugins available via this package are published in Plugins Standardisation page.

Abccompany Customisation

This is collection of various customisations that are required for Jenkins and other peripherals to function effectively in Abccompany environment. This would include contents like log rotate configuration, yum repository configuration, Init v5 script, etc. These customisations are embedded within the Jenkins Installer & Jenkins Plugins RPM as configuration file.

Scripted Configuration

This is a collection of various RPM scripts (pre/post - install, remove) that are packaged within the Jenkins Installer & Jenkins packaging rpm. These scripts follow the guidelines are published in RPM Guidelines

Jobs Standardisation

This is taken from the standard template already existing on BuildManager

The below is the template MAVEN release build

Release build Screenshot

The live link is below

https://buildmanager.uk.abccompany.com/view/Templates/job/Release-Job-Template/configure

The below is the MAVEN metrics build template

Metrics Build Snapshot

The live link is below

https://buildmanager.uk.abccompany.com/view/Templates/job/Metrics-Maven-Template/

Jenkins Jobs should be standardised in the following ways

Template

The Template job should be a disabled job which is copied and then the relevant data filled in

There will be two templates initially, one for metrics builds and one for continuous integration builds

Naming

Jobs will have a standard name in the following format:

The name will start with CI or Metrics followed by a hyphen

The name of the build will be next with hyphens for separators.

The type of build will be next Branch/Trunk.

If a Branch the branch name will follow e.g. Metrics-Blackbox-Trunk, CI-FinlandCompiere-Branch-c303

Retention Time

Jobs are kept for 90 days as per FA Team retention policies.

Artifacts for builds as a convenience will be kept on buildmanager for 5 builds.

In a metrics build we keep 3 builds just for the ability to check errors/debug in builds as "mvn site" can sometimes cause problems.

Labels

Each slave will have labels describing is capabilities including size (standard,large), jvm's (java5, java6, java7) and mysql (mysql5.0, mysql5.1, mysql5.5) installed these can then be used in conjunction to limit which slaves the build can be run on.

Configuration

Due to standardisation of Abccompany' main jobs there should be little need for individual job configuration (local config is the exception).

Because of job copying however a divergence from the template could occur. There should be a regular review of job configuration to ensure compliance with the template and if a change has to be made to the template or a new template created that change should be mirrored in all builds based on that template.

Unless required by a specific build the jdk should be left at default for the node the build is built on.

Timouts should be standardised to the following table

Build Type

Timeout

Notes

Release-Job-Template

60

Metrics

120

Metrics builds need a considerable amount of additional time

History has shown that we should provide at least 1024MB of memory for builds. Specific builds may necessitate more.

We use clean install as it is deemed better to create all the artifacts and only deploy to buildrepo as a publishing action if all the modules pass so we do not get partial build artifacts on a failed build.

Other options can be found on a build by build basis atbuild idiosyncrasies

Tagging

Tag Base URL http://<url of tag directory for project>/${env['BUILD_TAG']}

Tag Comment Tagged by Jenkins svn-tag plugin. Build:${env['BUILD_TAG']}

Tag Delete Comment   Delete old tag by SvnTag Jenkins plugin.

Post build actions

Metrics builds publish there results and call sonar.

Sonar reuses Cobertura results to speed up the process.

as a convenience to the development teams rpms are published to enable quick retrieval as buildrepo renames the rpm and can lead to confusion.

Notifications

Notifications will be done by email to groups set up by the relevant teams as given by example build-notification.uk.abccompany.com

Permissions

Anonymous should have access to read and browse the workspace.

Selected users will have the ability to force, configure and release builds..

Build Triggers

For metrics builds a timed build should be used, preferably between 22:00-06:00 to enable the slaves to be used for CI during business hours.

For normal builds it will be fired by a svn post commit hook and it will also build monthly to ensure completeness of build. This is accomplished through a plugin in the jenkins master and a post commit hook on the svn repository.

Post Build Actions

Continuous Integration Builds

The main post build action will be to publish to the artifact repository the required artifacts.

We will also publish certain artifacts to the front page of the build to allow easy collection for the development teams.

Certain builds may require other actions but this will be on a build by build basis and should be exceptional.

The build will be tagged in the subversion repository.

The notification of the required team should be included here to cope with broken builds

Metrics Builds

publishing of the different metrics will be published here as a convenience to development teams.

Sonar will be ticked

If the build is from a branch Sonar will require the branch name to be input so that the build is unique from the trunk build.

You have to type in seperate maven opts for sonar this should normally be -Xmx1024m to enable enough memory for most builds, this might have to be raised for larger builds.

Deployment

The standard process will be a clean install build, copying the required artifacts back to the master then using the deploy to maven repository post build activity this eliminates the chance of partial deployments of snapshots. Note that some builds are still standardised around a clean deploy and linking to the artifacts this lessens io traffic and is much faster than transferring the artifacts back to the master. (Note this exploits the JNLP performance of migrating artefacts from slave to master).

LikeBe the first to like this

jenkins jobs standartisation Edit Labels

Slave Setup

Node Names

Nodes will be named in lower case the same as the host name they are based on e.g lysander

Location for work space

The work space will be /data/jenkins-slave/ if this does not exist on the slave this will  have to be created. This assumes /data is a separate partition.

The permissions will be set to 755 for the Hudson user.

Executors

There shall be 1 executor per node.

Utilization

Unless the build machine has specific builds dependent on it the standard utilization will be "Utilize this slave as much as possible". This will enable maximum usage of limited slave resources.

Launch Method

On all platforms it will  be jnlp as presently it is much faster to archive artifacts back to master.

When using JNLP, certain configurations available via bash profile are not available and existing free-style builds running Maven will need to be configured to use a specific Maven version otherwise the builds would fail. We would need the information about the impact it will have on builds, or the type of build setup in relation to the launch method selected.

LikeBe the first to like this

jenkins slave Edit Labels

4 Comments (Reverse Order)

 User icon: soumen.trivedi

Unknown User (soumen.trivedi)

Review comments have still not been incorporated. We need more elaborate explanation around why, what & how these are achieved.

In short, do not agree that this page is complete and ready to be signed off. Happy to discuss further if necessary/required.

Creating a Jenkins Slave

Following the kickstarted jenkins master server, the slave options are currently non-functional (TBA at future date).

The following instructions should be followed.

On the slave, install the slave plugin using the below command:

mkdir -p /data/jenkins-slave

yum install RPMNAME --nogpgcheck -y

Icon

If you are using the slave on a build manager server other than the main buildmanager, you must edit the file /etc/sysconfig/jenkins-slave and change JENKINS_URL to your target build manager server, and then restart the service

On the master, go to manage jenkins > manage nodes > new node

Enter the slaves hostname as the node name, and choose dumb slave and click next.

Example screenshot of settings page:

Important points:

No of Executors:1

Remote FS Root: /data/jenkins-slave

Launch method: Launch slave agents via JWS

Availability: Keep this slave on-line as much as possible

After clicking save, run the below on the jenkins slave server:

service jenkins-slave restart

The build manager server should now see the slave as online