Jenkins Configuration
Jenkins Standardisation: This is the root for all documentation
relating to the standardisation of Jenkins build server.
Jenkins Load Testing
Jenkins Master Setup
Jenkins Packaging
Jobs Standardisation
Plugins Standardisation
Slave Setup
User Experience Standardisation
Jenkins Load Testing
Section for all Jenkins Load Testing Contents.
Work In Progress
This section is under active development.
Content
Benchmarking servers
Load Test Strategy, Scenarios and Results
LikeBe the first to like this
jenkins load testing infrastructure requirements scenarios
solution development stress benchmark
Jenkins Master Setup
Hardware Requirements
TBC after load testing
Software Requirements
Sun Java 6
Latest released Jenkins packaged by Abccompany Build Team
Setup Process
Install latest ss-fat-jenkins-gen & ss-fat-jenkins-plugins-gen
RPMs from RHN/Spacewalk:
yum install ss-fat-jenkins-gen ss-fat-jenkins-plugins-gen
Upgrade Process
Upgrade to latest ss-fat-jenkins-gen &
ss-fat-jenkins-plugins-gen RPMs from RHN/Spacewalk:
yum update ss-fat-jenkins-gen ss-fat-jenkins-plugins-gen
Securing the Jenkins server
Abccompany Internal CA
The security standard is to use Specsaver internal CA for securing
internal resources (cost effectiveness). However, on Linux these CA
certificates are not distributed & installed in browsers. In order to stop
getting certificate warnings, please follow the instructions here.
As passwords are used to log in to the Jenkins server
(authentication with Active Directory as per standard detailed below), the
Jenkins master must only be accessible over SSL. In order to do this in an
efficient & easy to configure way, we standardise on the use of Apache
httpd to provide the SSL configuration & proxy requests to the Jenkins master.
Apache httpd must do the following:
Bind to port 80 & 443
Send a client side redirect (permanent) all requests sent to port
80 to port 443
Send a client side redirect to an FQDN if the master is requested
using only a hostname (e.g. redirect all requests to https://buildmanager to
https://buildmanager.uk.abccompany.com)
Be configured to serve the full certificate path all the way back
to the root CA
The Jenkins manager certificate should be created with both the
short name & the FQDN as CN & subjectAltName to allow both to be used
without certificate errors in the browser.
Apache config can be based on the following (should go in
/etc/httpd/conf.d/buildmanager.conf). Remember to change the hostnames if
required:
<VirtualHost *:80>
ServerAdmin uk.buildnotifications@abccompany.com
ServerName buildmanager.uk.abccompany.com
ServerAlias buildmanager
ErrorLog logs/buildmanager.uk.abccompany.com-error_log
CustomLog logs/buildmanager.uk.abccompany.com-access_log
common
RewriteEngine on
RewriteCond %{HTTPS} off
RewriteRule (.*)
https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>
<VirtualHost *:443>
ServerAdmin uk.buildnotifications@abccompany.com
ServerName buildmanager.uk.abccompany.com
ServerAlias buildmanager
ErrorLog logs/buildmanager.uk.abccompany.com-error_log
CustomLog logs/buildmanager.uk.abccompany.com-access_log
common
RewriteEngine on
ProxyPass /
http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
ProxyRequests Off
SSLEngine on
SSLCertificateFile
/etc/httpd/conf/ssl.crt/buildmanager.cert.pem
SSLCertificateKeyFile
/etc/httpd/conf/ssl.key/buildmanager.key.pem.nopass
SSLCertificateChainFile
/etc/httpd/conf/ca-bundle/Specs-CA-Web-chain.cert.pem
</VirtualHost>
Standard Configuration
All configuration must be done after installation. This allows for
multiple masters with different configurations. Configurations are checked into
SVN after any change (see more details below) so if a new master has to be
built as failover, it is simply a case of checking out configurations.
Number of executors
Default setting: 0
No builds should run on master in order to distribute builds to
slaves. This keeps the master from being under load & accessible by both
users & slaves to keep uptime of this critical piece of the build
infrastructure as high as possible.
Default quiet period
Default setting: 5 seconds
Best practice is to ensure that every check-in does not break a
build & contains all changes related to changed functionality. A very small
quiet period, such as 5 seconds, encourages the practice of a build for each
commit. This practice helps quickly identify commits that break builds or fail
tests, allowing for quicker turnaround in solving problems.
Define default log rotator value
Days to keep builds: 90
Max # of builds to keep: -1
Days to keep artifacts: -1
Max # of builds to keep with artifacts: 5
These settings keep build logs on Jenkins for 90 days, but only
keep artifacts of those builds on the master for 5 days. Logs will be
accessible for 90 days. The build retention policy of the Framework
Architecture Team is 60 days - this should be catered for by the actual build
configuration (following standards) copying artifacts to the disk pool. The
disk pool cleanup will cater for the build retention policy. If this setting is
set too high, we are likely to hit disk space issues regularly as all builds
archive back to the master. This is exactly what the disk pool is for (cheaper
storage).
SCM Sync configuration
Never bother me with commit messages: false
Display SCM Sync Status: true
Commit message pattern: [message]
Source control type: Subversion
*Repository URL: http://svn/buildprojects/CI_Tools/Jenkins/trunk/jenkins-config*
This ensures that every change to any configuration file is
checked in to Subversion. Displaying the SCM sync status shows an icon at the
bottom of the screen showing the status of the SCM sync - whether it is up to
date or not. The repository URL will need to be created & only used for one
Jenkins instance (obviously) so it doesn't get overridden by another Jenkins
instance. Credentials will need to be entered just as with any other SVN build.
Screenshot
The standard configuration is as below in a screenshot (copied
from test Jenkins master http://10.4.23.113:8081:
Jenkins Packaging
Jenkins packaging is done via Abccompany strategic packaging
methodology (i.e. RPM). The package delivery structure is based on the below
concept:
Jenkins Installer
This is the base installer provided by Jenkins. This install
contains the actual Jenkins software and it is published in various formats to
support industry wide acceptance.
To effectively utilise our current Development & Build
Technology stack, leverage the auto-retrieval capabilities of Jenkins installer
from a remote repository and deliver Abccompany customised Jenkins solution, we
use Maven builds. This provides the ability to enforce standards and uniformity
in Jenkins installation on target platforms within Abccompany.
There are several dependencies which the installer require, and
using Maven build helps to integrate such dependencies seamlessly.
Jenkins Plugins
This is a collection of approved and supported Jenkins plugins
within Abccompany. The list of plugins available via this package are published
in Plugins Standardisation page.
Abccompany Customisation
This is collection of various customisations that are required for
Jenkins and other peripherals to function effectively in Abccompany
environment. This would include contents like log rotate configuration, yum
repository configuration, Init v5 script, etc. These customisations are
embedded within the Jenkins Installer & Jenkins Plugins RPM as
configuration file.
Scripted Configuration
This is a collection of various RPM scripts (pre/post - install,
remove) that are packaged within the Jenkins Installer & Jenkins packaging
rpm. These scripts follow the guidelines are published in RPM Guidelines
Jobs Standardisation
This is taken from the standard template already existing on
BuildManager
The below is the template MAVEN release build
Release build Screenshot
The live link is below
https://buildmanager.uk.abccompany.com/view/Templates/job/Release-Job-Template/configure
The below is the MAVEN metrics build template
Metrics Build Snapshot
The live link is below
https://buildmanager.uk.abccompany.com/view/Templates/job/Metrics-Maven-Template/
Jenkins Jobs should be standardised in the following ways
Template
The Template job should be a disabled job which is copied and then
the relevant data filled in
There will be two templates initially, one for metrics builds and
one for continuous integration builds
Naming
Jobs will have a standard name in the following format:
The name will start with CI or Metrics followed by a hyphen
The name of the build will be next with hyphens for separators.
The type of build will be next Branch/Trunk.
If a Branch the branch name will follow e.g.
Metrics-Blackbox-Trunk, CI-FinlandCompiere-Branch-c303
Retention Time
Jobs are kept for 90 days as per FA Team retention policies.
Artifacts for builds as a convenience will be kept on buildmanager
for 5 builds.
In a metrics build we keep 3 builds just for the ability to check
errors/debug in builds as "mvn site" can sometimes cause problems.
Labels
Each slave will have labels describing is capabilities including
size (standard,large), jvm's (java5, java6, java7) and mysql (mysql5.0,
mysql5.1, mysql5.5) installed these can then be used in conjunction to limit
which slaves the build can be run on.
Configuration
Due to standardisation of Abccompany' main jobs there should be
little need for individual job configuration (local config is the exception).
Because of job copying however a divergence from the template
could occur. There should be a regular review of job configuration to ensure
compliance with the template and if a change has to be made to the template or
a new template created that change should be mirrored in all builds based on
that template.
Unless required by a specific build the jdk should be left at
default for the node the build is built on.
Timouts should be standardised to the following table
Build Type
Timeout
Notes
Release-Job-Template
60
Metrics
120
Metrics builds need a considerable amount of additional time
History has shown that we should provide at least 1024MB of memory
for builds. Specific builds may necessitate more.
We use clean install as it is deemed better to create all the
artifacts and only deploy to buildrepo as a publishing action if all the
modules pass so we do not get partial build artifacts on a failed build.
Other options can be found on a build by build basis atbuild
idiosyncrasies
Tagging
Tag Base URL http://<url of tag directory for
project>/${env['BUILD_TAG']}
Tag Comment Tagged by Jenkins svn-tag plugin.
Build:${env['BUILD_TAG']}
Tag Delete Comment Delete old tag by SvnTag Jenkins
plugin.
Post build actions
Metrics builds publish there results and call sonar.
Sonar reuses Cobertura results to speed up the process.
as a convenience to the development teams rpms are published to
enable quick retrieval as buildrepo renames the rpm and can lead to confusion.
Notifications
Notifications will be done by email to groups set up by the
relevant teams as given by example build-notification.uk.abccompany.com
Permissions
Anonymous should have access to read and browse the workspace.
Selected users will have the ability to force, configure and
release builds..
Build Triggers
For metrics builds a timed build should be used, preferably
between 22:00-06:00 to enable the slaves to be used for CI during business
hours.
For normal builds it will be fired by a svn post commit hook and
it will also build monthly to ensure completeness of build. This is
accomplished through a plugin in the jenkins master and a post commit hook on
the svn repository.
Post Build Actions
Continuous Integration Builds
The main post build action will be to publish to the artifact
repository the required artifacts.
We will also publish certain artifacts to the front page of the
build to allow easy collection for the development teams.
Certain builds may require other actions but this will be on a
build by build basis and should be exceptional.
The build will be tagged in the subversion repository.
The notification of the required team should be included here to
cope with broken builds
Metrics Builds
publishing of the different metrics will be published here as a
convenience to development teams.
Sonar will be ticked
If the build is from a branch Sonar will require the branch name
to be input so that the build is unique from the trunk build.
You have to type in seperate maven opts for sonar this should
normally be -Xmx1024m to enable enough memory for most builds, this might have
to be raised for larger builds.
Deployment
The standard process will be a clean install build, copying the
required artifacts back to the master then using the deploy to maven repository
post build activity this eliminates the chance of partial deployments of
snapshots. Note that some builds are still standardised around a clean deploy
and linking to the artifacts this lessens io traffic and is much faster than
transferring the artifacts back to the master. (Note this exploits the JNLP
performance of migrating artefacts from slave to master).
LikeBe the first to like this
jenkins jobs standartisation Edit Labels
Slave Setup
Node Names
Nodes will be named in lower case the same as the host name they
are based on e.g lysander
Location for work space
The work space will be /data/jenkins-slave/ if this does not exist
on the slave this will have to be created. This assumes /data is a separate
partition.
The permissions will be set to 755 for the Hudson user.
Executors
There shall be 1 executor per node.
Utilization
Unless the build machine has specific builds dependent on it the
standard utilization will be "Utilize this slave as much as possible".
This will enable maximum usage of limited slave resources.
Launch Method
On all platforms it will be jnlp as presently it is much
faster to archive artifacts back to master.
When using JNLP, certain configurations available via bash profile
are not available and existing free-style builds running Maven will need to be
configured to use a specific Maven version otherwise the builds would fail. We
would need the information about the impact it will have on builds, or the type
of build setup in relation to the launch method selected.
LikeBe the first to like this
jenkins slave Edit Labels
4 Comments (Reverse Order)
User icon: soumen.trivedi
Unknown User (soumen.trivedi)
Review comments have still not been incorporated. We need more
elaborate explanation around why, what & how these are achieved.
In short, do not agree that this page is complete and ready to be
signed off. Happy to discuss further if necessary/required.
Creating a Jenkins Slave
Following the kickstarted jenkins master server, the slave options
are currently non-functional (TBA at future date).
The following instructions should be
followed.
On the slave, install the slave plugin using the below command:
mkdir -p /data/jenkins-slave
yum install RPMNAME --nogpgcheck -y
Icon
If you are using the slave on a build manager server other than
the main buildmanager, you must edit the file /etc/sysconfig/jenkins-slave and
change JENKINS_URL to your target build manager server, and then restart the
service
On the master, go to manage jenkins > manage nodes > new
node
Enter the slaves hostname as the node name, and choose dumb slave
and click next.
Example screenshot of settings page:
Important points:
No of Executors:1
Remote FS Root: /data/jenkins-slave
Launch method: Launch slave agents via JWS
Availability: Keep this slave on-line as much as possible
After clicking save, run the below on the jenkins slave server:
service jenkins-slave restart
The build manager server should now see the slave as online
