Deploy scripts, how hard can it be?

Obviously an integral part of a Continuous Delivery process is to get the artifacts deployed. The same deployment procedure should be used for every deploy. Every deploy means the deploy for every component test, functional acceptance test, load test, rollback test, user acceptance test and yes production as well.

For us this presented our first big challenge. We wanted the same deployment mechanism for all deployments made into environments owned by e development organization and into production owned by operations. Yes we are a legacy NoDevOps organization and there is no changing that anytime soon. I'll cover is more in another post. Basically each project deals with deploy scripts in their own way some have sort of same scripts for all environments some don't. We wanted to change this.

We wanted two things same mechanism for all deploys and triggered the same way. This meant at we had to agree on how to do it with our operations department. This also basically ruled out any sort of third party tooling like chef, or what ever. We felt that we didn't have the leverage and mandate to push a tool on them. We where crossing our fingers that they wouldn't shoot down our proposal to trigger deploys from jenkins.

They actually didn't shoot down using jenkins but they forced us to set up a jenkins at e developers wouldn't have access to. As in most legacy organizations devs aren't allowed to touch a production deploy. They also had some really good input on our initial rudimentary deploy scripts.

We had written some rudimentary scripts early on just to get something deploying. These where quite non-generic hard coded bash scripts. They handled the transfer of artifacts from nexus to the target server, running of liquibase scripts and restart of jboss and mule servers.

So we had the input from operations how to make the scripts better (required to use for prod deploy) and our desire to make the scripts generic. How hard can it be? It's just moving some wars and stuff to a server where we need to put it in the right place and then restart some stuff.

We decided at we first wanted to do the changes operations required as that would allow us to deploy our first delivery to productions. Problem was that while we where doing this we got more deliveries and more components that needed deploying to component test servers. So we ended up with huge set of copy pasted scripts in different stages of development. This bag of copy pasted scripts required maintenance and slowed our development of the main line scripts.

This is really a sad story in our development. It took us several months to get the scripts rewritten to match the requirements of operations. By the time this was done we so desperately needed our generic scripts that we had to throw out the mudball that our scripts had turned into and rewrite them again. This time it went much faster few weeks but the migration of all the other components and deliveries that used different versions of old scripts took quite some time as well.

Here are a few reasons we got into this mess (other then not having a DevOps organization).

1. We didn't have a standardized packaging strategy for our components and deliveries, making it hard to generalize scripts.
2. Having a script developer who was good with Linux but no JBoss, Mule or liquibase skills AND not pairing him with a developer possessing these skills.
3. Starting too late with the rewrite of the rudimentary scripts. We knew for a few months what we had to do but didn't do it.
4. Last but not least leaving rollback mechanisms and rollback pipe building to the last minute.

Get your packaging sorted out early and get your deployment mechanism in place well in time before first production deployment. Also have your rollback strategy sorted out and in testing as part of your pipe early, well before production. I'll cover rollback in another post.

So no deploy scripts aren't that hard if you do things in the right order and don't kill your self by doing mistakes on all levels.