Pages

Showing posts with label Tools. Show all posts
Showing posts with label Tools. Show all posts

Friday, February 27, 2015

Pipes as Code

Finally we have started to move away from having build pipes as a chain of Jenkins jobs. There has been alot written of the subject that CI systems arnt well suited to implement CD processes. Let me first give a short recap on why before I get into how we now delivery our Pipes as Code.

First of all pipes in CI systems have bad portability. They are usually a chain of jobs set up through either a manual process or through some sort of automation based on a api provided by the CI system. The inherrited problem here is that the pipe executes in the CI system. This means that it is very hard to test and develope a pipe using Continuous Delivery. Yes we need to use Continuous Delivery when implementing our Continuous Delivery tooling otherwise we will nto be able to deliver our CD Processes in a qualitative, rapid and reliable way.

Then there is the problem of that data that we collect during the pipe. By default the data in a CI system is stored in that CI systems. Often on disk on that instans of that CI server. Adding insult to injury navigation of the build data is often tided to the current implementation of the build pipe. This means that a change to the build pipe means that we can no longer access the build data.

For a few years now we have been off loading all the build data into different types of storages depending on what type of data it is. Meta data around the build we store in a custom database. Logs go to our ELK stack, metrices to Graphite and reports to S3.

Still we have had trouble delivering quality Pipes. Now that has changed.

We still use a CI Server to trigger the Pipe. On the CI server we now have one job "DoIt". The "DoIt" job executes the right build pipe for every application. Lets talk a bit on how we pick the pipe.

Each git repo contains a YML file that says how we should build that repo. Thats more or less the only thing that has to be in the repo for us to start building it. We ingore all repos without the YML files. So we listen to all the gerrit triggers and ignore ones withouth

The YML is simply pretty much just

pipe: application-pipe
jdk: JDK8


We describe our build pipes in YML and implement our tasks in Groovy. Here is a simple definition. 

build: 
    first:
     - do: setup.Clean     - do: setup.Init
    main:
     - do: build.Build     - do: test.Test
    last:    
     - do: log.ReportBuildStatus
last: 
   last:    
     - do: notify.Email

Each task has a lifecycle of first, main, last. The first section is always executed and all of the "do´s" in the first section are executed regardless of result. In the main secion the "do´s" are only execute if everything has gone well so far. Last is always executed regardless of how things went.

The "do´s" are references to groovy classes with the first mandatory part of the package stripped. So there is a com.something.something.something.setup.Clean class.

A Context object is passed through all the execute methods of the "do´s". By setting context.mock=true the main executing process adds the sufix "Mock" to all "do´s". This allows us to unit test the build pipe inorder to assert that all the steps that we expect to happen do happen in the correct order.

When alot of things start happening its not really practicall to have a build task all that verbose especially since we have multiple pipes that share the same build task. So we can create a "build.yml"  and a "notify.yml" which we then can include like this.

build: 
    ref : build
last: 
   last:    
     - do: notify

So this is how our build pipes look and we can unit test the pipe, the tasks and each "do" implementaiton.

Looking at a full pipe example we get something like this.

init:
    ref: init
build:
    parallel:
        build:
            ref: build.deployable
        provision:
            ref: provision.create-test-environment  
deploy:            
    ref: deploy.deploy-engine
test:            
    functional-test
        ref: test.functional-tests
    load-test: 
        ref: test.load-tests
release: 
    parallel:
        release:
            ref: release.publish-to-nexus
        bake:
            ref: bake.ami-with-packer
last: 
    parallel:
        deprovision:
            ref: provision.destroy-test-environment
        end:

            ref: end

Thats it.!

This pipe builds, functional tests, load tests and publishes our artifacts as well as baking images for our AWS environments. All the steps report to our meta data database, elk, graphite, s3 and slack.

And ofcourse we use our build pipes to build our build pipe tooling.

Continuous Delivery of Continuous Delivery through build Pipes as Code. High score on the buzzword bingo!

Monday, July 7, 2014

Continuous Deployment in the Cloud Part 2: The Pipeline Engine in 100 lines of code

As I talked about in my previous post in this series we need to treat our Continuous Delivery process as a distributed system and as part of that we need to move the Pipe out of Jenkins and into a first class citizen of its own. Aside from the facts that a CI Tool is a very bad Continuous Delivery/Deploy orchestrator I find the potential of executing my pipe from anywhere in the cloud very tempting.

If my pipe is a first class citizen of its own and executable on any plain old node then I can execute it anywhere in the cloud. Which means I can execute it locally on my laptop, a simple minion in my own cloud or in one of all of the managed CI services that have surfaced in the Cloud.

To accomplish this we need five very basic and simple things

  1. a pipeline configuration that defines what tasks to execute for that pipe
  2. a pipeline engine that executes tasks
  3. a library of task implementations
  4. a definition of what pipe to use with my artefact
  5. a way of distributing the pipeline engine to the node where I want to execute my pipeline
Lets have a look.

Define the pipeline

The pipeline is a relatively simple process that executes tasks. For the purpose of this blog series and for simple small deliveries sequential execution of tasks can be sufficient but at my work we do run a lot of parallel sub pipes to improve the throughput on the test parts. So our pipe should be able to handle both.

We also want to be able to run the pipe to a certain stage. Like from start to regression test and then step by step launch in QA and launch in Prod. Obviously of we want to do continuous deployment we don't need to worry too much about that capability. But I include it just to cover a bit more scope.

Defining pipelines is no real rocket science and in most cases a few archetype pipelines will cover like 90% of the pipes needed for a large scale delivery. So I do like to define a few flavours of pipes that gives us the ability to distribute a base set of pipes for CI to CD.

Once we have defined the base set of pipe flavours the each team should configure which pipe they want to handle their deliverables.

I define my pipes something like this.

name: Strawberry
pipe: 
  - do:
     - tasks:
       - name: Build
         type: mock.Dummy
  - do:
     - tasks:
       - name: Deploy A
         type: mock.Dummy
       - name: Test A
         type: mock.Dummy
     - tasks:
       - name: Deploy B
         type: mock.Dummy
       - name: Test B
         type: mock.Dummy
    parallel: true     
  - do:
     - tasks:
       - name: Publish
         type: mock.Dummy
  
A pipe named Strawberry which builds our services then deploys it in two parallel pipes where it executes two test suites and finally publishes the artefacts in our artefact repo.  At this stage each task is just executed with a Dummy task implementation.

The pipeline engine

We need a mechanism that understands our yml config and links it to our library of executable tasks.

I use Groovy to build my engine but it can just as easily be built in any language. Ive intentionally stripped down some of the logging I do but this is basically it.  In about 80 lines of code we have a engine that loads tasks defined in a yml, executes then in serial or parallel and has the capability to run all tasks, the tasks up to one point or a single task.

@Log
class BalthazarEngine {

    def int start(Map context){
        def status = 0
        def definition = context.get "balthazar-pipe" 
        
        for (def doIt:  definition["pipe"] ){
            status = executePipe(doIt, context)
        }
        return status
    }
    def int executePipe(Map doIt, Map context){
        def status = 0
        if (doIt.parallel == true){
            status = doItParallel(doIt,context)
        } else {
            status = doItSerial(doIt,context)
        }
        return status
    }
    
    def int doItSerial(def doIt, def context){
        def status = 0
        for (def tasks : doIt.do.tasks){
            status = executeTasks(tasks, context)
        }
        return status
    }
    
    def int doItParallel(def doIt, def context){
        def status = new AtomicInteger()
        def th
        for (def tasks : doIt.do.tasks){
            def cloneContext = deepcopy(context)
            def cloneTasks = deepcopy(tasks)
            th = Thread.start {
                status = executeTasks(cloneTasks, cloneContext)
            }
        }
        th.join()
        return status
    }   
    
    def int executeTasks(def tasks, def context){
        def status = 0
        for (def task : tasks){
            //execute if the run-task is not specified or if run-task equqls this task
            if (!context["run-task"] || context["run-task"] == task.name){
                log.info "execute ${task.name}"
                context["this.task"] = task
                def impl = loadInstanceForTask task;
                status = impl.execute context 
            }
            
            if (status != BalthazarTask.TASK_SUCCESS){
                break
            }
            
            if (context["run-to"] == task.name){
                log.info "Executed ${context["run-to"]} which is the last task, done executing."
                break
            }
        }
        return status
    }
    def loadInstanceForTask(def task){
        def className = "balthazar.tasks.${task.type}"
        def forName = Class.forName className
        return forName.newInstance()
    
    }
    def deepcopy(orig) {
        def bos = new ByteArrayOutputStream()
        def oos = new ObjectOutputStream(bos)
        oos.writeObject(orig); oos.flush()
        def bin = new ByteArrayInputStream(bos.toByteArray())
        def ois = new ObjectInputStream(bin)
        return ois.readObject()
    }
}

A common question I tend to get is "why not implement it as a lifecycle in maven or gradle". Well I want a process that can support building in maven, gradle or any other tool for any other language.  Also as soon as we use another tool to do our job (be it a build tool, a ci server or what ever) we need to adopt to its lifecycle definition of how it executes its processes. Maven has its lifecycle stages quite rigidly defined and I find it a pita to redefine them. Jenkins has its pre, build, post stages where its a pita to share variables. And so on. But most importantly use build tools for what they do well and ci tools for what they do well and none of that is implementing CD pipes.

Task library.

We need tasks for our pipe engine to execute. The interfaces for a task is simple.

public interface BalthazarTask {
    int execute(Map<String, Object> context);
}

Then we just implement them. For my purpose I package tasks in "balthazar.task.<type>.<task>" and just define the type and task in my yml. 

Writing tasks in a custom framework over say jobs in Jenkins is a joy.  You no longer need to do workaround to tooling limitations for simple things such as setting variables during execution.  

Anything you want to share you just put it on the context.

Here is an example of how two tasks share data.

  - tasks: 
    - name: Initiate Pipe
      type: init.Cerebro
  - tasks: 
    - name: Build
      type: build.Gradle
      command: gradle clean fatJar

I have two tasks. The first task creates a new version of the artefact we are building in my master data repository that I call Cerebro. (More on Cerebro in the next post). Cerebro is the master of all my build things and hence my version numbers come from there. So the init.Cerebro task takes the version from Cerebro and puts it on the context.

@Log
class Cerebro implements BalthazarTask {
    @Override
    def int execute(Map<String, Object> context){
        def affiliation = context.get("cerebro-affiliation")
        def hero = context.get("cerebro-hero")
        def key = System.env["CEREBRO_KEY"]
        def reincarnation =  CerebroClient.createNewHeroReincarnation(affiliation, key, hero)
        context.put("cerebro-reincarnation",reincarnation)
        return TASK_SUCCESS
    }
}


My build.Gradle task takes the version number from cerebro (called reincarnation) and sends it to the build script. As you can see I can use custom commands and in this case I do as fat jars is what I build. By default the task does gradle build. I can also define what log level I want my gradle script to run.

@Log
class Gradle implements BalthazarTask {
    @Override
    def int execute(Map<String, Object> context){
        def affiliation = context["cerebro-affiliation"]
        def hero = context["cerebro-hero"]
        def reincarnation = context["cerebro-reincarnation"]
        def command = context["this.task"]["command"] == null ? "gradle build": context["this.task"]["command"]
        def loglevel = context["this.task"]["loglevel"] == null ? "" : "--${context["this.task"]["loglevel"]}"
        
        def gradleCommand = """${command} ${loglevel} -Dcerebro-affiliation=${affiliation} -Dcerebro-hero=${hero} -Dcerebro-reincarnation=${reincarnation}"""

        def proc = gradleCommand.execute()                 
        proc.waitFor()                               
        return proc.exitValue()
    }
}

This is how hard it is to build tasks (jobs) if its done with code instead of configuring it in a CI tool. Sure some tasks like building Amazon AMI´s take a bit more of code. (j/k they don't). But ok a launch task that implements a rolling deploy on amazon using a A/B release pattern does but I will come back to that specific case.

Configure my repository

So I have a build pipe executor, pre built build pipes and tasks that execute in them. Now I need to configure my repository.

In my experience 90% of your teams will be able to use prefab pipes without investing too much effort into building tons of prefabs. A few CI a few simple CD and a few parallelized pipes should cover a lot of demand if you are good enough at putting an interface between the deploy tasks and the deploy as well as the test tasks and the deploy tools.

So in my repo I have a .balthazar.yml which contains.

balthazar-pipe: Strawberry

Distributing the pipeline engine and the task library

First thing we need is a balthazar client that starts the engine using the configuration provided inside my repository.  Simply a Groovy script does the trick.

@Log
class BalthazarRunner {
    def int start(Map<String, Object> context){
        Yaml yaml = new Yaml()
        if (!context){
            def projectfile = new File(".balthazar.yml")            
            if (projectfile.exists()){
                context = yaml.load projectfile.text
            } else {
                throw new Exception("No .balthazar.yml in project")
            }
            
        }
        def name = context.get "balthazar-pipe" 
        def definition = yaml.load this.getClass().getResource("/processes/${name}.yml").text 
        BalthazarEngine engine = new BalthazarEngine()
        
        context["balthazar-pipe"] =  definition
        context["run-to"] = System.properties["run-to"]
        context["run-task"] = System.properties["run-task"]
        return engine.start(context)
    }
}
def runner = new BalthazarRunner()
runner.start([:])

Now we need to distribute the client, our engine and our library of tasks to the node where we want to execute the pipeline with our code repository. This can be done in many ways.

We can package balthazar as a installable package and install it using yum or similar tool. This works quite well on build servers but it does limit us a bit on where we can run it as we need "that installer" to be installed on the target environment. In many cases its really isn't a problem because if your a Debian shop then you have your deb everywhere and if your a Redhat shop then you have your yum. 

I personally opted for another way of distributing the client. Partially because Im lazy and partially because it works on a lot of environments. When I make my balthazar.yml I also checkout the balthazar client project as a git submodule.

So all my projects have a .balthazar.yml and a balthazar-client folder. In my client folder I have a balthazar.sh and a gradle.build file. I use gradle to fetch the latest artefacts from my repo and then the shell script does the java -jar part. Not all that pretty but it works.

Summary

So now on all my repos I can just do...

>. balthazar-client/balthazar.sh

... and I run the pipe configured in .balthazar.yml on that repo. Since all my tasks integrate with my Build Data Repository I get ONE view of all my pipe executions regardless of where they where executed.

CD made fun! Cheers!

Monday, June 2, 2014

Blog Series: Continuous Deployment in the Cloud

For my next few posts I am going to focus on writing a series of articles how to do Continuous Delivery & Deployment in a cloud environment. Ive always been a bit cautious when it comes to tutorial style blogs, talks and articles. I usually find them to be too shallow and then never reveal the true issues that need to be solved. This often leads to bad, premature and uninformed decisions made by the consumer of the tutorial.

So instead I am going to try provide a much richer series of articles that focus on how to Architect, Test, Deploy and Deliver in a Cloud environment.

In my conference talks I often talk about the key of having a build data repository that is separate from the build engine (CI Tool). More then once I have been asked if I can open source this our tool. Well Im not sure has been my answer. So instead Ive decided to implement a new similar tool and open source it. Im going to over architect it a bit on purpose as its going to be the main example in the article series. The Process Implementation in this series is going to use that tool as its build repo.

This is how the Process Implementation will look like. I will go deeper into this picture as I move forward but a few quick words about it.

The key to a scaleable CD process is for it to be independent of its runtime environment. The CD process drawn here can be executed from a Build Environment and/or a Dev Environment. The dev can push from his own environment right into production or he/she can let the build environment do it from him/her. Regardless of where the process is initiated it will be executed in the same way and it will integrate with the build data repository to which it reports any events that happen on that build and its also from where it gets its version number.

Will I talk about tooling this time? Yes I will. This article will be based on Git as SCM, AWS as Test and Prod Runtime Environments, Travis CI as Build Environment and Gradle as Build Tool but I still have not decided upon test tool most likely it will be RESTAssured.

This series will take some time to write and will mostly be done during the later part of this summer and the fall. If you are interested in this article series and/or the build data repository then please +1 this article to show me that there is interest.

Thanks

Tuesday, January 1, 2013

Upps our Continuous Delivery process became mission critical

At some point something changed with our Continuous Delivery process, it became mission critical. When we started working on the process it was basically a side project that another Tomas and I had. We added a consultant early in our project and he ended up doing some of the work on the first version of our deployment scripts but it wasnt anything organized and not part of any proccess or tools team.

When we increased the number of developers and started seeing issues with stability and scaleability we also started to realize that our process had become mission critical. In fact or continuous delivery process had become more important to us then our mail system.

Now we had a mission critical hobby project with the following setup.
  • No official Owner.
  • No official Developers.
  • No official Operations professionals involved
    • Operations only supporting the OS of the Jenkins and nexus instance.
  • One "live" instance of Jenkins on a super small virtual node. 
    • All development done on live instance.
  • One "live" instance of Nexus with a very small disk.
    • All development done on live instance.
  • Small number of test servers, virtual but not cloud nodes.
Having about 30 developers really depending on a process that is setup like this is obviously a no go.

We started to figure we need to put more effort into it when we where to do our first rewrite of our deploy scripts. Still we didn't think in terms of production mission critical system. We needed a resource and I kept insisting we needed a CM, more on that in an upcoming post. We had architecture and test working together building the application around the process. But we needed some more hands building the deploy scripts and also someone who could help us with the complexity of our system configuration. As I wrote in the entry on deploy scripts this didn't work out well at all. Mostly because the CM ended up working alone in a corner of the organization but also because he didn't share our vision of continuous delivery. Between all discussions trying to get us to implement branching strategies he was writing deployment scripts without any JBoss or DB competence. Obviously this didn't work out all that well and it was during this script rewrite that we started to realize that our process was mission critical. The new deploy scripts where very unstable and as mentioned our tests had stability issues.

Now we started realizing that we have a mission critical system at our hands and we need to start treating it as such. Still this was a bit of an unknown entity in our landscape operations only support our office it and our customer deliveries while development supports tooling. While this for sure falls into tooling department the development organization isnt equipped to support a mission critical system. Still we had to do something about it so this was when we created our tools team, we refer to it as a platform team as it was intended to own certain components such as logging, help desk, ect. But main focus was to be continuous delivery. Our lacking development environment was another area of responsibility that we moved to this team, more on that as well in another entry.

The team consisted of our CM, application DBA, a newly added senior Java developer and my self as  architect/lead.  It was obvious from the onset how effective it is when you have resources (with full range of competence) that can focus on the process. This made us much more responsive to bugs in the process and faster in implementing changes.

We still at this date have not solved all the infra structure issues but most of it is being worked by the tools team and a new resource in our operations department who is responsible for our tooling serves. Still we don't have a Jenkins test environment and still the operations responsibility of Jenkins and Nexus aren't really well defined. But we have resources dedicated to the process and when something isn't working we handle it as bugs.

The biggest lesson is that its really important to get dedicated resources from dev and ops early. Getting two 50% resources is better then one full time as one isolated resource is a huge bottleneck and has a hard time prioritizing his work. Also make sure to have a bug/enhancement process in place early. Priorities should be made based on user experience, same as with any system in production. Also as soon as the process is in use by the developers you need a test environment for Jenkins (or what ever build server you use to drive the process) as its a production system after all.

I think the reason we got a bit blindsided by the process becoming mission critical is that we haven't had anything similar in our landscape before. There is actually one thing that has grown mission critical at about the same rate hand in hand with our CD process and that's our JIRA server. In fact we have an even bigger dependency on our JIRA if it goes down our developers have no clue what to work on and get stranded very quickly. For us this is a new type of mission critical systems. Previously they have only been supporting systems.

Another reason is that the continuous delivery community talks about how easy it is to get started and how we can just take small baby steps from our nightly build CI. It is both true and the way to go. I just guess I wasn't reading the fine print which says "and then it becomes mission critical". 

Saturday, December 8, 2012

The impact of Continuous Delivery on the role of the Tester

Continuous Delivery really does change  way we work. It's not just tools and processes. We know that because every talk, every blog says it does but when you really see it its quite interesting.

The changes to the level of responsibility required by each developer has been one of the hardest thing to manage. Don't check shit in or you will break the pipe! Don't just commit and run, it's your responsibility that the pipe is green! All that is frustrating and has been the single most time consuming on our journey, but its worth a post of its own.

What I'd like to focus on is the changes to he role of the tester. Previously we have worked mostly with manual GUI driven testing. Our testers have tons of domain knowledge and know our systems inside out. We have worked with test automation in some projects and I have some test automation experience from the past. I've been trying to champion test automation for years but we have had a bit of a hard time getting it of the ground. When we have its not been test automation but rather automated tests that require some kind of tester supervision.

In our current project, as I've written about in previous posts , we decided to go all in. When we started we where a team of just architects. I was leading the work on the test automation. We got really good results working a lot with test architectur and test ability architecture in our application. But after some time we started to suffer from from lack of tester input in our testing. We obviously ended up with a very happy case scenario oriented suite.

So we started to bring on testers to our project but what profiles should we look for. First we started to look for just our notion of what a tester was and had always been in our projects. We happily took on testers with experience of testing and test managing portals, order systems, corporate websites ect, ect. But our delivery at this time didn't have a GUI all testing was done using Fitnesse. We didn't really get the interaction between dev and test we where hoping for.

We did get use for our testers for partner integration testing, which was manual (using rest client). But it wasn't really working well because they didn't know the interfaces and the application as they hadn't worked with th test automation.

We have been having this same experience for over a year. Our testers don't seem to be able to get involved in our test automation. But we do have a few who are and we are super lucky to have found them because we have not really had much of a clue when recruiting.

Our developers who are very modern and in a lot of cases super interested in test automation have constantly been bashing us about our choices of test tools. As I talked about in my previous post they totally refused SoapUI and arnt all that fond of Fitnesse either, even though they prefer it alot over SoapUI. But my take has always been "we need testers to feel comfortable with the tool, it's their domain, let them pick".

After we decided to move to REST assured I started to realize the problem I've failed to grasp for the six years or so I've been working with test automation. There are two sets of testers. GUI testers and technical testers. A GUI tester will always struggle with test automation. He/she has little to none experience or education in coding. The technical tester has a background as a developer or started developing as part of a automation interest.

Still even the technical eaters we have who have experience from test automation have had a transition period coming into the continuous world. Testers do tend to accept manual steps that arnt acceptable in a continuous world. It's not ok to just quickly verify this bug manually because its so simple and changing the test case that had a gap is a lot of work.  Its not ok to just add that user into e DB manually. It not ok to verify against the DB.

In the past our demand in GUI clicking testers has been high and our demand in technical testers has been low. At least in our old nonCD world the TestPyramid, http://martinfowler.com/bliki/TestPyramid.html, was upside down.

The Continuous Delivery expansion will drive a shift in what we look for in testers. Our tester demographics will move towards matching the pyramid. We will still need the GUI testers but their work will move more towards requirements gathering and acceptance testing. While I think we will see a new group of testers, with a much stronger developer background, come in and work with the automation.

This new group of technical testers or developers with super high understanding of testing is still hard to find but I hope we will see more of them.

Picking the tool for Component Testing

As i discussed in the previous post we realized we need to test our component, not just unit test code and do functional testing. Biggest reason was that we where cluttering our functional tests with validation a that didn't belong at that level. Cluttering tests with logic that doesn't belong at that level was increasing our test maint costs, something we knew rom the past we had to avoid.

Since our application is based on REST services exposed by components with very well defined responsibility its very well suited for component testing.

The responsibility for our components was clearly defined. A component test is a black box test with the responsibility of validating the functionality and interfaces of the component. This definition was somewhat a battle as some how our testers seem (our project and others in our company) to insist that they want to validate stuff on the database. I don't like beeing unreasonable but this was one thing I felt I had to be unreasonable on, black box means get the f*** out of our database. I still feel a bit bad about overriding our test managers on this but I still feel strongly that it had to be done. So all tests go through public or administrative interfaces on our components. Reason for this is obviously that I wanted our tests to support refactoring and not break when we change code.

The upside on forbidding database access was that we had to make a few administrative interfaces that we could eventually use for building admin tools.

So now what tools should we use? We where using Fitnesse for our functional testing but Fitnesse biggest weakness is that it doesn't separate test flow from test data. With functional testing this isn't really a big issue as each test is basically a flow of its own. But with component testing and it by nature beeing more detailed we saw that we would get much more tests cases per flow. Another weakness is that Fitnesse doesn't go all that well together with large XML/json document validation.

We do our GUI testing with selenium Fitnesse combo and that we would continue doing. But for our REST service testing our first choice was SoapUI. We prototyped a bit and decided we could accomplish what we wanted with it so we started building our tests. This was the single biggest mistake we made with our continuous delivery process.

Back in the days when we just did functional tests for our deliveries in Fitnesse we had a nice process  of test driven development. Our developers activly worked on developing testcases and fixtures and the tests went green as the code hit done. I really like it when it's functional test driven developement and think this is the best form of tdd. This went totally down the drain when we started using SoapUI for our component tests.

Our developers refused to touch SoapUI and started handing over functionality to testers for testing after they had coded it. This resulted in total chaos as we got a lot of untested checkins. Backloaded testing works extremely bad with a continuous delivery process. Especially since we didn't use feature flags.

This put us in an interesting dilemma. Do we choose a test tool that testers feel comfortable working with or do we choose a tool that developers like? I personally am very unreligious when it comes to tools. If it does the job then I don't care so much. But I've always had the opinion that test tools are for testers and its up to them, devs need to suck it up and contribute. Testers always seem to like clicky tools so I wasn't suprised that they wanted to use SoapUi.

We where sitting with a broken test process and our devs and testers no longer working together. Fortunately our testers came to the same conclusion and realized this tool was a dead end for us. The biggest killer was how bad the tool was suited for teamwork and versioning, even enterprise edition. Each and every checkin caused problems and you basically need to checkin all your suites for each reference you change. Horrible.

So after wasting nearly 3-4 months, growing our test dept and killing our dev process on SoapUI we decided to switch to RESTassured. For some of our components this is a definite improvement and its definitely a improvement process wise as our developers are happy to get involved. But I do still see some posible issues on our horizon with this choice. Though the biggest change is for our testers and how we as an organization view the tester role and that will be the topic of my next post.

One very nice thing though, our continuous delivery process is maven based so the change of tooling didn't affect it. Each test is triggered with mvn verify and as long as the tool has a maven plugins don't really care what tool they use for their components.