What is Continuous Regression Testing?

In my previous entry I said that our business case was Continuous Regression Testing. I will make a few entries on the subject.

Our definition was quite clear on what level of testing we wanted to achieve.

We define it at the System Test level, meaning regression testing of our end to end functionality as specified by our Use Cases. Important here is end to end and driven by Use Cases.

As an example say that our system would have a "Register User" Use Case which states that the user should be able to register through a portal and receive an confirmation email. In this case our end to end delivery is from our web front to our mail server. We would be creating an automated Test Case which registers the users through the portal and then the test evaluates that there is a mail sent to the email address of the registered user.

Me and others in our company (not on the team at the time) at the time have had quite a bit of experience working with automated System Tests. Yet we ran into quite a few mines (AGAIN!!) when building our test suites. I will cover these later.

So we had a clear understanding on how we wanted to test and we knew we wanted to test it often. For us testing on each code commit and so heavily relying on test automation was way out of our comfort zone. But our business case was strong and we knew why we wanted to go down that road.

Early on we made a few key decisions. One was to test everything on every commit. Second was to only build each artifact once. We discussed this and came to the conclusion that we would need to retest after a rebuild. We felt our process would be simpler and more correct by doing this.

Our architecture was based on a few well defined components (publishing rest services) that together formed our deliverable.

This is a simplistic view of something similar. Some components where internal and some where externally integrating.

We where aware that we most likely would need to test the components in a continuous way but we ignored it on purpose. Our focus had to be on our delivery as well and we all know too well that we need to prioritize well when working on deadlines. Its easy to get carried away on theoretical models and a desire to solve everything but its not the way we work. We always want something deliverable and then iterate it.

So our initial pipe ended up looking something like this.  Each of our components where built, unit tested and released on commit. The our deployment assembly was updated one component at the time. Each time the assembly was updated we deployed it and tested it on our test server.

It was a very simplistic start to our Continuous journey but it was very adequate for us and most importantly it solved our primary concern Continuous Regression of the customer application.

Its important to remember that the application was new and our team was small at the time.

Where to start with Continuous Delivery

A lot of people have already started the journey without really knowing it. Its few companies today that don´t utilize some key parts of a Continuous process. When reading books and listening to speakers at conferences its easy to feel totally stone age. But the key is to remember that the journey is baby steps towards a vision. But Continuous Delivery is just progression of Continuous Integration and a lot of the basics are built upon in the progression.

By realizing we have something we can start to think about what we really need out of a Continuous Process. In some sense we all want to release all the time and we want our artifacts to be server images deployed on cloud nodes monitored by super cool graphs of all the metrics we can possibly think of. That is all nice but we all do need to provide a, hate the word, business value. So where is the business value in a continuous process for our organisation. That is the question we should ponder before we start.

In fact we had didnt have to ask our selfs the question in that way. Instead we where provided a challenge. For us it was key to solve system regression testing. Test automation and lowering cost in regression testing is obviously one of the key reasons to have Continuous Delivery. For us the key business case was to solve Continuous Regression testing. Continuous what ever else would come as a bonus.

Knowing that Continuous Regression Testing and not Continuous Deploy was our main priority made it easy for us to prioritize our efforts. Our efforts went largely into Testability Architecture in our application, Test Architecture in our test automation frameworks and automating a pipe that would deploy and test each code commit.

So really as with any agile development find your business value, focus on delivering it with a minimal effort "good enough" approach and iterate from there.

Over the next coming posts I will go though what we focused om in order to achieve Continuous System Regression Testing.

What drives my interest in Continuous Delivery?

I want to mitigate the risk of my presence in a project.

My weaknesses as a individual both privately and professorially are that I'm horrible at following instructions I cant do it once but I'm even worse at doing it multiple times. I´m feel physically sick if I need to do repetitive tasks, I get demotivated and hence I do an even worse job. I also tend to forget to do things because I've got my mind full of other stuff. I'm also very bad at testing. Ive worked with test automation for nearly six years now and I still suck at  good test cases.

What I am good at is creative problem solving, finding solutions to problems and thinking outside the box. I'm also quite good at inspiring people.

This is a horrible combination for any project. This guy who cant follow instructions constantly comes up with new "bright ideas", cant event test them and even worse he gets people to engaged and enthusiastic towards his crazy ideas, do we really want him on our team???

Yes I am a high risk and I need to be mitigated.

In all seriousness, I do love change, I thrive in an ever changing environment. I love the positive energy towards solving problems and not dwelling over them. I love high pace and new challenges. I also do know that in order to do what I love to do the "maintenance" of our work needs to be minimized as its a huge time and cost sink for each project.

The "maintenance" part isnt just our production applications. Its our code, our tests, our mechanisms and of course our production deliveries. Every hour not spent maintaining these is spent on bringing in new business value into the organization and solving fun new challenges.

If I can contribute towards lowering the onetime and run time costs of a delivery then Im satisfied because I know my work has really mattered. Im also satisfied on a personal level because I know that Im spending time solving new problems and not maintaining old problems.

I want to come to work and feel satisfied...