Wat zeggen de experts?

04. Getting Testing Done in the Sprint – Undone Backlog Item

info.nospam@nospam.clemensreijnen.nl (clemensreijnen) — Wed, 25 Jan 2012 09:03:49 +0000

A tip straight from the previous Scrum Guide (The guide updated and the undone TIP is removed, read The 2011 Scrum Guide – A Lot of Changes! ). But, the TIP still works, although it isn’t an ‘official’ tip anymore…

Undone End to End Testing

End to end testing, is not an agile specific testing practice, it’s a common sense testing practice. Create test cases to detected defects anywhere in the chain. This type of testing has been growing the couple of years enormous due to the rise of Service Oriented Architectures and more recent Cloud Computing. as the TMap end-To-End Testing books describes

“End-to-end testing is a form of testing in which the (business) process is tracked through an interconnected series of systems, which may be based in different organizations, aiming to answer the following questions:
- Does the IT process support the business process adequately? - Are the process and systems integrated with each other in the right way?”
TMap.net End-To-End Testing

End-to-End testing is seen as a challenges in Agile environments, with the most important challenge that systems under development aren’t ready for End-to-End testing. Not every service is finished to test the complete end-to-end business process. Stubs are a solution, but can these be made in the current sprint and are stubs good enough to say something about the quality of the system. And, when the system are ready and a change in a service is needed, how and can we re-run all the end-to-end tests, is there time enough.

All these challenges with end-to-end testing in agile projects can be summarized in two topics:

Not Ready Systems in a Sprint
Not Enough Time in a Sprint

Scrum solves this problem, this problem of not able to be compliant on the definition of done at sprint finish, by defining ‘Undone work’. the scrum guide mentions this in a TIP and in the final thoughts of the scrum guide

Some organizations are incapable of building a complete increment within one Sprint. They may not yet have the automated testing infrastructure to complete all of the testing. In this case, two categories are created for each increment: the “done” work and the “undone” work. The “undone” work is the portion of each increment that will have to be completed at a later time. The Product Owner knows exactly what he or she is inspecting at the end of the Sprint because the increment meets the definition of “done” and the Product Owner understands the definition. “Undone” work is added to a Product Backlog item named “undone work” so it accumulates and correctly reflects on the Release Burndown graph. This technique creates transparency in progress toward a release. The inspect and adapt in the Sprint Review is as accurate as this transparency.

Within the Visual Studio 2010 Scrum process template this means that you have to link or create tasks to a specific ‘Undone Work’ product backlog item. this ‘Undone Work’ product backlog item accumulates the undone work regarding to End-to-End Testing (and others) over time and can be monitored just as other backlog items.

http://www.slideshare.net/xwarzee/path-to-agility-ken-schwaber

Tool Support

In TFS with the Scrum process template we can create Backlog Items very easy.

So, creating an undone backlog item isn’t that hard, the challenge is take it with you every sprint. for example in my undone backlog item I’ve got some undone tasks from sprint 1 and some from sprint 4. The iteration paths for these tasks still point to the sprint 1 and 4 sprints.

This will result in a wrong sprint backlog, because the tasks who point to other sprints as the current sprint won’t appear in the sprint backlog. So, when moving this Undone backlog item to another sprint because not all tasks are done, you also have to change the iteration path for the corresponding tasks.

Just keep in mind TFS doesn't update the hierarchy, you have to do it manually (or use Excel, my favorite).

Past Tips:
01. Getting Testing Done in the Sprint - The Team and Activities
02. Getting Testing Done in the Sprint – Regression Test Sets
03. Getting Testing Done in the Sprint – Test Automation

Next Tips:
05. Getting Testing Done in the Sprint – No Double, Triple Testing
06. Getting Testing Done in the Sprint – PBI Implementation Sequence
07. Getting Testing Done in the Sprint – Risk and Business driven Tests
08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
09. Getting Testing Done in the Sprint – Test Tasks on the Board
10. Getting Testing Done in the Sprint – Done
11. Getting Testing Done in the Sprint - The Customer Test Team
12. Getting Testing Done in the Sprint – The Test Branch
…
…

03. Getting Testing Done in the Sprint – Test Automation

info.nospam@nospam.clemensreijnen.nl (clemensreijnen) — Tue, 24 Jan 2012 21:05:48 +0000

Automate your testing effort, or better your test execution, is a must to get your testing done during the sprint. Automate your test execution shrinks the time needed to execute the tests, which leaves more time for the team to specify new test cases. On the other hand it also brings some inflexibility in your testing, you need to maintain the test automation code and update it when functionality change, which again cost time which the team better could use to specify and execute test cases.

How to.

So, there need some strategy what to automation and how to automate. The decision what test cases are candidate for automation depend on several decisions. For sure the regression set is a good candidate (see Tip 02. Getting Testing Done in the Sprint – Regression Test Sets) which is a collection of test cases based on risk classifications and business value (see TIP 07. Getting Testing Done in the Sprint – Risk and Business driven Tests).

With test automation you also must take in to account that the test cases is automatable, maintainable and has some ROI. Together with this ROI of test automation also comes the design for testability. It should be easy to create test automations, the design of the application should have this in its design and architecture.

How many, how much effort you have to put in test automation has several investment levels. From zero completely no investment till, a very high investment of changing the architecture for testability and automate s many scenario's as possible.

Tools support.

See this video for the Visual Studio test automation support.

Also Visual Studio test automation support has several investment levels, in the presentation below at slide 38 and 39 these levels are discussed. The rest of the presentation discusses the different tactics of test automation with Coded UI.

Coded UI - Test automation Practices from the Field

View more presentations from Clemens Reijnen

Technical meeting automated testing with vs2010

View more presentations from Clemens Reijnen

Past Tips:
01. Getting Testing Done in the Sprint - The Team and Activities
02. Getting Testing Done in the Sprint – Regression Test Sets

Next Tips:
04. Getting Testing Done in the Sprint – Undone Backlog Item
05. Getting Testing Done in the Sprint – No Double, Triple Testing
06. Getting Testing Done in the Sprint – PBI Implementation Sequence
07. Getting Testing Done in the Sprint – Risk and Business driven Tests
08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
09. Getting Testing Done in the Sprint – Test Tasks on the Board
10. Getting Testing Done in the Sprint – Done
11. Getting Testing Done in the Sprint - The Customer Test Team
12. Getting Testing Done in the Sprint – The Test Branch
…
…

02. Getting Testing Done in the Sprint – Regression Test Sets

info.nospam@nospam.clemensreijnen.nl (clemensreijnen) — Tue, 24 Jan 2012 21:04:55 +0000

Some teams rerun every test every sprint, this is time consuming and isn’t worth the effort. Having a clear understanding what tests to execute during regression testing raises the return of investment of this testing effort and gives more time to specify and execute test cases for the functionality implemented during the current sprint.

How To.

So, collecting a good regression set is important. There area lot of approaches how to get this regressions set, most of them are based on risk classifications and business value (see TIP 07. Getting Testing Done in the Sprint – Risk and Business driven Tests).

From “A good regression test is invaluable.”

The principle is that from each test case a collection of additional data is determined into the test cases for the regression test are ‘classified’. Using these classifications all cross sections along the subsets of test cases can form the total test that are selected.

Automation of this regression set is almost a must (see next TIP 03. Getting Testing Done in the Sprint – Test Automation).

Tool support.

Microsoft Test Manager has an interesting capability to control the amount of regression testing that need to be done during the sprint. the data diagnostic adapter “Test Impact”, it give information about test cases which are impacted by code changes

and see page 63 of the training guide http://www.slideshare.net/clemensreijnen/mtlm-visual-studio-2010-alm-workshop

How to create a regression set in Microsoft Test Manager.

Read:
Guidance for Creating Test Plans and Test Suites and read this http://blogs.msdn.com/b/anutthara/archive/2010/09/22/guidance-for-creating-test-plans-and-test-suites.aspx and this Management, preserving and organization of manual test cases within VSTS 2010.

One thing to keep in mind: In MTM2010 the copying of test cases between test plans copies the reference by default and not the test case. So, a test case can be referenced in multiple test plans and suites, changing a test step will change it on all locations.

Creating regression sets for developer tests and functional system tests are mostly more easy. Unit tests are small and run fast, so no problem to run them all. Unit Integration tests are a bit more challenging, you need to balance a bit more what to run when, but both developer tests and functional system tests, aren’t often the biggest bottlenecks in getting testing done in the sprint. When they are, try to do the same approach as mentioned above… balance what to run when.

Past Tips:
01. Getting Testing Done in the Sprint - The Team and Activities

Next Tips:
03. Getting Testing Done in the Sprint – Test Automation
04. Getting Testing Done in the Sprint – Undone Backlog Item
05. Getting Testing Done in the Sprint – No Double, Triple Testing
06. Getting Testing Done in the Sprint – PBI Implementation Sequence
07. Getting Testing Done in the Sprint – Risk and Business driven Tests
08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
09. Getting Testing Done in the Sprint – Test Tasks on the Board
10. Getting Testing Done in the Sprint – Done
11. Getting Testing Done in the Sprint - The Customer Test Team
12. Getting Testing Done in the Sprint – The Test Branch
…
…

01. Getting Testing Done in the Sprint - The Team and Activities

info.nospam@nospam.clemensreijnen.nl (clemensreijnen) — Tue, 24 Jan 2012 20:59:09 +0000

Every Agile ALM presentation the same question is asked: How do I get testing done in a sprint?

There isn’t a one solution/ practices/ to do/ how to fits all for this. So, I made a small list of tips you could do (or recommended to do, or must do… ), which can help to get testing done in a sprint.

Tip 1: Have testing knowledge in the team.

Actually not a tip, it is a must. This is a kind of obvious but not common and I think the hardest thing to accomplish. It is: get testing knowledge available in your team. When you don't have it, you will fail.

How to

Now, it isn’t that easy to accomplish this collaboration. Test and dev are different, a nice quote from this ‘test’blog:

In the D-world, the world of the Developers, we think Generalist Testers are pencil-pushing, nit-picky quality geeks. Mostly they are beside the point and are easily replaced. They seems to like making much noise about little defects, as if we made those errors deliberately....

In the T-world we don't hate the Developers for their perceptions. We are disappointed about the poor quality of the software.
Bad assumptions on the part of Developers are more to blame for the problems than are software weaknesses.
We never(or seldom) get software what will work right the first time. No, in the T-world we think that developers forget for whom they are building software, it looks like they are building for themselves......

Try to combine these two worlds in one team, you definitely need to come up with a Collaborative Culture:

The three most important concerns are:

Trust.
- a topic closely associated with trust when it refers to people is Identity.
Collaborative culture.
- A collaborative culture consists of many things, including:
  - Collaborative leadership;
  - Shared goals;
  - Shared model of the truth; and
  - Rules or norms.
Reward.
- a “reward” for successful collaboration is most often of a non-financial nature.

Show me the value, seems to be the magic word.
Test adds knowledge, knowledge during the grooming the backlog. Helping the product owner with defining proper acceptance criteria. And, help find improper written backlog items with inconsistencies for example. (this is my favorite test activity I didn’t know about before started reading TMap, Assessing the Test Base). In both ways helping the product owner and the team to focus on value.

Tools support, Visual Studio 2010

Tools can help, when using the VS2010 TMap Testing Process template you will notice that the test activities get an important place, helping the tester to get on board.

The ‘Testing’ Tab with test activities, next to the implementation tab.

Still two different worlds, test and dev in this way… but it give a good visual reward of being connected. Probably many teams won’t need an additional visualization of testing effort and can use the scrum process template in combination with their testing methodology, but this will help them to get started.

Interesting topic is the introduction of the manual test tool ‘Test Manager’ with Visual Studio in the current release. It helps teams to get more connected, it shows the pain points where the collaboration isn’t seamless. So, adopting MTM can be a good start for agile teams to get testing aboard. But, be aware the interactions are more important as tools. The tools won’t fix bad collaboration, mismatching identities, lack of trust and won’t give any reward.

Next Tips:
02. Getting Testing Done in the Sprint – Regression Test Sets
03. Getting Testing Done in the Sprint – Test Automation
04. Getting Testing Done in the Sprint – Undone Backlog Item
05. Getting Testing Done in the Sprint – No Double, Triple Testing
06. Getting Testing Done in the Sprint – PBI Implementation Sequence
07. Getting Testing Done in the Sprint – Risk and Business driven Tests
08. Getting Testing Done in the Sprint – Write Logical Acceptance Tests
09. Getting Testing Done in the Sprint – Test Tasks on the Board
10. Getting Testing Done in the Sprint – Done
11. Getting Testing Done in the Sprint - The Customer Test Team
12. Getting Testing Done in the Sprint – The Test Branch
…
…

Teams and TFS Groups in TFS11, the backlog, board and security settings.

info.nospam@nospam.clemensreijnen.nl (clemensreijnen) — Fri, 20 Jan 2012 09:48:23 +0000

Back to the ‘teams’ topic, it is new in TFS11 … (see below the previous posts about this topic)

Beside using the new team concept in TFS11, you also just can do it the ‘old’ way. Put all your team members in the different TFS Groups (readers, contributors, … ) and give them access to the team project. Just as you did the past decade. But, you have to understand what happens when you do it. You also can mix things, create teams and only add TFS groups to it, or give people specific rights within a team… actually you can make it a complete chaos. In this post some usages scenarios. (at the end of this post a final conclusion, maybe you want to read that one first.)

Scenario 1: A team project with no teams and only team members in TFSgroups configuration.

When you create a team project, you get by default a team with the team project name, prefixed with ‘team’. This default team has one member, the creator of the team project. Which as you can see I deleted.

When you browse to the administration page, you will notice that you can’t delete the default project. but, with no members it can’t do any harm.

Next, you can start adding members to the different TFS groups. For example I added several people to the contributors group.

Following step, make security settings for the different groups... for example for the iterations

or the areas. You can find it at the same places as you can in VS2010.

Interesting is that the security settings for the source repository only can be set in Visual Studio.

When you have a team project without a team (beside the default team), you do have a backlog, with a board.

Everything works as expected, only the capacity planning seems to be broke. The ‘only members in tfs groups’ project gets a bit more complicated when you start working with area’s. For example add two sub-areas and mark one of them as current.

Good to know when you mark the parent area as the current one you can set a child area as default. When adding backlog items by using the backlog it will automatically set the area property.

Doing this and adding PBI’s and backlogs to the this area. Will result in a new backlog and new task board, helping the team members (which are only in TFS groups) to focus on this specific area only. Set the area setting to the parent area, including all sub area.. will result in a combined backlog.

The only thing what stays the same when changing the areas for your team project are the Team favorites (we don’t have a team :) and the queries behind it don’t have a current area path clause. You could add it yourself.

Conclusion

It is possible to create projects without using the new Team concept. Things look a bit strange, we have team favorites but no team members for example. But the main functionality works. It only gets a bit complicated when you start using areas and make different people responsible for different areas. You have to swap between the admin page and the home screen when you want to see a different backlog / functional area.

Scenario 2: teams only

I created two additional teams for the two different areas, added the team members to the default team and added the default team to Team 1 and 2.

All teams are in the contributors TFS group, resulting in the fact that all team members have the same security settings.

Having you teams setup in this way, gives them the easy capability to swap between the teams. The backlogs are up to date and the team capacity planning works. One annoying thing, the team favorites, the work item queries, are the same for the whole project so I have to create queries for every team and add them as team favorite.

Scenario 3: teams with tfs groups

Another scenario, for playing with security, teams and TFS groups is… adding the team members to TFS groups and add these groups to the teams as members. Now, you have members in your team with different security settings. Doesn’t feel really scrum, but it works.

Only the TFS Groups Readers and Contributors in the the default team and Team 1 and 2 have only the default team as team member.

Scenario 4 and 5 Teams with team members with specific security settings.

The last scenario, is where you add only users to your team and set explicit security settings per team members.

I won’t recommend this, you really can’t administer this.

Final Conclusion.

So, after this small journey across the different possibilities of configuring security settings in TFS11, I must say… Be careful, you can create chaos which you aren’t able to administer anymore, and think up front about your areas and team structure, it influence the usage and capabilities of the nice agile tools.

Scenario 1: No Teams only TFS groups with Team members.
You can use the backlog and task board, the capacity planning doesn’t work (yet?). It gets complicated with multiple areas and multiple backlogs. For example when you want to use a bug backlog, don’t use this scenario go for number two or three.
Scenario 2: No TFS groups only Teams with Team members
This feels like the only real agile solution, all team members have the same security settings and the usages of multiple backlogs is easy.
Scenario 3: Teams with TFS Groups with specific security settings
When you want to have team members with different security settings, this is the way to go. Areas and multiple backlogs are fully functional including the capacity planning.
Scenario 4: Teams with Team members with specific security settings
Forget this one… you will go crazy over time
Scenario 5: Teams with Team members with specific security settings and with TFS Groups with specific security settings.
You not only will be crazy you probably will have to go to the hospital when you try to maintain this scenario.

For now… number three is my favorite, number two is more pure… one is ok, four and five not ok.

TFS11 Scrum board update remaining work without opening the work item…

info.nospam@nospam.clemensreijnen.nl (clemensreijnen) — Thu, 19 Jan 2012 11:45:14 +0000

Very nice MSFT added the capability to update the remaining hours without having to open the work item. and even more nice it has some intelligence behind the drop down, because it populates the dropdown list with values in range.

TFS11 on Azure January 2012 update – request feedback

info.nospam@nospam.clemensreijnen.nl (clemensreijnen) — Wed, 18 Jan 2012 05:19:50 +0000

The latest update on TFS ( see: Team Foundation Service Planned Maintenance Tues Jan 17th ), fixed an issue with builds, they look stable again. And my Azure build service executes them happy. Now its time to expand this scenario with test controllers and test agents for automatic test execution and test environment provisioning.

But, I want to point to the new menu item ‘request feedback’ … it is the feature for getting early feedback about your product, for example during the sprint review.

It is a really nice feature supported with a tool similar as the Test Professional, Test Runner.

More explained with a walkthrough for the dev preview can be found in Brian Kellers dev11 VHD and the handson lab: Building the Right Software - Generating Storyboards and Collecting Stakeholder Feedback with Visual Studio 11 (docx).

How it works, I can start asking feedback … about a feature I just created. For example in this screen I’m asking if Brad the Product Owner can look at the website for typos.

It generates an email for Brad, asking to start the feedback session.

Now this is where TFS Azure isn’t complete, it can’t send the mail and the feedback tool URL points to the MSDN subscriptions download page. But, when you look at the Brian Keller hands on lab you know what happens.

there isn’t a feedback work item type..

A very promising feature, special in distributed teams and with TFS on Azure we can ask any one for feedback.

So, get your product owners informed… they will get involved in dev11.

Deck: The agile cloud development scenario in 10 acts…

info.nospam@nospam.clemensreijnen.nl (clemensreijnen) — Fri, 09 Dec 2011 14:10:59 +0000

The presentation from last SDN event … about Hosted TFS

TFService

View more presentations from Clemens Reijnen.

Coded UI Test Automation Practices Deck

info.nospam@nospam.clemensreijnen.nl (clemensreijnen) — Fri, 25 Nov 2011 07:49:43 +0000

Had a great time in Munchen during the Microsoft Visual Studio ALM Days 2011, below the deck I used during my session.

Coded UI - Test automation Practices from the Field

View more presentations from Clemens Reijnen

the \final notes slide\ is a scrolling screen… her are the topics

keep it simple
DRY
start automating from the beginning
developer and test engineer in same room
don’t automate everything
know what already have been verified
run them always
fail fast fail often
learn as a team
tools will help you
code rules for better testability
test automation in the same sprint
start from scratch or tune your manual tests
make clear test method names
separate the test intent from the test steps
no related tests
all test must leave the application in the same state before the test started

some photo’s of the alm days… I lost the glove on the photo, so if you found it… ping me

(update about the lost glove: Brian Harry found him)

ALMDays

VIEW SLIDE SHOW

DOWNLOAD ALL

By: The SEMAT Vision (Agility@Scale: Strategies for Scaling Agile Software Development)

Mon, 31 Oct 2011 11:10:39 +0000

[...] are several reasons why I'm involved with SEMAT:The industry clearly needs something like this. Ivar Jacobson has been writing and speaking for awhile now about how our industry behaves in a similar manner to [...]

kicking-off Architecture Tooling Guidance Refresh 1.1

Sun, 26 Sep 2010 05:57:00 +0000

This Monday we are kicking off the refresh of the Visual Studio Architecture Tooling Guidance.

If you are a subject matter expert or have feedback on the previous release, please contact us!

(More info about this refresh can be found on Willy’s blog)

[DUTCH] Artikel over Extensibility in Visual Studio 2010

Sat, 25 Sep 2010 14:54:16 +0000

Interesse in ‘Extensibility’ in Visual Studio 2010? Lees dan dit artikel van Clemens en mij dat onlangs gepubliceerd is op de web site van het .NET Magazine. Hierin behandelen we het ‘Modeling Feature Pack’ en de ‘Feature Builder Powertool’…

Visual Studio 2010 Architecture Tooling Guidance

Mon, 28 Jun 2010 19:05:08 +0000

Interested in Visual Studio 2010 and Architecture? Check out the latest delivery of the Visual Studio ALM Rangers. Some more info on this product can be found in this post from Willy, our Team Lead in this very interesting project. It was great fun working on this piece of guidance!!

The Rangers involved with this project are: Alan Wills (MSFT), Bijan Javidi (MSFT), Christof Sprenger (MSFT), Clemens Reijnen (MVP), Clementino de Mendonca (MVP), Edward Bakker (MVP), Francisco Xavier Fagas Albarracín (MVP), Marcel de Vries (MVP), Michael Lehman (MSFT), Randy Miller (MSFT), Tiago Pascoal (MVP), Willy-Peter Schaub (MSFT), Suhail Dutta (MSFT), David Trowbridge (MSFT), Hassan Fadili (MVP), Mathias Olausson (MVP), Rob Steel (MSFT) and Shawn Cicoria (MSFT).

Model Based Testing (part 1)

Rob Kuijt — Sun, 27 Jun 2010 06:11:07 +0000

Model Based Testing is a powerful concept for testing information systems, from which, despite the increasing complexity, more quality and flexibility is expected. It is a methodology that fits well with companies with experience in structured design and testing. Model Based Testing will also support and lower the threshold for companies who want to make a first step towards improving their test basis towards a better, faster and cheaper test process.

The core of Model Based Testing is the use of tools to first, create the actual design and secondly to compare the simulation of the process and predicted outcome of the test generation with the requirements at an earliest possible stage. Thus, the various "paths" through the procedures / processes including related business rules are tested and checked much earlier.
Subsequently, automatic test generation creates efficient and reliable high quality test cases which excel in compactness and low cost.

Model Based Testing helps to improve the overall development process: the impact of demands and wishes becomes clear in a fast manner, design errors are found at an early stage, debugging code caused by wrong assumptions and interpretations decreases and reuse of designs in new projects is easier.
In short, developers achieve greater results in less time and at lower cost.

The tools needed to support Model Based Testing are available and are developing rapidly based on the experience of cutting edge companies like Sogeti. It becomes increasingly easy for companies to make the choice and take advantage of Model Based Testing.

Rob Kuijt

Cloud computing and Application Lifecycle Management

Wed, 17 Mar 2010 10:27:51 +0000

Recently, I noticed that the in last couple of months the amount of of time that I spend on cloud computing (Microsoft Azure in particular) is increasing quite rapidly. I am currently involved in a few initiatives/projects around Microsoft Azure and I suddenly realized that cloud computing has a positive impact on the way we think about Application Lifecycle Management (ALM). When we think about cloud computing from a project delivery or operations perspective (and not only from a technical perspective) there are absolutely interesting advantages that cloud computing can bring us.

[For more information about the way we think about ALM and a better understanding of the image I am using below, have a look at this post]

From a project delivery perspective, a cloud project has the advantage that we don’t have to buy the hardware and software that we need during for the development, testing and running the application in production (because we use the compute and storage power from the cloud). Of course we have to pay for using the Azure platform but these costs are likely less compared to buying bare metal and licenses for the complete lifecycle of the application. Another advantage of using a cloud platform is the time it takes to get our environment approved and up and running which potentially decreases the time to market of the application. If we have a look at the image below which represents the lifecycle of an application we can see that less hardware and an environment that doesn’t take long to get it up and running have a positive impact on the application lifecycle of the application. From a project delivery perspective lower costs (less hardware and licenses) for the project and a decreased time to market have a positive impact on the complete lifecycle of the application (represented by arrow 1 in the image below).

Further, one of the important goals of ALM (at least in our opinion) is increasing the added value of the application for the business user. Cloud computing enables some very interesting scenarios that potentially bring a lot of value to the business user. For example, the unlimited scalability of the platform (at relative low costs) makes it possible to deliver completely new (business) services for markets that couldn’t be reached that easily in the past. Also integration between companies, networks, applications, etc. becomes much easier with applications running in the cloud. The new scenarios that we can deliver by using cloud computing potentially add extra business value to the applications we are delivering and therefore have a positive impact on the complete lifecycle of the application (represented by arrow 2 in the image below).

From an operations perspective, the fact we have less hardware and software to maintain (back-up, monitoring, patching, etc.) also has a positive impact on the lifecycle of the application (represented by arrow 3 in the image below).

So, besides all the technical challenges of cloud computing that I am very interested in, I like to think of cloud computing as “just another delivery form” for our software development projects with a very interesting positive impact on the application lifecycle of the application that we deliver.

Layer Diagrams: Application Architecture Guide

Mon, 15 Mar 2010 08:30:14 +0000

A few days ago Microsoft patterns & practices released a (preview) set of Layer Diagrams for Visual Studio that comply with their Application Architecture Guide 2.0. The diagrams are included in a simple VSIX package that can be downloaded from the Visual Studio Gallery. The image below gives you some idea about what to expect from this package. The diagram also contains a link to the complete architectural guidance on MSDN.

The layer package is a great start but there might be more opportunities for integrating architectural guidance in Visual Studio 2010. Right now, the package contains 5 layers diagrams so it shouldn’t be too difficult for an architect to decide which of the layer diagrams he should choose from the toolbox. But, what if we have additional layer diagrams or in addition to the diagrams we also have predefined architectural inspections and/or validations that the architect can choose from? In that case we might need additional guidance to help the architect choose between the various diagrams, validations and inspections. Below you can see a screenshot of a prototype we did in this direction where we use a WPF form to ask the architect some questions. Additional questions pop up based on the selection the architect makes (clicking yes or no on the form) to help him decide what exactly he needs for his architecture. In the end, based on his selections, the the environment is prepared for him.

This is just a prototype and not production ready but it might give you some ideas about how we think we should provide guidance. In the past couple of months Clemens and I have done some work on these ideas but we haven’t really finished it yet. Now Visual Studio 2010 becomes close to RTM it is time to finish this so expect some updates in this direction.

Stay tuned!

History of Test Automation

Rob Kuijt — Sun, 10 Jan 2010 20:44:14 +0000

Test Automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, test design, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process. (Wikipedia)

Working with Model-Based Testing I get many questions on Test Automation. To be honest, around a year ago, I could not answer them at all. After a year playing with test industrialization my knowledge is better. But still, especially the timeline (when was a specific type of test tooling introduced) was a gap in my mind. An overview: History of Test Automation would be very practical.

On internet, I couldnt find the complete Test Automation timeline I needed, so I started creating one

Inspired by the powerpoint slides: Generations of Test Automation by dr Mark Utting, I made a slide show (kind of e-learning component) which gives an overview how the (automating of the) test process did evolve since the 1940's.
I created some nice graphics, added some definitions and expanded the presentation with more detailed slides and background information from TMap®, Wikipedia, James Bach, William E. Lewis, Gunasekaran Veerapillai and Marvin Niven (thanks guys!) and uploaded the slide show/e-learning component for future use

Interested? See History of Test Automation

Test Case Design: Manually or Automated?

Rob Kuijt — Sun, 20 Dec 2009 06:42:37 +0000

There are several tools that can help with the creation of test cases. So why don't we succeed in automating the Test Cases Design Process? In this short article I will explain that, if test collaborates with design, we can make huge progress on this topic!

4 Examples of Test Case Design
Lets imagine a business process that is documented by the design team on one page. Test will create the test cases. In the first example the description is done in plain text. In that case it is not possible to automate the test, and also it is not possible to use a formal test specification technique:

Example 1: Test Design from plain text: many interpretations and assumptions...

Creating test cases from plain text may look easy, but when you ask 5 test engineers to create the test cases you get 5 different sets without any insight in the quality of the coverage.
Choosing the process flow diagram technique for describing the business process, will result in less interpretations and assumptions, therefore in much better test cases:

Example 2: Manual Test Design: Process Cycle Test (TMap®)

The advantage of formal Test Case Design techniques like Process Cycle Test (PCT) is that it can be automated! How? The first step of deriving test cases with PCT, is to identify the paths and path combinations within the process flow diagram. Step 2: Instead of manually combining those path combinations to test cases, the path combinations (joined with a short description) can be inserted in a Test Design Tool. Step 3: The test cases are generated. Comparing with manual test case design: much faster; less knowledge is needed, but you need a tool* (license):

Example 3: Using a Test Design Tool: much Faster!

Most of the time, designers make these process flow diagrams in modeling tools (like MsVisio, Protos, Aris, BiZZdesigner,..). A common feature of the modeling tools is exporting the models in XML-format. Model Based Testing tools* do read XML! So, lets skip all manual steps and generate test cases directly from the process flow diagram:

Example 4: Model Based Test Case Design

Model Based Test Tools* can generate test cases within minutes. Of course, before using the test cases you must do a sanity check to confirm the tool understood the model correctly but nevertheless the time cant be beaten manually.

Collaborate!
Test Case Design can be automated (Im already working with these tooling on a daily basis).
But, besides getting the right tool(s), there is an important condition: the Test Base must have a minimum level of quality.
For instance: instead of describing business processes in plain text, they should be specified with the help of activity diagrams or process flow diagrams (and that is not (yet) common knowledge within the design processes). To get an automated Test Design process, lets join the forces: Together, Design and Test can make projects much faster and cheaper!

*) Example of a Test Design/Model Based Testing Tool I personally often use: STaaS/COVER (Sogeti Netherlands)

More return on development with Application Lifecycle Management

Tue, 24 Nov 2009 14:58:09 +0000

This is a translated version of an article that I wrote for Software Release Magazine.

Application Lifecycle Management

In the past ten years, the costs of IT projects dropped significantly. In addition, the number of projects that turned out to be successful, rose. Nevertheless, only forty percent of all IT projects succeeds. This means that it takes less time today for a project to fail. Application Lifecycle Management (ALM) may help improve the return on projects. An efficient deployment of ALM requires the right scope and focus.

Today, many organizations regard Application Lifecycle Management (ALM) as one of the answers to their bad performing IT departments. With ALM, they try to get more grip on software development by integrating, coordinating and controlling the various phases of development. ALM guides an organization from software development until software implementation and management. Very often, an organization will limit its focus to optimizing the developer’s work processes and the communication between developers and project managers. An ALM tool is rolled out and its features are used to manage the progress of the project as well as the quality of the code. Deploying an ALM tool in this way is a step in the right direction but in practice it is not a guarantee for success. Without the right focus, software development will remain a stand alone activity without any relation to other parts of the organization, including business and operations. Additionally, research shows that companies spend on average 30 percent of their available IT budget on newly built applications. This means that they neglect an area of 70 percent, where optimization is also possible.

More return by shift in focus

With the right scope and focus ALM has much to offer. Figure 1 shows a schematic view of the lifecycle of an application that is to be developed. In this view de x-axis represents the time and the y-axis the value/cost of the application. In this figure, the extended curve shows the lifecycle of the application from development until end-of-life. The figure provides insight in the different phases of an application’s lifecycle. It also enables for determining the impact of ALM.

Figure 1. The lifecycle of an application

Reduce development costs

The lifecycle of an application starts in the first phase of development. As of the start of the project, costs are made for design, programming and testing. At this time, the application offers no value. All development costs are therefore to be regarded as costs. Often, organizations focus on these costs when deploying ALM and optimizing the software development process. Figure 1 shows however, that the development phase only accounts for a small part for the full lifecycle of the application.

Time to market

The figure also shows that the application will only add value to the organization when it has gone live. This means that the ALM activities need to be focused on getting the application (or a part of it) live as soon as possible. One of the ways to do this is using agile development methods, including iterative delivery. Shortening the time to market not only results in faster added value, it can also provide competitive advantage. In general, organizations that are first in addressing new needs or market changes profit the most from these developments. Organizations that are trend followers profit less; or even worse, they have to invest in order to stay in the market.

Added Value

It is clear that an application will add value when it has gone live. Many organizations do no recognize this added value. ALM activities should focus on getting as much added value as possible. As the application is developed for the end-users , it is key to involve this group as much as possible in the development process. User involvement, support from the (executive) management, defining clear business goals and optimizing requirements are all equally important.

One of the ways to realize this is to optimize the communication between the user organization and IT and to create a common involvement for all stakeholders. In this way, stakeholders are better geared to state their demands. They are also better able to determine the consequences of their choices and to change priorities and requirements during the project, together with the project team. Here, it is also cost-effective to use short iterations, as changes in scope, priorities and requirements can easily be made. In this way, an organization can address new insights during the project, which may increase the added value of the application further.

Operational costs

When developing the application it is advisable to acknowledge in an early stage the need for application management. By creating consensus on the requirements for the management department, operational management costs of the application can be reduced significantly. The focus on management is paramount in an ALM approach.

Extending the lifecycle

By adding value and reducing costs, a developed application will provide return in the long run. By focusing on the value of an application and by constantly monitoring this aspect, an organization is better capable of determining the moment when value is replaced by costs. This early insight helps in deciding what to do with the application: adjust or phase out?

Phase out

When an organization decides to abandon an application, the knowledge of the application will lead to less abandoning costs. This is definitely the case when an organization combines this knowledge with the optimizations, provided by ALM in the earlier development phase of the application. An example is the right documentation on the application interface to other systems.

Figure 2. The new lifecycle of an application

The new lifecycle

Figure 2 shows the new lifecycle of the application. This is the result of broadening the ALM focus as described earlier. The green field in the figure depicts the extra return of the application. This is possible by speeding up the go-live process, a longer life of the application and more added value for the user. The red areas in the figure represent the decreased development costs and the costs for abandonment.

Priorities of an organization

Figure 2 shows that the return on ALM increases when an organization not only focuses on the development phase, but also on the other phases of an application’s lifecycle. This optimization obviously pays off, but the question is how to relate this to the goals of today’s organizations. Many organizations focus on cost reduction, compliancy and risk management. How can ALM be related to these three priorities?

Cost reduction

A too strong focus on cost reduction may well lead to an imbalance in this area, resulting in a paralyzed organization in terms of productivity. By continuously executing cost reducing measures, tools and communication channels are lost. In the end, this can affect the productivity of employees negatively. By combining cost savings and productivity improvements this issue can be addressed.

A combination of ALM activities with a so-called high-performance workplace can be the answer. The high performance workplace is a physical or virtual environment which is especially designed for knowledge and information workers. It supports them optimally in executing non-routine duties. In these duties, exploring, learning, innovating, collaborating and managing are key.

The current generation of ALM tools already pays attention to optimizing communication and collaboration between stakeholders in an IT project. This shows that the awareness on effective collaboration is growing. It also requires the focus to be placed on the process and the human aspect of collaboration. Important success factors are creating joint goals and making sure there is a shared vision of the truth.

Compliancy

The current compliancy requirements demand a high level of control. In recent years, this control is expressed in continuous process optimization. The need for a process approach led to reduction of flexibility within an organization. It is becoming increasingly difficult to address market changes and needs without losing control. The focus on processes, tools and management also led to a situation in which up to twenty percent of project costs can be allocated to the developed software. The rest of the costs are related to project support and meeting internal and external requirements. The new generation ALM tools and corresponding methods, enable an organization to meet these strict compliancy requirements without affecting flexibility. These tools provide the necessary mechanisms and means of control to link requirements, quality metrics and the solution. This makes it much easier to prove that the solution meets all requirements. The full support of agile methods within the ALM tools ensures the required flexibility.

Risk management

Experience from the past shows that risk is inherent to projects. However, the higher the risk, the higher the return, as optimists say. It is not necessary for organizations to exclude all risks. They need to assess risks continuously on the return they can provide. The collaboration in a project, the commitment from stakeholders and the combination of business and ICT knowledge enable the right assessment of risks. This may lead to a situation in which a risk that is regarded as unacceptable by individual members, is controllable or even desirable.

Conclusion

ALM is gaining in popularity. Many organizations take their first steps in this area and start to purchase ALM tools. Seemingly without thinking, they focus on the development phase. This is an excellent first step but still they should not stop here. By broadening their focus and incorporating the full lifecycle of an application in their approach, they are able to increase their return on ALM significantly. The broader approach offers more insight in the added value of the application. By combining ALM and a high performance workplace, and by putting the human aspect first, it is possible tot create an environment in which collaboration is optimized. The result is a software development process with predictable results and sufficient flexibility to contribute to the three main priorities of an organization: cost reduction, compliancy and risk management.

ALM and MBT

Rob Kuijt — Mon, 19 Oct 2009 17:08:29 +0000

Both ALM (Application Lifecycle Management) and MBT (Model Based Testing) are hot. I'm pretty busy running from customer to customer starting/supporting MBT proof of concepts and assignments. With some explanation all my contacts are convinced that the combination of ALM and MBT is THE roadmap for the future.

The advantages of ALM can already be implemented by choosing for the new generation of tools of Microsoft (VSTS2010) and IBM (JAZZ etc.) are presenting to strongly support the collaboration of the different roles in and around the ICT. Model Based Testing has not yet that same level of maturity. Most participants in the Model Based Testing world are very ambitious: They try to Generate Test Cases for direct automatic execution, based on test specific models.

I think this is a bridge to far for most companies using for instance mainly ERP-software. In the ERP-scope most of the testing is still done manually, and jumping to this ultimate form of Model Based Testing (Generate Test Cases for direct automatic execution) will be very tricky and/or costly. That's why we (Sogeti) are introducing also a basic form of Model Based Testing (STaaS/COVER), using the same models as the developers do.

"MBT: ...a bridge to far..."

MBT: Generate Test Cases for direct automatic execution
Does this mean that Generating automated tests is a dead end? NO! It is a very good approach for testing high risk functionalities, which are to risky or to complex to test manually.
Making test specific models is a costly business and top designers/test engineers are needed to make it work. So be sure to pinpoint these heavy coverage and tooling only on the high risk areas. AND!! Do not try to make the test specific models in the Testing Silo! Look for collaboration with the design department to get these test specific models in the same configuration mnagement as the models the devlopers use to build the software. Preferably in an "ALM Center Configuration Management".

MBT-basic: Using the same models as the developers do
On the other hand, when the ambition of Model Based Testing is lowered to only generating the test cases, it is a complete different story. Especially the generation of the (more global) logical test cases can be done from the same (functional and/or requirement) models the developers use as a bases for creating the software. For instance process flows (activity diagrams), decision tables and pseudo code are a perfect basis for generating (logical) test cases.
Enrichment of these models (in collaboration with the design department) will bring us to the next step: Generating the physical test cases. In that case the Test Cases are no longer Configuration Items (costly maintenance!) but Test Cases become Work Items (if desired: directly generated from the same models the developers use to build the software).

Rob Kuijt

ALM: the end of the whispering game

Rob Kuijt — Sat, 17 Oct 2009 18:35:06 +0000

Who don't know the whispering game? For me it was one of my favourite games in my childhood. Whispering a tricky word in the ear of the child next to you and waiting for the twisted result at the end of the cycle....

Nowadays not only children do play this game. In the ICT world practically everyone is playing this game. Only they invented a new name for it: The Waterfall Model .

ALM (Application Lifecycle Management) makes an end to the twisted results of this widely implemented whispering game. Instead of working in Silos with their own project and configuration management, in ALM the requirement, workflow and configuration management is organized in the center of the process. So everybody is working on the same artifacts.

In ALM everbody is working on the same artifacts. No whispering risks at all.. ;-)

And from my archive..
Some famous twisted results of the whispering game:

WSCF.blue Beta 1 is out!

Sat, 04 Jul 2009 20:31:37 +0000

I Just wanted to let you know that we have just released the Beta 1 of WSCF.blue. This great tool supports a Contract First Approach for developing webservices in Visual Studio 2008. Some time ago, I took the project lead (together with Christian) for this great tool. Unfortunately, I have been kind of busy lately so we didn’t made a lot of progress in the last couple of months. Just recently a couple of new members joined our team which resulted in this Beta 1 release. In this release we added MSI support which was one of the key requested features for this tool. For some more info on this release I suggest to have a look at this post from Benjamin (one of our new team mebers!). Our new team members inspired us a lot and brought some great ideas so expect some cool new features soon! Let us know what you think of this release on our forum.

Thanks Buddhike, Benjamin and Alex Meyer-Gleaves for getting this new release out!!

Model-Based Testing is not "Only Fun" anymore....

Rob Kuijt — Sat, 20 Jun 2009 12:09:24 +0000

For several years now, Model-Based Testing is mainly used within the (technical) development processes. In the Black Box testing community it was still "just a theory", and in my case: fun to play with. Today is different! I'm completely into Model-Based Testing nowadays. MBT turned, for Black Box testers, into serious business!

Wikipedia:
Model-Based Testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT).

For quite some years I'm working with a tool called COVER. Initially COVER was build to help lazy testers, like me, to avoid boring manual activitiesand it worked! Using this tool I could easily amaze my surrounding with stunning Speed and Quality. Especially deriving test cases from formal specifications like pseudo code or activity diagrams was fun to do; instead of "solving a boring puzzle" time, it was done in seconds (as a matter of speaking). However, most of the testers in my surrounding didn't recognize the ease of this way of working (or were not lazy enough?) so COVER didn't have much users.

Today is different. The attention for Model-Based Testing is growing rapidly. Previously, when a project had to make test cases, it was accepted that it was done manually and, everybody knew, it should take quite some time.
Today the budgets are lowered...What to do?...Less productivity?
My suggestion: When you've less money to spent, be creative to get the same results!

First Model then Test!

Model-Based Testing is a nice example of creative thinking!
Within a few months the relatively unknown possibilities of Model-Based Testing tools became very popular for Black Box testers within Sogeti. Especially the collaboration-part (designers working together with testers) of Model-Based testing is an eye opener. Everybody knows that it is smart to find defects as early as possible, but it is very difficult to motivate teams to perform good design inspections. Now MBT tools like COVER can do a big part of those intensive inspections, because using the models for generating test cases will directly show the defects in those models in a very early phase of the project! And beside that, it generates a big part of the test cases!

So, when the Designers create models that COVER understands, projects will perform Better, Faster and Cheaper! ....and it works!
Some (early) successes from our proof-of-concepts and implementations:

The amount of time needed for creating test cases dropped 50% (and sometimes more);
The maintenance of the (regression) test cases became in average 70-80% cheaper;
And maybe most important: We have proved to find defects much earlier than the project did manually!

Collaboration will change the world of testers!
Using MBT is in fact collaborating within the Application Lifecycle [ALM] supported by standardization and automation. Many defects, found in functional testing, have their origin in misinterpretations and assumptions of the requirements and/or specifications. The early usage (generating test cases) of formal models, just after or during the design phase, will create a much more solid base for software coding. For the testers this will mean less and less defects, so besides the generated test cases, the improved quality of the software will also fasten the test process.
(PS. When both software and test cases are generated from the same models, be sure that the test objective is clear!)

This industrialization cant be stopped anymore. The time of creating most test cases by hand has had its peak. Of course, not all test cases can be generated, and also manual testing itself will never disappear, but handcrafted test cases will be less and less common practice. The work of Black Box testers will, in my opinion, shift in two directions: 1) Earlier in the life cycle: joined modeling and helping and supporting development to find defects as early as possible (Master Test Plan consultancy), or 2) the coming market End-to-End testing: the importance of complete chains of applications is growing rapidly. End-to-End Testing will become a separate specialism.

Examples
Back to Model-Based Testing: To give an idea what type of models can be used, and how test cases are generated, I've made a some slides (with examples) about the Sogeti MBT tool: COVER. Look for yourself if MBT will affect your work!

It did affect mine!
;-)
Rob Kuijt

Architectural Inspections: Implemented in Visual Studio Team Architect 2010

Thu, 28 May 2009 20:05:24 +0000

Currently, Clemens and I are writing a whitepaper about Architecture, Application Architecture Guide 2.0 and Visual Studio Team Architect 2010. (VSTA). In addition to this paper we are also working on some ‘tooling’ that we plan to deliver with the paper. Since we are not done with the paper and tooling yet and this blog becomes a bit too quite I decided to start sharing some of our thoughts and work in this space on this blog.

One of the topics in the paper is, what we call, ‘Architectural Inspections’. Without going into too much details just yet we can think of an Architectural Inspection as a ‘check’ to help us verify the correctness of (parts of) an application architecture. The concept isn’t totally new, in fact the Application Architecture Guide 2.0 comes with an organized checklist that sums up important inspections that an architect can use during the design and/or validation phase of an architecture. Although a checklist is a great start, we think that a standalone checklist doesn’t get the most out of these so called Architectural Inspections. In our opinion it will be much more powerful if we can include these inspections in our Application Lifecycle Management practice, integrate them in the Visual Studio IDE and provide the right guidance at the right moment!

To validate our thinking, we collected all the inspections in the Applications Architecture Guide 2.0 checklists and stored them in an XML format. In fact, we used the Team Foundation Server 2010 (TFS) Work Item Type XML format which enables us to easily upload our Architectural Inspections into TFS as work items. In addition to the ‘core’ Architectural Inspection data, like title, status, description (where we explain what we need to validate and can add additional guidance) we added some meta data to categorize our Architectural Inspections and make it possible to do some grouping. For example, we can categorize our Architectural Inspections per ‘Cross Cutting Concern’ (Logging, Validation) or ‘Layer’ (Service Contract, Business Logic, etc.) or ArchType (Mobile, Rich Client, Service, etc.), or whatever we think makes sense. In addition we have build a little tool that lets us upload these Architectural Inspections into TFS as work items. Currently we store our Architectural Inspections as normal ‘Task’ work items and abuse some ‘hidden’ fields to store the meta data that we need. However, we already realized that we are better of defining our own work item type for our Architectural Inspections. So, this is probably the next thing on my ToDo list…

Below you can see a screenshot of (a very basic prototype of) the tool that we are using to upload our Architectural Inspections into TFS. As you can see we haven’t spend too much time on the User Interface yet and the data in the screenshot is just dummy data that doesn’t make too much sense.

However, the most important thing right now is that by using a tool like this we (as an architect designing an architecture) can easily decide which Architectural Inspections make sense for the architecture we are designing and add only those inspections into our Application Lifecyle. This means we can, for example, add only those inspections that apply to the layers or cross cutting concerns that our architecture requires. (In a future post we will demonstrate how we can even relate the inspections to layers in our Layering Diagram.)

Another thing that we think is important is to have a clear overview of all the inspections that are considered and/or executed during the design and/or implementation of the architecture of the application. Knowing that the guidance and best practices of a particular inspection wasn’t properly implemented or worse totally neglected is important information and (potentially) tells us something about the quality of the application. Of course sometimes it makes perfectly sense not to spend time on cross cutting concern X. However, at a later time we can’t recall the reasons for not spending effort on them. The fact that we now have our Architectural Inspections stored in TFS (as work items) makes it possible to track the current status (by using the status field (Active, Closed, Rejected?) )and provide us with valuable information about the design decisions (captured in the description field?) that are made during the lifecycle of our application.

Last but not least we think that, to get Architectural Inspections fully integrated in the Application Lifecycle, we need a proper way of visualizing them. In fact, an overview of these inspections and their status might be good starting point for a quality check or valuable input for our testers. The most common way for visualizing the status of work items would obviously be to create a report in TFS. However, we thought we better get some experience with another cool new feature of VSTA 2010 so we decided to visualize our inspections in DGML. So, what we did is, we create a little utility that extracts the Architectural Inspection information out of TFS and generates a nice DGML diagram for that. Below you can see a screenshot of how our first implementation of this looks like. (again, we might need some UI improvements and some real data)

The little icons in the nodes (representing an inspection) display the status of the inspection. At this moment the green check means the inspection has the ‘Closed’ status in TFS and the warning sign means it has the ‘Active’ status (so nothing has been done with it yet).

There is a lot more to tell about the things we have been working on and the thoughts we are still having about Architectural Inspections, Application Architecture Guide and VSTA 2010 extensibility. We are currently busy improving and refactoring all of the above. In the coming period we will share some other VSTA extensions that we are working on and if things goes as planned everything will end up in the whitepaper and/or downloadable assets. So, stay tuned and of course we are very interested in your opinion, concerns, etc. so leave us a message!

Visual Studio Team System 2010 - Episode 4: Quality Check

Rob Kuijt — Fri, 24 Apr 2009 08:04:09 +0000

In this episode I will discuss the different practices around the Quality Check in order to do this important check for ALM as efficient and effective as possible.

Previous episodes:
Visual Studio Team System 2010 - Episode 1: A Focus on Testing
Visual Studio Team System 2010 - Episode 2: No Risk No Test
Visual Studio Team System 2010 - Episode 3: The Lifecycle

In the last episode Clemens talked about the support VSTS2010 will give on the collaboration at artifact level. Different roles in the lifecycle work together on artifacts. Each of them adds their knowledge, vision and ideas to the solution from their view point. These artifacts are accessible by every role in every phase of the project, adding value throughout the lifecycle. People are enabled to collaborate, making applications together, and not only by telling what they are doing but most important by working seamless together on the application.

TMap® Quality Check
A subsequent measure for increasing the quality of the developed artifacts is an evaluation activity: for instance the review.

The review is a method of improving the quality of an artifact by evaluating the work against the requirements and/or guidelines and subjecting it to peer review.

The review on the requirements and/or design can be carried out as a static test activity before the coding starts.
In the review, the following points can be checked, independently of the set requirements:
1. Has the artifact been realized in accordance with the assignment? For example, are the requirements laid down in the technical design realized correctly, completely and demonstrably?
2. Does the artifact meet the following criteria: internally consistent, meeting standards and norms and representing the best possible solution? Best possible solution means the best solution that could be found within the given preconditions, such as time and finance.
3. Does the artifact contribute to the project and architecture aims? Is the artifact consistent with other, related artifacts (consistency across the board)?
4. Is the artifact suitable for use in the next phase of the development?

Why?
Like testing, the Quality Check is a measure to provide insight into the quality of delivered products and the related risks when taken into the next phase of the lifecycle. If the quality is inadequate timely measures can be taken, such as rework by the designers. However, there is never an unlimited quantity of resources and time. In theory it is important to relate the Quality Check effort to the expected risks. A pragmatic approach to determine the Quality Check effort is to look at some past projects and answer the question:

"How many defects, detected in Acceptance Tests, could have been found much earlier, if we had done a Quality Check? (According to the above points)"
In my experience 10-20% of the defects could have been found much earlier in the lifecycle when the Quality Check was done properly. And because defects in the Acceptance Test are quite expensive, you only have to find 1 or 2 serious defects in the Quality Check to make it economic worthwhile. So my strong recommendation is:

7 Hints and tips
1. Performing a good Quality Check is a kind of inborn specialization. Find a person who is good in recognizing texts with high potential risk on assumptions and or interpretation errors. In other words: find a pencil-pushing, nit-picky quality geek!
;-) Most professional testers are proud to have these qualifications.

2. Before checking an artifact: What is the quality of its source? Is the source of the AUC (Artifact Under Check) ready? authorized? stable? If not, consider to do also a check on the source of the artifact to quantify the possible changes in the (near) future,

3. If a previous version of the artifact is available, and the quality of that version is known: make use of a so called Comparison Tool to find and check the differences. AND! Check always the consistency of the change register, especially when the change register is used in the next phase to implement the changes.

4. Combine the Quality Check with the estimation activity for the next phase. If the estimation is done by another person, let them work together!

5. Use a checklist as a reference! For your own protection, a checklist prevents that too much attention is paid to the use of standards and correct spelling or even to these aspects alone (This can be a cause of friction among the various people involved.) Partly owing to the diversity of design techniques and information sources that, it is not possible to create one general checklist per artifact type. Therefore, checklists should be created specific to the situation per organization and per project. Of course you can use examples like Testing Requirements or Testing Use Cases as a start for the creating of your checklist.

6. Make always clear what checks you will perform. By communicating your checklist you can prevent a lot of misunderstandings on later findings.

7. Audi alteram partem (hear the other side). Don't report on findings/defects without a fair hearing in which the author of the artifact is given the opportunity to respond to the "accusations" against his work.

Collaborate
For some, it is very tempting to do a review in their own silo: Get the stuff to check, find as much as defects as possible, receive applause for the prevented damage and do that over and over again
Wrong! Dont be a Scrooge! (see my blog: "Does Scrooge exist?") An essential part of the Quality Check is the Learning Cycle. By performing Quality Checks the quality of future AUCs (Artifacts Under Check) must grow. So collaborate with the designer, information analyst or whoever made that artifact, and get your applause on the better quality of artifacts instead on the number of defects.

In the next episode Clemens will explain how the tools support the Quality Check, as well as the collaboration around it, to get the optimum results.

Making money with Application Lifecycle Management

Fri, 20 Feb 2009 21:26:33 +0000

A few days ago I was asked by one of my colleagues why I am spending a lot of my time experimenting with Visual Studio Team System 2010 (Team Architect), Blueprints, App Arch Guide and Application Lifecycle Management (ALM) in general. He noticed me ‘living’ in VSTS 2010 CTP for some time now and he was wondering if it isn’t a bit too early for this and what I did to convice to management to let me do this. My immediate answer to this question was ‘No, it is not to early!’ and I explained that we (Inter Access) expect VS 2010 to help us optimizing our Application Lifecycle Management practice. This answer was a bit too vague for my colleague and of course the next question was how will we benefit *exactly* from investing in VSTS 2010 and ALM. Will it make our life easier?, will it makes us better people?, will it improve quality?, will it save us time?, will it save us money?

Exactly these same questions popup when discussing ALM with customers. Apparently making the business case for ALM (and/or VSTS licenses) isn’t always easy. How come?

From our experiences we learned that currently most people and organizations are relating ALM to their development activities (Software Development Lifecycle). Therefore it is only logical that this is the area where people are trying to identify their benefits (costs savings) from ALM. But is this correct? Is this focus too limited? Shouldn’t we focus on more than only development when it comes to cost savings? Especially if we keep in mind that, on average, only 30% of the IT budget is spend on new application development (the remainder is spend on maintenance/operations)!

How come most of us still only focus on development? Is it because we still focus too much on the tools instead of facilitating collaboration between ‘Business’ ‘Development’ and ‘Operations’?

Everybody experienced in VSTS 2005 and/or VSTS 2008 will come to the conclusion that these tools mainly focus on the different roles within the development team (developer, architect, project management). Source control, unit testing and quality assurance features of these products provide us with a professional development environment and help us improving the overall quality of the products that we deliver. Work item management, a centralized store, reports, portals, etc. improve the collaboration within the development team and support project management in tracking progress, staying in control and managing risks adequately. All of this is great and potentially boost the performance of the development teams but experience learns that these benefits don’t come ‘out of the box’! Installing the tools doesn’t make the development team collaborate by default and most certainly doesn’t stimulate collaboration with the Business and Operations!

Now we know where most of us focus on for their ALM related activities, let see how this relates to the complete application lifecycle. For this we will use an the graph below were the x-axis represents time and the y-axix represents value and negative value displayed as costs.

Obviously the lifecycle of the application starts with its development. During this phase we have to make costs to design, develop and test the application. At that time the application doesn’t bring us (actually the business) any value and the complete development phase of the project only costs money. From the moment the application (parts of it?) are installed into production the appliaction starts to generate value till the moment it needs to phase out where it starts to cost money again.

What we see is, that most organizations are focusing on reducing the developments costs and (sometimes) try to shorten the time to market. Btw. it doesn’t come as a surprise that these are exactly the areas where the current releases of Visual Studio Team System focus on.

Reducing costs and make the application add value earlier is good but if we have a look at the image above we can see that the application lifecycle doesn’t end at the moment the application goes into production (where lifecycle line crosses x axis). So, wouldn’t it be great if our ALM practices help us optimize (reduce costs and/or increase value) during the remainder of the application lifecycle also?

For example, one of the things we can do to increase the business value is to practice a proper User Experience design (see this post of my colleague Andries for more info on this). By taking ‘Operations’ into account during the design and development phase of the application we can reduce operations costs during the remainder of the lifecycle. These things combined will result in an application that is more successful for a longer period of time (because it adds more value and costs less to maintain). Also, because we have done a good job developing the application, we know exactly what it does, where it is interfacing with (something VSTA 2010 will help with) and most importantly when it stops adding value which will help reducing the ‘phase out costs’ of the application.

Adding this to the graphical representation of our application lifecycle results in a graph that looks like this.

Based on this, we can now draw our new application lifecycle which might looks something like this (dotted line is new lifecycle).

The good news is that the green area between the ‘old’ and ‘new’ lifecycle is the area were we can make money by adding extra value. The red colored areas is the place where we can make money by reducing costs. Doesn’t that look great???

Please note that ‘Reduce operations costs’ might be misunderstood from this graph. We don’t mean less value but less costs. I didn’t know how to display this correctly :-)

Of course, all of these things don’t come by itself. We have to actually work for that to make that happen and we can’t do everything at once. In this post I am not going to detail all the steps that we can do to make that happen and where we can use the current or future tooling for. However, hopefully this last image makes it very clear that there are others areas, besides development, within the application lifecycle where we can either reduce costs or increase value. So, if anybody aks you why they should invest in ALM this image should give you a starting point for your discussion…

At least, it *did* help me explain why I should spend my time on ALM and experimenting with VSTS 2010, Blueprints and App Arch Guide :-)

Visual Studio Team System 2010 Episode 2: No Risk No Test

Rob Kuijt — Tue, 10 Feb 2009 19:08:30 +0000

In episode 1, Clemens introduced the focus of Visual Studio Team System 2010 on the collaborative effort between the tester and the other roles in the application lifecycle. In this second episode I will start with a short introduction on testing.

Testing is not an aim in itself! Testing is, in fact, a balancing act. What risks must be covered, what results are to be delivered and how much time and money can be spend, based on rational and economic grounds?

The right test strategy will balance among risks and costs .

Testing supplies insight in the difference between the actual and the required status of an object. Where quality is roughly to be described as 'meeting the requirements and expectations', testing delivers information on the quality.

In this, there is no difference for developers who are testing, specialist testers during system testing or generalist testers during the final acceptance test. Choosing the right test strategy is a joined effort between every tester and the other roles in the application lifecycle. This collaboration is needed because, beside insight in the business risks, much (technical and test) knowledge is needed to find the most important defects as early as possible at the lowest price.

The collaboration doesn't stop after choosing the best test strategy. After designing the test cases, also test execution must be organized as a joined effort. Collaboration between the different roles in the application lifecycle is not self-evident. By nature, it looks like developers and testers don't (want to) understand each other. Having this attitude, it is difficult to find out you need each other to deliver software of good quality.

...dont (want to) understand each other.

The consequence of this virtual (and sometimes real) wall is calamitous for quality. Many bugs arise by doing wrong assumptions and interpretations of the, in generally, unclear specifications and requirements. Reporting, analyzing and resolving these bugs take a lot of time, especially when they prove to be not reproducible or wrong.

An Example: After a development project of 5 man year, testing is done by a team of generalist testers. With much enthusiasm the test team executes the test cases they designed during the building of the system. After 3 months the verdict is given: Negative release advice, because of (8) blocking and (22) big defects, the test team gives the advice NOT to go into production. As you may understand this was a big disappointment for both the project team as the business department. A taskforce was established to keep the delay (damage) in control. After the defect analysis the general feeling became better, after filtering the test faults (20%), the functional wishes (20%) and changes (not passed through to the testers; 30%) the repair costs were approximately 200 hours ( for the remaining 30%). In fact the development team had done a great job. Better communication from both sides, before and during test execution, probably had made the verdict a, celebration worthy, positive release advice. Much, much better than this cold shower!

The above paragraphs show that the collaboration between the tester and developer is important for success. If the collaboration between the different roles in the application lifecycle is not actively stimulated and facilitated, the proverbial wall will arise.

In the next episode Clemens shows us how Visual Studio Team System 2010 will support and stimulate this collaboration...

Testing or Documenting?

Rob Kuijt — Mon, 12 Jan 2009 21:28:21 +0000

How sure is a company about the quality of their Calculation Component(s)? Are they bug free? Even after, for instance, nine changes? Testing changed components is tricky, because you never get the time to test a Calculation Component with lots of values, because most of the testing is still done manually.

There are many test tools that could do this job much faster, but they are complex to use and mostly pretty expensive.
So I did some thinking.I want to do lean and mean Calculation Component Analyzing!
Preferably: User-friendly, quick, low cost and in a way that the outcome is easy to check!

Personally I like pictures instead of figures. So I started some PHP programming, and voila these are my results:
The first version is a service that can analyze a one parameter Calculation Component. Let me give an example.
The test object is a web component with one input parameter. In my home made analyzer service I give the URL of the test object, the start input value, the end input value and the step size (see fig.1)

Fig.1 Input screen of my home made Analyzer
After sending the input, the analyzer performs, in this case, 21 calls to the test object, and responses with the following graphics:

Fig.2 Output screen of my home made Analyzer
This output gives a first impression of the calculation. It looks like: output=input*input (if input > zero) or output=input*input (if input >= zero).

But to be sure, let's make the steps in the range smaller(step=10):

Thats strange! The input value 10 gives a response: zero!

Again, let's choose step=1:

Now I'm pretty sure that the calculation function is: output=input*input (if input greater than ten)

And for a last check (in this example) I choose step=0.1:

Yes, it still looks like: output=input*input (if input greater than ten)

Conclusion:
I think that this kind of functionality is valuable, not only for testing, but also for documenting Calculation Components.
Above that, it is:

Easy to use (one simple input screen),
Low cost (I did build the function in 2 hours (with the help of open source: PHPLOT)),
Quick (the above analyzing took me 3 minutes (including capturing the pictures for my blog)),
Simple to read.

And it's FUN to play with!
Now I'll try to find some use for this kind of functionality, and l start thinking how to handle (and present the outcome) for a two parameter Calculation Component.

Rob

Blueprints: Visual Studio 2010 (2)

Mon, 12 Jan 2009 14:07:57 +0000

In an earlier post we mentioned that it is relatively easy to get the current Blueprints bits running on the Visual Studio 2010 CTP by modifying the .MSI in Orcas. At that time we forgot to mention that we need a few extra steps to really get things going with Blueprints in Visual Studio 2010.

When trying to build a Blueprint solution in Visual Studio 2010 we will notice the following error in the error window.

As we can see, the build task ‘BASM’ is failing to retrieve the correct path. This task is implemented in the ‘Microsoft.SoftwareFactories.Blueprints.Builds.Tasks.dll’ that can be found in ‘..\Program Files\MSBuild\Microsoft\Blueprints\2.0’. It turns out that the execute method of this tasks looks for a (hardcoded) ‘String Value’ called ‘Blueprints’ under the ‘HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\MSBuild\SafeImports’ tree. Because we replaced all ‘9.0’ in ‘10.0’ in the Blueprints .MSI to get it to install on Visual Studio 2010 this value doesn’t exist under ‘9.0’ anymore (but does under ‘10.0’).

To fix this we can either make sure to skip this particular replacement when modifying the .MSI in Orcas or manually add the Blueprints ‘String Value’ under the ‘HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\MSBuild\SafeImports’ and give it the value ‘C:\Program Files\MSBuild\Microsoft\Blueprints\2.0\Microsoft.SoftwareFactories.Blueprints.targets’.

Another issue occurs when debugging our Blueprint in Visual Studio 2010. Currently, there is no property page implemented for the Blueprint project type (.bpproj) and therefore starting up Visual Studio 2008 is hardcoded in the Blueprints core. To get around this we can add an empty C# ‘Class Library’ project to our solution, set this project as the ‘StartUp’ project and make this project startup Visual Studio 2010 (property page) when debugging. Although this solution does work it makes the Visual Studio instances in my Virtual PC image VERY slow (don’t know why). Another option, that does work for me, is to leave the Blueprint project as the ‘StartUp’ project, let it start up a Visual Studio 2008 instance, (and simple ignore it) manually start another Visual Studio 2010 instance and attach this instance to the debugging process of the Visual Studio instance we started the debug session in.

Now everything is in place to *really* start developing Blueprints for Visual Studio 2010 CTP!

Does Scrooge exist?

Rob Kuijt — Wed, 26 Nov 2008 19:19:08 +0000

Fiction or reality? On a, further nice, kind of conference meeting, I did met Ebenezer Scrooge!
I didn't know that such Scrooge-type testers do exist!
:-((

Scrooge was fully focused on finding bugs, and if I say fully, I mean FULLY !
Scrooge was completely focused on his own world, collecting as much as bugs as possible, and enjoying it with a kind of scary laughter..... I must say his test results seems to be excellent, very fast in creating test cases, and he knows and uses more test techniques, I ever did. And he has, as we say it, a nose for finding bugs. But his eyes did spit fire when I suggested helping the developers to find the bugs earlier in the cycle. "Why should I destroy my own work" Scrooge replied..... I must confess, I didn't have a response right away. I know that most of the (good) testers enjoy finding bugs. But such a fanatic ego-centric type was new and a complete surprise for me. In fact, I think that Scrooge-type testers, especially this fanatic, are a disgrace for the test profession I love.

So I tried to convince him to change his attitude. Of course I didn't succeed the remaining 15 minutes we met. Probably he needs a visit from the ghosts of the past, present and future!
;-)

For the Scrooges among us
For the Scrooges in the test world I have a message: Finding bugs can deliver applause from your surrounding and it may look that your manager is pleased with the extra test hours (he can send a bigger invoice). But in the long term no one (beside you) is happy with a Scrooge attitude. The business users don't get their systems on time, the developers won't help you if you need them, and if the project manager becomes aware of this attitude, he will kick you out (so your manager can't send any invoice at all). So broaden your small ego centric world! Adopt Application Lifecycle Management [ALM] and find bugs as early as possible in a collaborative driven attitude.

For the non-Scrooges
How do we fight this irritating phenomenon? In my opinion, the best testers are the ones helping (actively) the developers to build better software. Luckily some of the big test gurus of this world preach the same opinion (see the entry measuring testers in the weblog of James Whittaker).

Wild thought
Can we make testers responsible for the quality of the software? For instance: Is it possible to award testers depending decreasing defect rates. I think a kind of Collaboration-bonus awarding mean time between failures and/or decreasing defect rates can work! Has someone a suggestion for the formulation of such a performance indicator or Collaboration-bonus?

I'll look for some like-minded friends and give it a try in the coming period. Im sure this entry will be continued....
And again: Suggestions are very welcome!

Rob

Wat zeggen de experts?

04. Getting Testing Done in the Sprint – Undone Backlog Item

Undone End to End Testing

03. Getting Testing Done in the Sprint – Test Automation

02. Getting Testing Done in the Sprint – Regression Test Sets

01. Getting Testing Done in the Sprint - The Team and Activities

Teams and TFS Groups in TFS11, the backlog, board and security settings.

Final Conclusion.

TFS11 Scrum board update remaining work without opening the work item…

TFS11 on Azure January 2012 update – request feedback

Deck: The agile cloud development scenario in 10 acts…

Coded UI Test Automation Practices Deck

By: The SEMAT Vision (Agility@Scale: Strategies for Scaling Agile Software Development)

kicking-off Architecture Tooling Guidance Refresh 1.1

[DUTCH] Artikel over Extensibility in Visual Studio 2010

Visual Studio 2010 Architecture Tooling Guidance

Model Based Testing (part 1)

Cloud computing and Application Lifecycle Management

Layer Diagrams: Application Architecture Guide

History of Test Automation

Test Case Design: Manually or Automated?

More return on development with Application Lifecycle Management

ALM and MBT

ALM: the end of the whispering game

WSCF.blue Beta 1 is out!

Model-Based Testing is not "Only Fun" anymore....

Architectural Inspections: Implemented in Visual Studio Team Architect 2010

Visual Studio Team System 2010 - Episode 4: Quality Check

Making money with Application Lifecycle Management

Visual Studio Team System 2010  Episode 2: No Risk No Test

Testing or Documenting?

Blueprints: Visual Studio 2010 (2)

Does Scrooge exist?

Visual Studio Team System 2010 Episode 2: No Risk No Test