S/W Automation Quality Analysis

Headless Automation Testing

2011-02-28T10:21:00.000-08:00

Headless Testing means testing without GUI.

There are couple of testing such as Unit testing, Component Testing, Module Testing, Integration Testing, CC (Code Coverage of Condition) Testing \ Base Path Testing, Mutation Testing, DB Testing, API Testing and or Web Service Testing may fall under Headless Testing. As you can see most of testing are white box or gray box testing.

In XP\XT (Extreme Programming\Extreme Testing) of agile, It is the first step in implementing story.

Open Source tools such as Watir, Warin, Junit, Nunit, HTTPUnit, HTMLUnit, WebTest Canoo, Fitness, Selenium RC and scripts like shell, vbscript, groovy, SoapUI (and some good paid tools Autoczar, QTP, SilkTest, TestComplete) are helping to automate this type of testing and play effective and efficient role in the QA feedback of product lifecyle.

It actually tells us where is the issue. So, aways say "Yes" to Headless Testing in your project.

Groovy and Jemmy

2009-12-31T07:41:00.000-08:00

In this demo, we have opportunities

1. To Bind your Java application with groovy console.

2. To Automate manual steps in groovy console.

1. Bind your Java application with groovy console.

//BindJavaApplication.groovy

import org.netbeans.jemmy.*;
import org.netbeans.jemmy.explorer.*;
import org.netbeans.jemmy.operators.*;

// Fire up the SwingSet2 Application

new ClassReference("SwingSet2").startApplication();

// Get a reference to the SwingSet JFrame and put it into the console's script binding.

mainFrame = new JFrameOperator("SwingSet")

def bind = new Binding()

bind.setProperty("testScript",this)

bind.setProperty("mainFrame",mainFrame)

// Fire up the Groovy console.

def console = new groovy.ui.Console(this.class.classLoader,bind)

console.run()

Thread.sleep(700000)

2 . Automate manual steps in console.

import org.netbeans.jemmy.*;

import org.netbeans.jemmy.explorer.*;

import org.netbeans.jemmy.operators.*;

// Get first button object to click
buttonObj = new JButtonOperator(mainFrame,0)

// to click on button with title “OK” buttonObj = new JButtonOperator(mainFrame,”OK”)

t = new Thread() {

buttonObj .push()

}

t.start()

Hey, what we have just done; we have just clicked first button of SwingSet.

Cheers !! Enjoy Automation.

~NV

AFT (Automation Framework Team) should not take developers' false catches

2009-07-27T07:23:00.000-07:00

What does It mean by "developers' false catches" ?

It means those implementations or application unexpected behaviors which should not be automated with automation framework.

Why those should not be the candidates for automation? It is observed that adding these kinds of requirements in automation may set short ROI for organizations but it never becomes for long terms – You may ask “why” again, below are couple of examples which put more lights on this question.

Case-Study : #1
“Product Team has new requirements in build 1.2.x like objects inputs are changed with prefix or suffix strings and that affects more than 1000 automated test-case(s)”

Due to limited resources and short dead line to certify build(1.2.x) of the product, Product lead comes to AFT and asks for help. He requests to update Automation Framework in such a way that manages prefix and suffix inputs and his team does not have to update their test-case(s). As part of the service team (AFT), we provided solution at framework level. These solutions may help them to certify product but it also carries live wires to take cares are as

1. As per Automation Framework development perspective,
Automation Framework becomes dependent to product builds.
According to Standard Automation Framework protocols, framework should not depend on product; Independent framework can be useful to automate other products as well and such framework would have maintenance and complexity low which helps to give consistent behavior for automation projects.
2. As per Product team perspective,
Product team loses track for test-case(s) on product behavior. Since, Automation Framework injects input data for build 1.2.x ; There becomes no control to update those input data by product team and they lost transparency of test-case(s) with test-data inputs which may raise high effort require to map test-data inputs with testcase(s) per builds for both product team and Automation Framework Team.

Case Study #2
"Product Team has more than 800 test-case(s) automated which works fine with couple of synchronization points (like status as “Processing On Server”). Next iteration, team gets new build and say for “Processing On Server” is broken."

We always face some issues with product to automate. Some are very common when application behavior changed and automation framework has synchronization issues but they concern when they becomes blocker issues for team to certify product. I see lots of organization does not care for this kind of issues since they are not affecting to their core functionality to certify build manually. Because of automation works on synchronizing objects behavior; those issues become “real matters” to resolve them. In one of the experience I have, these issues (broken synchronization points or application unexpected behaviors) made automation in batch executions hanged and crashed the application.

In such condition, product team wants to certify build and request to resolve this kind of issues at automation framework level. As part of solution provider team(AFT), we resolves these issues at automation framework level and helped them to certify build.

But, Why these issues should be resolved at application level not automation framework level? Because

1. As per Automation Framework perspective,
To resolve these kind of issues, automation framework injected with codes to handle broken synchronization points. This becomes un-necessary degrade test-case execution performance and pulls garbage code to automation framework project.
2. As per Product team perspective,
Most of the time, These issues are related to performance of the application and so they should be fixed at application level with priority in QA cycle.

Concluding as …
AFT supports product teams to unblock them to certify builds by giving hot fixes or work-around at framework level but however it is the product teams' responsibility to have priorities to fix those issues (application unexpected behaviors - false catches) at application level and deliver quality product.

After all, we deliver product only; not product with automation framework to handle product’s uncertainties :)

Please, send your precious comments here,

With Regards,
NV.

[Photo: In this Nov. 19, 1978 file photo, Philadelphia Eagles' Herm Edwards (46) pounces on a ball fumbled by New York Giants quarterback Joe Pisarcik (9), right foreground, in the last minutes of the game. Credit: G. Paul Burnett]

Everyone is talking about automation tools like QTP,...

2009-07-27T07:02:00.001-07:00

Everyone is talking about automation tools like QTP, WinRunner, LoadRunner,Silk Test, Selenium Bla Bla bla ... but Once these tools come into real practice and he\she has to automate test-case(s) with them in existing automation framework.

How far we can go with our goal?

This post gives opportunity to discuss on characteristics of test-case(s) while doing designing for test-case(s) in automation. Below are the couple of thoughts, I believe to have in automated test-case(s).

§ Concise - Test-case(s) should be as simple as possible.
i.e. Test-case(s) should not call multiple other complex Test-case(s). It should have minimum dependency.

§ Self Checking – Test-case(s) should have verification steps and It
should report in its results such that no human interpretation is
necessary. For example, I have found in one project; team has around 1500 test-case(s) – out of more than 44% test-case(s) has steps like

1. Open Menu Item.

2. Enter A/C number and other Inputs on screen say for A.

3. Click on Button "Next" on screen A.

4. Click on Button "Next" on screen B.

5. Click on Button "Next" on screen C

6. Click on Button "Close" on screen D – here, one product was created.

Here, all screens (A to D) have Button "Next" and "Close" and that raises some real concern to take care. Why? Because of default values of each screen are missed to verify. Test-case for each screen should have all default values' verification steps so that user\automation tool can confirm that he has achieved so and so functional flow screen while running specific scenarios. If any new screen with "Next" and "Close" button introduces in middle of screen(A-D) in new build – this will capture by automation test-case(s).

§ Repeatable – Test-case(s) can be run repeatedly without human
intervention. DDT(Data Driven Test) or test-step parameterization is one of the big benefit of automation.

§ Robust – Test-case(s) should produce same result now and forever.
Automated test-case(s) should have consistent output. It is not an easy task to achieve this characteristic. In hybrid Automation Framework, Automation team can create BO keywords to setup baseline and this keywords should be practice well enough that test-case(s) can have baseline to initialize and other test-case(s) in queue can execute regardless of previous test-case(s) passed or failed.

§ Necessary – Everything in each Test-case(s) should contribute to the specification of desired behavior. All pre-requisites should be set well – Test-case(s) should have none or minimum pre-requisites to set manually.

§ Clear – Each step in Testcase should be easy to understand.

§ Efficient – Test-case(s) run in a reasonable amount of time. It manual test-case(s) takes 100 minutes then automated test-case(s) should not take more then 50-60 minutes at the best efficiency.

§ Specific – Each failure step should point to a specific piece of broken functionality

§ Independent – Each Test-case(s) can be run by itself or in a suite
with an arbitrary set of other tests in any order. To avoid dependency in Hybrid automation framework, automation team can develop BO keywords and add them for each test-case(s) to avoid dependency.

§ Maintainable – Automated test-case(s) should be maintained and extended with testcase management tools like TestLink, HPQC, Test Director or many other freeware tools which can easily plug with automation framework.

§ Traceable – Test-case(s) should be traceable to the requirements.

So, when we have Test-case(s) to automate – have above protocols in mind to do your tasks more effective and efficient. I would like to have more thoughts on this ; please, add your precious comments here.

Enjoy Automation / NV

Some Genuine Confessions [Fun]

2009-01-08T21:23:00.001-08:00

One daily dose from TakeITLite :)

Mozilla add-ons for Bugzilla

2008-12-12T08:59:00.000-08:00

Some Mozilla add-ons which are worth to install to improve your performance.

BetterBugzilla 0.3.4 - gives new look for your bugzilla.

Sage - allows to create feed on search query of your bugzilla.

Newsfox - allows to read rss feeds in web-browser email layout; if feed items update -- this will pops up and mark as unread for that item in feed; you can use this extension to point your bugzilla feed so you can get notice for all activities happening in bugzilla.

~Enjoy,NV

How to automate test-scenarios which have Java Objects built on JMesa , JQuery, JSON and AJAX technology.

2008-11-29T04:07:00.000-08:00

I had a challenging project to automate Java Objects on the fly – sharing some part of it.

Challenges I faced
1. Objects which are built on top of JMesa , JQuery, JSON and AJAX technology -- are not recognized by professional automation tools (such as QTP and SilkTest).

2. How to automate Dynamic Graphs which are implemented with technologies like Flotr and 3D Canvas.

3. Web application needs to automate with cross-browsers ( IE 6.0+ , FF 2.0+ or Safari 2.0+ ) It focus objects screen rendering issues which are different from one browser to other.

How can we overcame
• Challenge #1 and #2 can be resolved with below automation approach.
Verify HTTP Responses when user does some actions like objects fire specific AJAX calls; We simulate these AJAX calls ( HTTP requests ), capture valid\expected response, set them as baseline and in QA loop we simulate user actions again with automation, capture actual response and compare with baseline.

• Challenge #3 can be resolved with below automation approach.
Simulate request for each browser to server, Capture valid\expected HTTP response for each browser such as IE 6.0+, FF 2.0+ or Safari 2.0+ from server then set them as baseline and in QA loop we send request again per browser, capture actual HTTP response for each browser (such as IE 6.0+, FF 2.0+) and compare with expected baseline response – This will uncover objects which have screen rendering issues.

I recommend Canoo WebTest automation tool as It works well with AJAX application and understand HTTP request and response.

How to claim web product certified by QA

2008-11-24T20:50:00.000-08:00

Below V&V activities' document with their approved status need to produce by QA.

Thanks,
Nimesh Vadgama.

QA Project Checklist

2008-07-29T06:45:00.000-07:00

Here is I want to share qa project management checklist I used in real scenarious.

Project Initiation
Prepare System Test Estimates
Define System Test Approach
Define Testing Scope
Prepare DRAFT System Test Plan
Review System Test Plan
Prepare Test Schedule
Request Test Resources
Request Business Expert
Request Management Support
Request Environment / Technical Support
Request Facilities (Desk, Chair etc.)
Setup Test Project Folder
Revise Test Estimates
Define Entrance / Acceptance Criteria
Agree Communication Channels
Agree Reporting Procedures, Methods and Frequency
Define Exit Criteria
Design Release Notes Template

Test Preparation
Agree Builds / Drops Schedule and Contents
Agree Release Notes Contents and Format
Agree Error Management Procedures
Define System Test Roles and Responsibilities
Assign Test Roles and Responsibilities
Assign Test Case Preparation Primary Responsibilities
Assign Test Case Preparation Secondary Responsibilities
Prepare High Level Test Cases
Prepare Low Level Test Cases
Define Test Environment Setup (Network / Server)
Define Test PC Setups and Configurations - clients
Review Test Plan
Review Test Schedule
Setup Test Execution Progress Tracking Database

Build System Test Environment
Setup Test Environment (Server)
Setup Test PC's
Setup Bug Database
Verify Test Environment (Shakedown)
Verify Bug Database External Access
Setup Test Data
Setup and Install Test Peripherals (card readers, receipt printers)

Prepare System Test
Review System Test Cases
Revisit System Test Cases
System Test Readiness Review
Verify Entrance Criteria reached
Receive B36 Build 1
Install B36 Build 1
Execute Acceptance Tests
Review Acceptance Test Results ( accept - yes / no )

Execute System Test
Execute System Test
Execute Cycle 1 - GUI Tests
Execute Cycle 2 - Functional Tests
Execute Cycle 3 - Scenario Tests
Log and Track Defects
Maintain and Administer Error Management System
Progress Measure and Reporting
Measure Progress - Actual vs Planned
Manage and Track new Builds
Perform Build Regression Tests
Close Regress Bugs / Re-open "Not Fixed" bugs
Measure Error Statistics and Metrics
Report Error Statistics and Metrics
Track and Record Error Turnaround Time
Escalate Issues as appropriate
Perform Final Regress Test

Signoff
Signoff System Test
Produce Post-Testing Report
Wash up & Lessons Learnt Meeting
Review Exit Report
Cleanup Test Environment
Return Peripherals
Post Execution Test Case Review
Handover Test Documentation

Let me know if anybody needs more details.

How developers fix their bug -- funny forwarded email.

2008-06-02T22:31:00.001-07:00

Forwarding one funny email from yahoo group ~ enjoy testing - Nimesh

##########################################################

#include <stdio.h>

#define LAST 10

int main()
{
int i, sum = 0;

for ( i = 1; i < = LAST; i++ )
{
sum += i;
}

/*-for-*/
printf("sum = %d\n", sum);
return 0;
}

##########################################################

And the developer fixes it this way

##########################################################

#include stdio.h;
#define LAST 10

int main()
{
int i, sum = 0;

for ( i = 1; i < = LAST; i++ )
{
sum += i;
}

/*-for-*/
printf("sum = %d\n", sum);
return 0;
}

##########################################################

Communication and right attitude continuously adding precious drops in the sea of success.

2008-06-02T16:07:00.001-07:00

Someone has well said "Communication matters" -- I would like to go further ahead.

Yesterday, I went to PathMark to get some stuff. There were some schemes going on Tide (laundry detergent) with light bulbs on discount. One Customer came to customer service and started demanding he needs two bulbs free with Tide but the fact was that there was some discount if you purchase Tide with that light bulbs. The customer support guy explained the person that he could not give free two light bulbs as that was not scheme currently. Still the guy asking for free – he did not listen to customer support person and continuously telling him "I want free light bulbs". Customer Support Person explained him three or four times with his best knowledge that he could not give him but still that guy demanding. At last customer support person call to his Manager for help; Manager came and explained to the person that you got your discount and that is the scheme running currently but person started arguing like "what kind of scheme you guys have,…what is the logic behind you did not give me free light bulbs and lot more thing…" At last conversation with Manager, he asked PathMark's Customer Service Call center number and told that he would complain about this and that and left the Mall.

I was standing by that customer and seeing everything. I saw customer support guy – the gentle man has tried lot to make that person understood but that person was not open for it, he did not want to listen anything from him or from Manager – he just wanted free light bulbs along with Tide that was not actually scheme running. Customer Support Person and Manager got frustrated first as customer continuously demanding unfairly and wasting their and other customers' time who were in queue and at end that customer got frustrated when he left the Mall.

The same thing may happen with any entities in project cycle and It may damage project progress and lead to deadline delayed.

There are so many ways to make front person understand but of course both the parties would have will to listen each other point of views and analyze each other to get the right crux of the matter so that at end both can be benefited. As you see -- The patience to listen each other has major part in communication. There is one saying "for clapping you need both hands , one hand can't do".

Communication and right attitude continuously adding their precious drops in the sea of success ~ Nimesh :)

Tip to create workflow in JIRA quickly

2008-05-27T21:49:00.001-07:00

To my colleagues who want to create quickly new workflow and don't want to touch xml \ property files.

On Administrator \ Workflows

If you don't have default workflow then create default Workflow with default JIRA "workflow" - it will show status as "active" so we cannot update its scheme \ workflow steps.
Copy this workflow -- it has status as "Inactive" so we can update workflow scheme.
Go to link "steps" and add new status \ transitions for new steps as required in your workflow.

On Administrator \ Workflow Schemes

Create new "Workflow Scheme" then click on "Workflows" link operation of new workflow just created.
Assign workflow to scheme,... Issue type "Bug" and select our workflow from the list.
Assign other workflows for issue types : Task, New Feature or Improvement with default workflow ( named JIRA ) or your workflow.

Now our workflow scheme is ready to associate with our project.
Go to your project admin panel and set workflow scheme there.

Let me know if you need more detail :)

Takecare,
Nimesh.

Typical Project flow with QA Loop in Jira

2008-05-27T14:56:00.000-07:00

Last week, one of my friend needs some help to establish Project\QA workflows on Jira
So here I am happy to share some of my odd solution.

Project Flow problems or hard to implement in Jira were

* One stage splits with multiple stages in flow.
* More than one stages merges to one stage.

Here We have Brainstorming splits to UT - ST stage and Dev Review - QA Review merges with Review stage executed by System Group Members in project flow.

One issue splits with more than one issues, can also be implemented with sub-tasks feature but it may not push next stage in the project flow as the project progresses; all sub-tasks are linked to the parent task only, may not execute inline flow; may not give more control to administrator\lead and may be more complicated to maintain. Sub-tasks may not be merged to one task on transitions after Dev. Review / QA Review merges to Review stage.

Core objective is to implement flow that anyone can see each stage of the flow and can go back and forth from any stage, Dev and QA stage should work parallel and merge to "Review" and then System Verification should be executed as build in QA loop.
Administrator \ Lead of each stage would have full control to execute project development and would have transparency; Jira could help to push next stage after each stage done.

We first implement above flow as manually after we will find which actions need to automate.
Manually Implementation :
# Created issue types for all Project Concept, Brainstorming, ST Identification, UT Identification...until System Verification as Standard Issue type like task.
# Create work-flow schemes for each issue type we have created i.e. ProjectConcept work-flow will have step Created, Go for Brainstorming , Reopen.
# Enable Issue Linking from Administrator and create two link type as we have two flow one is task flow other is defect flow.
* Project Flow :
- Outward Description : created after , Inward Description : progressed to
* Defect QA Loop :
- Outward Description : Linked to , Inward Description : Linked by
# Associate each work-flow with its issue type.
# When flow step done -- create new issue type of the next flow step and link it with its previous parent issue type on Project Flow.
# On SV step make it open and add reference to each defect in its sv step.

Here is snapshot after implemented above actions :

We can automate manual actions
# Create next flow issue type when specific issue committed or completed,
# Assign to the lead,
# Add reference(link) to previous parent issue type.
Automation :
Here I will show one implementation from stage Project Concept to Brainstorming; Others can be implemented as same way.
For this we will create Jelly script and add them in Jelly Service of Jira.Script executes filter and get issues with issue-type : Project Concept , status : "Go for brain storming" and which does not have comment "Brainstorm added" and does manual actions and then add comment "Brainstorm added" to project concept(previous step of the flow) so next time when service executes same project concept will not create brainstorm step.
Created filter : GetProjectConceptsWithNonBrainstormingAdded at desk for this project.

Copy script code
To execute above script, update your FilterId with 10020 to yours filterId, to find filterId go to Find Issues / Manage and put cursor on your filter and see the filterId on status bar.

Please, feel free to ask any question regardings.

Thanks,
Nimesh Vadgama.

Combinations n Permutations in QA

2008-05-11T20:35:00.000-07:00

Combinations n Permutations are very useful in software testing.
It help to find possible test-data / testcases. This session will provide details about

1. Some basics about Combinations
2. Logic to find possible combinations for taking k numbers of items from n number items at a time.

1. Some basics about Combinations.
You might be come across with below example in old school days.
We have set { "ant", "bug", "cat", "dog", "pig" } and If we want to take 3 elements at a time then we would have possible combinations as
{ "ant", "bug", "cat" }
{ "ant", "bug", "dog" }
{ "ant", "bug", "pig" }
{ "ant", "cat", "dog" }
{ "ant", "cat", "pig" }
{ "ant", "dog", "pig" }
{ "bug", "cat", "dog" }
{ "bug", "cat", "pig" }
{ "bug", "dog", "pig" }
{ "cat", "dog", "pig" }

So for n=5 and k=3 in above we would have 10 possible combinations.
How we can count ?
We have formula :

Total number of combinations = n * (n-1) * (n-2) * ....* (n - k +1) \ 1 * 2 * ...* k

So, n= 5 and k= 3
Total number of combinations = 5 * 4 * 3 \ 1 * 2 * 3 = 10
We will use this formula to implement C# function Choose(n,k) to find total number of possible combinations in coming session.

2. Logic to find possible combinations for taking k numbers of items from n number items at a time.
The basic logic is to get all possible combinations item index.
So, If we have data[5] = { "ant", "bug", "cat", "dog", "pig" }
{ "ant", "bug", "cat" } = { data[0] , data[1] , data[2] }
and so we need to get { 0 , 1 , 2 }
{ "ant", "bug", "dog" } = { data[0] , data[1] , data[3] }
and so we need to get { 0 , 1 , 3 }

And for others as respectively we need to get
{ 0 , 1 , 4 }
{ 0 , 2 , 3 }
{ 0 , 2 , 4 }
{ 0 , 3 , 4 }
{ 1 , 2 , 3 }
{ 1 , 2 , 4 }
{ 1 , 3 , 4 }
{ 2 , 3 , 4 }

To find combinations we will use lexicographical algorithm.
In this algorithm, we find successor. Successor is the next possible combination.
Say for { 0 , 1 , 2 } we have successor { 0 , 1 , 3 }
The core logic to get successor is first to find right most element that must be incremented then we increment all right most elements for that we will create Combination Class and implement two constructor to set base data, n and k of the set as per requirement and then execute Successor Method and will store all possible combinations of object array and display as well.

Inspired by article of Dr. James McCaffrey on String Permutation.

Automate QA Feeds task [continuous to last post]

2008-05-01T19:56:00.000-07:00

Continuous post of CSV\XML\RSS Feeds are QAed

To validate Feed URL(s) are not broken ; we can use XML DOM. Below is script that validates broken link.

To validate ASCII Character , I have used below script.

You can create excel custom tool where you centralize all feeds , verifies broken resources before creating actual feed.

CSV\XML\RSS Feeds are QAed

2008-04-20T16:23:00.001-07:00

In Content centric organizations, feeds are one of the core part that needs to update every time when

New Contents are Uploaded.
New Categories \ Groups are introduced for Old Contents.

As we know first one is very repetitive task -- so need repetitive validation.
Second one is less repetitive but much costlier and involve high risk – so need more rigorous validation as well.

Both missions have some kind of common problems :

Feed Generator Software\Content System retrieve Content Data ( URL(s) \ Content Details ) which may have ASCII Characters and Memory Garbage and old inactive URL(s)– URL(s) where contents are not actually accessible by user.
Feed may contain non compatible XML data( like special characters > < & ) which may break XML \ RSS Feeds.

Feed may not have broken content URL(s) and Content Details but they may be mapped wrongly. It means Content Details for one Content may not present its own Content URL or in opposite.

So for QA, we expect to

1. Validate Broken Content URL(s)
Content URL(s) are usually redirects user to the subscription services.
They are paychecks for companies.

2. Validate Content Details are not broken.
Content Details are like Content Descriptions on which user read and subscribe service or purchase content from company like Content Name, Type, Usage , Marketing Phases.

3. Validate each correct Content has its own correct Content details.
Content details are mapped correctly with each other.

Validating above tasks MANUALLY is not a practical way as we all know in middle to big scale organization we would have more then 10,000+ contents in QA. There are some automation ways to validate above task more effectively and efficiently -- I will focus light on them in my next post .

NUnit basics for new bees

2008-04-18T20:47:00.000-07:00

In smaller to bigger project, here are some basics about how to automate Unit testing with NUnit framework on .Net Platform.

Where NUnit spot on SDLC?
In other words, we can ask when to start unit automation on project? It depends on various parameters say for example Complexity of Method(s) / API(s) or Classes in production cycle - How Method(s) API(s) / Classes are called by other Method(s) / API(s) / Classes in production cycle.

We can start NUnit automation after completion of any independent Method / API / its Class created at production and making automation continued towards parallel of development cycle of more dependent Method(s) / API(s) / Classe(s).

Say for example we have small class library project in C#.

Now lets start NUnit Automation in 7 simple steps :

Create New Test C# Project (Here Named TestProject) and add production class (.dll) reference.
Add reference NUnit.Frameworkto the TestProject.
Add attribute "TestFixture" to mark as Test Class and so NUnit GUI can recognize.
To create Test Category / Test-Cases group, add attribute “Category” passing with CategoryName as a param say for here [Category(“TestCategory1”)].
Add attribute “Test”to the test-cases which will teardown inside test-class.
To implement test-case , create Object of the AUT Class and call Assert with its various methods as required.
Save the project and Open this on NUnit GUI and last run the test.

Njoy, Nimesh - (Get more NUnit stuff on http://www.nunit.org )

Test Result analysis and reporting -- Art of good Test Engineer

2007-11-11T20:19:00.000-08:00

Test Environment Setup

2007-10-25T20:53:00.001-07:00

Objective
How to make your automation framework accessible through out all Test Environments in other sense how to make your automation objects ( scripts \ business objects ) reusable with other platforms with Automation Tools such as QTP \ Silk Test.

Problem
You have different test platforms such as
qa:6080 like build in qa 6080 port.
qa:7080 like build in integration qa 7080 port.
qa:8080 like build in staging qa 8080 port.
Live environment (like any production site)

We need to have centralized automation script that can get access to all above platforms.

Approach for problem solution
• We need to create
1.Function that can read current global environment configuration which may be in .xml.
2.Start-up Script that can add / update / set current environment from .xml or any other format – you can make GUI tool or use VBScript + Excel to maintain easily.

•We need to normalize all automation objects which are pointing to specific platform like filtering out all redundancy from automation objects – you can here use regular expression and concate current environment by calling function.

Execution
Execute your start-up script to set up current environment we want to go on.
Associate function to the script and call it as mention above.

QA Automation in V Model

2007-10-25T20:42:00.000-07:00

QA Automation in V-Model

Low level Testing

Unit Testing with NUnit (and MS Visual Studio), xUnit and HTMLUnit on Java and C# / ASP.Net plateform.
HTTP Request-Response Testing with Selenium, Canoo WebTest, SOAPUI and other free-ware tools.

Integration Level Testing.

Components Interface Testing.
Web-Service Testing with SOAPUI
API Testing with WebTest Canoo \ QTP \ SilkTest
Function \ Smoke Testing \ Regression Testing

High Level Testing.

Cross Browser \ Cross Platform Testing.
GUI Testing with Tools such as Mercury QTP, Silk Test, WinRunner.
Load \ Performance testing with LoadRunner.

~NV

QuickTest Pro and Traceability Matrix

2007-09-13T11:41:00.001-07:00

I like to share one automation experience or in other sense the challenge I had in automation.

I joined one company as an Automation Quality Analyst Consultant. It is fast growing Company. I was as a tool smith like person recommending tools for QA team to make manual testing tasks more easier\reliable and fast way. We had established freeware tools for testcase management, defect repository and qa\project process and developed
test-suites for regression and smoke testing in QTP.

As we all know regression test-suite is bulky one and at that time, mine was including more then 617 testcases automated and that’s why taking more than 3 hours to finish execution. I have split test scripts on functionality vise and created test batch but the problem is test batch report does not give status for each testcases in automation execution and that encourage me to develop “TM live” tool interfaced with QTP 9.1.

“TM live” means Traceability Matrix on live (parallel with automation suite execution)

Live Graphical Report - Demo

Increase ROI in your organization with Automation Testing

2007-09-13T11:02:00.000-07:00

This article offers answers for below questions with case-study

Why do we need automation s/w testing in organization? 

What if organization does not have fund to purchase license
for expensive automation tools?

In simple way, we can say the core objective of automation is helping to finish any testing\QA task more accurate, easier and faster way with less risk involved for failure. Say for example Tools like
# Functional automation tools QTPro, SilkTest, Canoo, Selenium, VBMacro.
# Load/Stress automation tools LoadRunner, SilkPerformer.
# Cookies reader tools.
# HTTP request / response reader tools.
# File comparison tools.
# Tesecase Management Tools.
# Web Page link / image finder Tools.
# Broken link finder Tools.
and many other tools we can categorize in automation testing fashion to make QA Process more smoother; Automation Testing also used for

Repetitive testing task activities…
As a human nature, manual tester can not perform repetitive tasks more accurately comparing to machine or software automation program. Test data-driven approach is one of the best solutions of automation for repetitive testing. Automation makes "cross browser testing" a peace of cake, you just change configuration file and you are ready to verify features on other browsers.

Is More Reliable…
In the sense of giving feedback more faster and cheaper way. Automation provides excellent feedback in well HTML / XML forms which are also open to plug into our QA statistical system like System that generates Traceability Metrix say for Mercury's Quality Center or Borland's SilkCenter and we can make qa process more transparent toward organization. Anyone can execute automation scripts at anytime, most of the times they are triggered at night. Automation always pays back; when new build comes in qa, we execute smoke automation scripts to check build health and trigger regression automation scripts to verify old features work with new features as expected in more faster and cheaper way comparing to manual testing – It works like safety net where not only qa team but development team also gets more confidence and open doors to feedback SDLC loop earlier.

Is the best candidate for some testing tasks like…
Simulate virtual users that generates load / stress on server to check page throughputs, bandwidth,CPU utilization, Memory leaks and other quality parameters.
To Compare large files / database objects like Validate pictures or binary files with their expected ones.
Gathering QA statistics like Metrix and Millstones.

There are many other reasons why organization should have automation testing, we will discuss later on. One other thought I have in my mind if organization does not have fund to purchase expensive license for tools (like qtp, silktest winrunner..)
then what qa people can do? In this scenarios, It is a good opportunity to use freeware tools. My past articles can help you guys to evaluate freeware tools

"Automation Test Strategies" -- find tool selection quality parameters
"Tool Evaluation"

You can find freeware tools on http://www.opensourcetesting.org/ which sure be helpful for qa process in your organization, couple of good tools I like to recommend are as below

Functional \ Regression Automation test tools
In my past projects experience, I have successfully established an automation framework with freeware tools – WebTest Canoo and Selenium.

WebTest Canoo is on XML\Ant based tool understands HTTP request and response of the site. Everything is xml based; your test-cases, test-driver and even it produces result in xml file.

Selenium is also a good tool available as freeware.It comes with browser extension; It has its own api function library; record/play utility in built. Selenium tests run directly in a browser, just as real users. It has in built script language conversion utility that can currently support writing tests in Java, .NET, Perl, Python and Ruby.

Test Management Tools
TestLink(http://testlink.org/wordpress),
QA Track(http://www.qatrack.org/),
Bugzilla Test Runner(http://sourceforge.net/projects/testrunner/)
I like TestLink tool most as it offers functionalities such as
# Test Case history/ versioning
# Unlimited structure of Test Specification
# Support for other databases (SQL standard, generalized DB interdace)
# Improved user management (editable)
# Attachments
# Revision of Standard terminology
# Enhanced reports
# XML import/export
# Custom fields for Test Cases
# Get Traceability Metrix and Milestones
# Can create suites like regression suite or smoke suite for the test
and then create build in testlink that allow us to attach testcases
in qa, we can also assign to team for execution and testlink gives
statistics for testing progress as well.

Defect Tracking Tools
Bugzilla(www.bugzilla.org/),
BugNet(http://www.bugnetproject.com/),
Jira(http://www.atlassian.com/software/jira/)
Jira is an award winner professional issue tracking web tool. Although
it is not freeware tool but it is less expensive.

Miscellaneous tools
FreeMind is a premier free mind-mapping software written in Java. You can create test-scenarios in more easy way and publish them into hierarchical presentation in XML/HTML format.
Eclipse to compare files.
HTTPWatch tool can read HTTP headers, cookies, caching and POST data.
IECookiesView v1.70 can read IE cookies details.
Markup Validation W3 Service checks the markup validity of Web documents in HTML, XHTML, SMIL, MathML, etc.
Link Checker - Checks anchors (hyperlinks) in a HTML/XHTML document. Useful to find broken links, etc.

In my future post, I will share my customized automation tools that made to satisfy more specific testing task.

Testing Priciples & Vocabulary

2007-09-11T20:50:00.001-07:00

The principles can be defined as
1. a rule or code of conduct;
2. a general or fundamental, law, doctrine or assumption;

In software domain, these principles focus lights to test engineers on software systems, how suppose to build them and how they expect to behave.

Principle #1
Testing is the process of exercising a software component using a selected set of test cases, with the intent of
(a) revealing defects,
(b) evaluating quality.

The term “defects” used in this and in subsequent principles represents any deviations in the software that have negative impact on its functionality, performance, reliability, security and or any other of its specified quality attributes.

Principle #2
When the test objective is to detect defects, then a good test case is one that has a high probability of revealing a yet undetected defect(s).

The test engineer expects to work as scientist does like he creates hypotheses to prove or disprove them; that is determine if the specific defect is present or absent. Base on hypotheses, tester must select specific inputs; determines correct outputs and then they will be executed. Test results are analyzed to prove / disprove the hypotheses.

Principle #3
The test results should be inspected meticulously.
Get more detail from article.

Principle #4
A test case must contain the expected output or result.
Expected outputs allow the tester to determine
(1) whether a defect has been revealed, and
(2) pass/fail status for the test. It is very important to have a correct statement of the output so that needless time is not spent due to misconceptions about the outcome of a test. The specification of test inputs and outputs should be part of test design activities.

Principle #5
Test cases should be developed for both valid and invalid input conditions.
This principle opens room for negative testing. Tester should create positive as well as negative test cases; he or she must not assume that the software under test will always be provided with valid inputs. Inputs may be incorrect for several reasons. The developer of the software component may be biased in the selection of test inputs and specify only valid inputs in the test cases to demonstrate that the software works correctly but good tester is one who is more apt to select invalid inputs as well.

Principle #6 The probability of the existence of additional defects in a software component is proportional to the number of defects already detected in that component.

QA Process implemented in 0 leveled organizations

2007-09-10T08:40:00.000-07:00

Test Plan IEEE 829-1983 explained by me

2007-09-10T03:23:00.001-07:00

Automation Test Plan Strategies

2007-09-09T22:18:00.000-07:00

Tools Evaluation

2007-09-05T18:38:00.001-07:00

Hello Guys,

This is my first post on this blog; hoping you all enjoy.

Freeware Test Tool Evaluation for Functional\Regression Testing.

(a = Active featured)

Freeware Test Tool for Web Content Accessibility Testing.

(a = Active featured)

* Above evaluation is an personal evaluation based on my experience and practices.

Thank you & Enjoy Testing;
~NV.