Archive for the ‘Software Testing’ Category
Release Management for Testers
The release management life cycle is an important work-flow which every testers needs to understand for an efficient and an effective environment setup.
Test-bed/Test environment is defined as the workspace in which the AUT is deployed for testing. Below are a set of steps that need to be followed sequentially to make the test environment fit for testing.
i. Test environment refresh (Refresh from Production or other mirrors) – Typically this refresh can be either incremental or complete. This would result in erasing the current data in the test environment and populating the same with latest production code. In case of refresh all test data in the test environment would be erased.
ii. Static Data setup – All static data in the test environment can either be setup manually or the environment can be refreshed only for static data flash. An automatic static data refresh would copy all static data from production tables to the test environment.
NOTE: In certain cases, both environment refresh and static data setup can be performed with a single refresh request.
iii. Test data setup – All the test data that needs to be either created or pumped into the test environment from production needs to performed before a new build is deployed into the test environment. Usually, test data setup activity is performed after a complete refresh.
iv. Build Request – The build and configuration team would be responsible for creating the test build.
v. Build Release – Once the build is ready, a separate request needs to be placed for the build to be deployed into the test environment.
vi. Request for any external interfaces – If in case there are any external interface requirement, this needs to be raised to the environment support team would be able to link these interfaces with the test environment.
Once the build is deployed, sanity checks are performed to check the fitness of the build. Depending on the build acceptance criteria, the build is passed for testing.
When ever a new build is released, the release management team would be involved in preparing the release notes which would be the basis for the tester to understand what modules are present in the build that would be tested. These release notes are extremely helpful in case of iterative development models in which several iterations of the application is deployed over a period of time.
A Software Test Life Cycle is defined as a sequential and heirachial strategy for verification and validation of the application under test.
Testing is considered to be the most important part of software quality. There are certain principles that needs to be adopted in case testing the application. The principles of software testing as below:
- Testing shows the presence of defects – There is no way that one can detect issues in the application without actually testing the application. Testing can be either static or Dynamic depending on the phase in STLC.
- Early testing – Earlier we start testing the lesser it costs to fix the defects. Defects that are found in the latter stages are much difficult to fix rather than the ones which are found in the earlier phase.
- Testing is context dependent
- Absense of error fallacy – Testing the application completely doesn’t ensure that the application is delivered defect free.
- Defect clustering – There is a possibility that while testing a particular module heavly, more number of defects are found in that module.
- Pesticide Paradox – Repeated testing with the same set of test cases evetually renders as the no more defects can be found in that particular module in the application. Repeated modification and tweaks to the test cases/scenarios are required to enhance the testing scope and test accuracy.
Below is a sample Software Test Life Cycle model that we usually use in our internal projects.
- Requirements analysis: Testing should begin in the requirements phase of the software development life cycle. During the design phase, testers work with developers in determining what aspects of a design are testable and with what parameters those tests work.
- Test planning: Set of activities that involve preparation of Test strategy, test plan and testbed setup together come under Test planning.
- Test development: Test procedures, test scenarios, test cases, test datasets, test scripts to use in testing software.
- Test execution: Testers execute the software based on the plans and test documents then report any errors found to the development team.
- Test reporting: Once testing is completed, testers generate metrics and make final reports on their test effort and whether or not the software tested is ready for release.
- Test result analysis: Or Defect Analysis, is done by the development team usually along with the client, in order to decide what defects should be treated, fixed, rejected (i.e. found software working properly) or deferred to be dealt with later.
- Defect Retesting: Once a defect has been dealt with by the development team, it is retested by the testing team. AKA Resolution testing.
- Regression testing: It is common to have a small test program built of a subset of tests, for each integration of new, modified, or fixed software, in order to ensure that the latest delivery has not ruined anything, and that the software product as a whole is still working correctly.
- Test Closure: Once the test meets the exit criteria, the activities such as capturing the key outputs, lessons learned, results, logs, documents related to the project are archived and used as a reference for future projects.
- Post Implementation Support: Once the application/patch does live, testers and support BA are involved in providing post implementation support that involves sanity checks on the live application.
- Warranty Support: Once the build goes live, certain project teams have a warranty phase where in any reportable event due to the current build would need to fixed, tested and pushed into live once again as a warranty fix.
Different Software Development models have different features and properties. Selection of the software development model depends on the nature of project and client. Here, I will try to give a comparison of various software development models with three parameters:
1. Contribution to Quality
2. Risks Associated
3. Context of adoption
Model Name: Waterfall Model
Contribution to Quality: Phase End Checks
Risks Associated: Expects a task to be well done in the first go
Context of adoption: When the requirements are structured and competence is high
Model Name: Software Development Lie Cycle Model (ETVX Model)
Contribution to Quality: Entry task verification exit definition and quality control through feedback.
Risks Associated: Final Product available only after the complete cycle
Context of adoption: When the requirements are quite structured, but scale is large. One may need to go back for rework if required.
Model Name: Prototype Model
Contribution to Quality: 1. Seeing is believing. 2. Go iterative and involve customer. 3. Produce working models to give faster delivery and have concrete feedback.
Risks Associated: The scrap may go uncontrollable
Sponsored Links: Context of adoption: When needs to try out things before making a commitment to deliver.
Model Name: Spiral Model
Contribution to Quality: 1. Avoid scrap as well as rework – do right the first time. 2. Analyze risks before undertaking the next enhancement.
Risks Associated: Inadequate experience and subjective method of risk management
Context of adoption: When the scale is large and planned; passed development would give confidence to move ahead or stop as suited.
Model Name: V-Model
Contribution to Quality: Plan for testing of artifacts much before their actual completion
Risks Associated: Testing techniques applied may remain subjective
Context of adoption: It is a philosophy that can be applied with any other models you choose.
Model Name: Unified Process Model
Contribution to Quality: 1. Iterative evolutionary use case centric development. 2. Defines workflows and milestones for better project management.
Risks Associated: Identification of phases needs experience.
Context of adoption: When the scale of project is large.
Model Name: Agile Models
Contribution to Quality: Anytime delivery through flexible design, limited scope and quick reviews.
Risks Associated: Flexible design techniques not well established. The designers lack an understanding of business need for agility.
Context of adoption: When business needs are dynamic and need to be fulfilled immediately.
Testing the software system or software application as a whole is referred to as System Testing of the software. System testing of the application is done on complete application software to evaluate software’s overall compliance with the business / functional / end-user requirements. The system testing comes under black box software testing. So, the knowledge of internal design or structure or code is not required for this type of software testing.
In system testing a software test professional aims to detect defects or bugs both within the interfaces and also within the software as a whole. However, the during integration testing of the application or software, the software test professional aims to detect the bugs / defects between the individual units that are integrated together.
During system testing, the focus is on the software design, behavior and even the believed expectations of the customer. So, we can also refer the system testing phase of software testing as investigatory testing phase of the software development life cycle.
At what stage of SDLC the System Testing comes into picture:
After the integration of all components of the software being developed, the whole software system is rigorously tested to ensure that it meets the specified business, functional & non-functional requirements. System Testing is build on the unit testing and integration testing levels. Generally, a separate and dedicated team is responsible for system testing. And, system testing is performed on stagging server.
Why system testing is required:
- It is the first level of software testing where the software / application is tested as a whole.
- It is done to verify and validate the technical, business, functional and non-functional requirements of the software. It also includes the verification & validation of software application architecture.
- System testing is done on stagging environment that closely resembles the production environment where the final software will be deployed.
Entry Criteria for System Testing:
- Unit Testing must be completed
- Integration Testing must be completed
- Complete software system should be developed
- A software testing environment that closely resembling the production environment must be available (stagging environment).
System Testing in seven steps:
- Creation of System Test Plan
- Creation of system test cases
- Selection / creation of test data for system testing
- Software Test Automation of execution of automated test cases (if required)
- Execution of test cases
- Bug fixing and regression testing
- Repeat the software test cycle (if required on multiple environments)
Contents of a system test plan: The contents of a software system test plan may vary from organization to organization or project to project. It depends how we have created the software test strategy, project plan and master test plan of the project. However, the basic contents of a software system test plan should be:
- Scope
- Goals & Objective
- Area of focus (Critical areas)
- Deliverables
- System testing strategy
- Schedule
- Entry and exit criteria
- Suspension & resumption criteria for software testing
- Test Environment
- Assumptions
- Staffing and Training Plan
- Roles and Responsibilities
- Glossary
How to write system test cases: The system test cases are written in a similar way as we write functional test cases. However, while creating system test cases following two points needs to be kept in mind:
- System test cases must cover the use cases and scenarios
- They must validate the all types of requirements – technical, UI, functional, non-functional, performance etc.
As per Wikipedia, there are total of 24 types of testings that needs to be considered during system testing. These are:
GUI software testing, Usability testing, Performance testing, Compatibility testing, Error handling testing, Load testing, Volume testing, Stress testing, User help testing, Security testing, Scalability testing, Capacity testing, Sanity testing, Smoke testing, Exploratory testing, Ad hoc testing, Regression testing, Reliability testing, Recovery testing, Installation testing, Idem potency testing, Maintenance testing, Recovery testing and failover testing, Accessibility testing
The format of system test cases contains:
- Test Case ID – a unique number
- Test Suite Name
- Tester – name of tester who execute of write test cases
- Requirement – Requirement Id or brief description of the functionality / requirement
- How to Test – Steps to follow for execution of the test case
- Test Data – Input Data
- Expected Result
- Actual Result
- Pass / Fail
- Test Iteration
SDLC is a systematic and orderly approach to solving problems related to software systems or in other words, we can say it is a structure imposed on the development of a software product.
Domain Analysis: This phase is very important. The more knowledgeable you are about the domain, less the work required. Another objective of this phase is to make the analysts who will later try to elicit and gather the requirements from the area experts or professionals. So, this phase is an important prelude to extracting and gathering the requirements.
Requirement Analysis: The most important task in creating a software product is extracting the requirements. Customers typically know what they want, but not what software should do, while incomplete, ambiguous or contradictory requirements are recognized by skilled and experienced software engineers.
Scope Analysis: Once the requirements are gathered from the client, an analysis of the scope of the development should be determined and clearly stated. This is often called a scope document.
Specification: It is the task of precisely describing the software to be written, possibly in a rigorous way. In practice, most successful specifications are written to understand and fine-tune applications that were already well-developed, although safety-critical software systems are often carefully specified prior to application development. Specifications are most important for external interfaces that must remain stable. A good way to determine whether the specifications are sufficiently precise is to have a third party review the documents making sure that the requirements are logically sound.
Software Architecture/Design: Architecture is concerned with making sure the software system will meet the requirements of the product, as well as ensuring that future requirements can be addressed. The architecture step also addresses interfaces between the software system and other software products, as well as the underlying hardware or the host operating system.
Coding: The design must be translated into a machine-readable form. The code generation step performs this task. If the design is performed in a detailed manner, code generation can be accomplished without much complication. Programming tools like compilers, interpreters etc… are used to generate the code. Different high level programming languages like C, C++, Pascal, Java are used for coding. With respect to the type of application, the right programming language is chosen.
Testing: Once the code is generated, the application / software testing begins. Different testing methodologies are available to unravel the bugs that were committed during the previous phases. Different testing tools and methodologies are already available. Some companies build their own testing tools that are tailor made for their own development operations.
Implementation: After the code is appropriately tested and approved, it is made available for business use i.e. moved into production environment.
Documentation: An important task is documenting the internal design of software for the purpose of future maintenance and enhancement.
Software Training and Support: As a part of the deployment phase, it is very important to have training classes for the software user. Users will have lots of questions and software problems which leads to the next phase of software.
Maintenance: Maintaining and enhancing software to cope with newly discovered problems or new requirements can take far more time than the initial development of the software.
Below is a sample checklist for Release Readiness Review and Pre Delivery Audit. It is useful both for Project Managers and Software Quality Analysts.
- Is delivery going against approved Contract/Statement of work/work order? If not, is the mail for approval from senior management available for the delivery?
- Has the requirements document / FSD validated and approved by client?
- Are all the requirements implemented as per approved requirement document or specification document or as agreed with the client? If not, is it documented in the release notes / delivery note etc and approval taken from customer / delivery head?
- Are all Non-technical / non-functional requirements satisfied / completed?
- Is the build verification done before each Testing?
- Are all test cases updated to ensure complete test coverage?
- Are all cycles of testing completed and defects closed ? If any defcet is open, is it documented in the release notes / delivery note etc and approval taken from customer / delivery head.
- Are all review defects closed?
- Is the test Coverage Goal met as mentioned in test plan?
- Is the traceability from requirements to source code to test cases to test results documented and available for review?
- Is the goal for System testing / functional testing or any other type of testing is met?
- Are all the test cases for acceptance testing identified and documented?
- Are all audits done for the all items as identified in the software quality assurance plan?
- Are the findings of FCA / PCA / other audits closed (if any) ?
- Has the test cases executed on target environment (or similar of target environment) ?
- Is acceptance criteria for system validation available?
- Does end product / application meets the acceptance criteria during system testing ?
- Is the code modification history updated and code files are labled properly in the configuration tool?
- Are all the work products, deliverables and code consistent?
- Are the CIs of the entire release package identified?
- Is each CI in the release package CI tested / reviewed / approved?
- Are all findings of testing / review of CIs closed?
- Is the FTP credentials or CD key etc documented? Are these communicated to client / customer?
- Is release media and its access verified before sending the information to client?
- Are the following documents / deliverables / work products ready: FSD, ADS, DDS, Test Plan, Test Case, Release Note, Installable Product, Source Code, Test Programs, Drivers, Read me File, License File, Known Problems List, Installation Procedure, Install Scripts, User Manual, Programmer’s Manual, Administrator Manual, Test Reports, Training Material or any other deliverable required by client?
- Is the delivery made from the static folder of VSS?
Apart from above points, the status of below points can also be checked:
- Baselining of deliverables in configuration management tool
- End of project backup and restoration of the backup
- Readiness of maintenance / support plan
- Is the problem reporting procedure documented and communicated to client?
- Are the contact persons documented and communicated to client??
- Is the post release defect reporting template ready?
- Whether request has been made to IT for making the project repository under VSS read only?
The failure mode and effects analysis and a variant including criticality analysis (FMECA) are iterative activities, intended to analyze the effects and criticality of failure modes within a system. The application of these analyses to software is sometimes termed SFMEA and SFMECA where S stands fot software.
Testers must be able to contribute to the creation of the FMEA document. This includes understanding the purpose and application of these documents as well as being able to apply their knowledge to help determine risk factors.
Areas of application:
FMEA should be applied:
1. where the criticality of the software or system under consideration must be analyzed to reduce the risk of failure.
2. Where mandatory or regulatory requirements apply
3. To remove defects at an early stage
4. to define special test considerations, operational constaints, design decisions for safety critical systems
Implementation steps:
FMEA should be scheduled as soon as preliminary information is available at a high level and extended to lower levels as more details become available.
For each critical function, module or component, iteratively:
1. Select a function and determine its possible failure modes, i.e. how the function can fail.
2. Define the possible causes for those failures, their effects and design mechanisms for reduction or mitigation of failures.
Befenits and Costs:
FMEA provides the following advantages:
1. Expected system failures caused by software failures or usage errors can be revealed.
2. Systematic use can contribute to overall system analysis
3. Results can be used for design decisions and justifications
4. Results may be used to focus testing to specific areas of the software.
Thanks,
Arun
www.focustesting.com
Integrated Work flow Model:
IST: Integrated system testing otherwise known as System Integration Testing is one such testing phase in which all the internal interfaces of the system is tested from both functional and operational perspective.
In IST we would concentrate more on the functionality of the system along with the integrity when it communicates with its interfaces. Functional testing techniques such as Boundary Value Analysis (BVA), Equivalence Partitioning and use case based testing, State transition table and decision table analysis. Since the system tends to fail more on the boundaries we give more emphasis on BVA and Equivalence class.
TAT – Technical acceptance testing
TAT is a testing phase in which the system is tested if it is acceptable as per the technical specifications. Once SIT is complete the system is passed for UAT or TAT. UAT/TAT is the final phases before the Production. In UAT phase, the business users typically the end users test the application from end to end business perspective.
If the system undergoes a change from the functional perspective then the system needs to be UAT’ed before the new build goes into production. But at the same time, when there are changes to the system from the technical perspective like change in the load balance of the servers, then the system needs to under go load and stress testing. These phases of testing come under TAT.
Once UAT or TAT is complete then the system would go into preproduction where the system is checked for back-out and finally it is released into production.
In other words TAT can be defined as a phase that verifies and validates the technical compliance of software regarding to the production’s standard.
A test suite that exercises the full scope of a system from a top-down perspective, but does not test any aspect in detail
Testing as we all know is a verification and validation process that certifies that a product works as per the requirements. But testing as per requirements alone wouldn’t make the product stable. This is where we introduce a stage/phase that we call as Dirty testing. This phase is also known as “Creative Testing“.
Dirty testing involves testing creative side of the equivalence partation. There are certain conditions in which the application would surely break if tested with test data other than the ones mentioned in the requirements. Dirty testing also involves testing out of box. Sometimes a tester’s experience of testing similar products plays a vital role in determining the depth in which negative testing is carried out.
Let us discuss a scenario where I had employed creative a.k.a dirty testing. The requirements does not always cover every business aspect. Based on your product experience If you find any gaps in the technical & functional requirements then you are sure to find gaps in the application which would ultimatly lead of failure.
Case Study: I was involved in the UAT phase of Testing an ATM (Automated Teller Machine) interface with Core Banking application.
All test cases where written by business users strictly based on BRD and FS. The software was strictly developed based on the BRD & FS. Not enough care was taken to program the application’s behaviour for alternative paths.
The main functions of the ATM machine are listed below:
-
Cash Withdrawal
-
Balance Enquiry
-
Fund Transfer
-
Cash Deposit
-
Mobile Phone Recharge
All test cases for the above scenarios where based on happy path and never considered the dirty path. Let us have a look at sample test cases for Cash Withdrawal:
Strictly Requirement Based Cases – Cash Withdrawal:
-
Cash Withdrawal from Current account using Maestro card
-
Cash Withdrawal from Savings account using Maestro card
-
Cash Withdrawal from current account when there is not enough balance
-
Cash Withdrawal from savings account when there is not enough balance
- Cash withdrawal using a blocked card using Maestro card
- Cash withdrawal using an expired card using Maestro card
Even though the above test cases covers both positive and Negative paths in the high level cash withdrawal scenario, the very aspect of alternate path is not covered. Out of my experience let me list down a sample of dirty test cases that I have come up with:
-
Cash withdrawal using a utility card
-
Cash withdrawal from a revolving credit card
-
Cash withdrawal using a store card
-
Cash withdrawal using damaged card
Well, all the 4 dirty test cases failed and the system almost crashed resulting in Sev-1 defects. Real show stoppers. This is when we decided, we would cover regular BRD/FS aligned test cases along with Dirty test cases which would surely improve the quality of the product delivered.
It would be great if you as readers can also share your experiences in creating dirty test cases. Please do write to me at arun.vijayaraghavan@focustesting.com
Thanks,
Arun
Arun Vijayaraghavan is a Senior QA working for one of the world's largest Investment Banks. Arun has extensive experience in Retail Banking, Investment banking and non-life insurance domains.
