Archive for October, 2009
Testing and debugging two activities that mutually exclusive but complimentary. Most people use these terms interchangeably. I have seen developers who claim they have tested the application and testers who claim that they are extremely good in debugging. There is a common perception among testers and developers that there is no difference between the two activities. Lets now define both the activities to understand the difference:
Testing as we all know is a process of verification and validation of the application under test. Testing on a broad spectrum is defined as questioning the software with intend to find defects. Testing is a pure QA activity.
Debugging is defined as a pure developers activity that involves executing small tests and finding issues in the code followed fixing them. Debugging involves small developement tests as a part of confirmation unit testing.
For further information, kindly comment on this blog.
Thanks,
Arun
Merged build testing is a stage before production patch deployment where several fixes to a build are merged together into a single build that needs to be tested.
The main reason why we go in for merged build testing is that, a particular fix might affect other fixes that are going into the build. In this case, a impact analysis document needs to be prepared with the list of fixes and the impact of one fix over the other. Such a document would help to narrow down on the possible areas affected across the system.
Merged Build testing can be either shaow and wide or narrow and deep. This depends on the fixes that are implemented on the particular build that is set to go live.
For further information, please write back to us or leave a comment on this blog.
Thanks,
Arun
We have seen that testing has several phases that include the very basic development testing (unit testing) followed by integration testing, system testing and finally user acceptance testing. There exists a testing phase that if often employed in most business critical application that is dependent on many internal and external interfaces. Often called as End to End (e2e) testing. e2e is often fitted between UAT and PAT (Production Assurance Testing).
Let us have a look at some case studies:
We are currently testing Credit card issuing module in a core banking application. It is critical that we not only test the CC module it is also critical that we test all the other modules which is directly or in-directly dependent on this CC system.
Let us now list down all modules (exhaustive but not complete) affected on a high level:
1. Credit Initiation (CI)
2. Print Engine – Contract needs to be printed and dispatched to the customer.
3. 3rd Party/Internal Plastic Card Issuing system – The plastics embossed and need to be encrypted with track2 data.
4. Online Services (Internet Banking) – Online banking is possible with Credit cards
5. ATMs (On-US & Off-US) – Cash withdrawal and Balance transfer is possible in case of On-US 6. trx and in case of Off-US trx the incoming payment needs to be cleared
7. Telephone Banking – All basic transactions that are possible with telephone banking
8. POS (Point of Sales – On-US & Off-US) – Most important of all, use of credit cards in POS.
System and UAT phasse of testing would deal with verifying the modules individually with more emphasis on functionality of the module. But in e2e phase, test scenarios are designed in such a way that each of the 7 modules listed above is tested for a single business process (BP). Let us sample out one business flow:
Test conditions (cases) covered in a single Business Process (BP) :
i. Customer contacts the call center/branch and orders for a Credit card.
ii. Customer details are checked and verified for credit worthiness by CI
iii. On approval, an instruction is sent to Print Engine and 3rd Party (internal) plastic generation system (Embossing and PIN generation)
iv. The Printed invitation letter along with details of the card issued – Print Engine
v. The activated card (with customer) is then used for various transactions in the below touch points:
– POS (On US & Off-US) – Check for the balance after the transaction
– ATM (On-US & Off-US) – Check for the balance after the transaction
– Phone Banking (Balance Enquiry/Transfer)
– Internet Banking – Check for the balance after each and every transaction
vi. Card maintenance – Change card status when the customer requests for a change.
– Check for statuses (Theft, Lost, Stolen, Positive Block, Post no Debit, Lost during Delivery, Damaged and Misused)
vii. Try the above transactions after changing the card status to any one of the above statuses.
ix. Issue a new card to a customer when the card status is Lost. Check if the new card is issued with proper sequence number.
x. Try some basic transactions with the newly issued card.
This is one business flow based on my experience. There atleast 5 other BP that needs to be tested for this CC system before it can be certified for production.
Importance of e2e & BP:
i. One significance of e2e testing is that we can check the hand-shake between all internal and external interfaces which would other-wise not be possible in system testing and in UAT.
ii. Emphasis is more on how the interfaces compliment each other along with the workability of individual module.
iii. Many hard to find defects can be caught when one complete BP is executed in the system as such a scenario is more realistic than test conditions that we comprehend for ST and UAT.
iv. By executing a single BP we can understand the entire work-flow or the architecture of the system which would in-turn help us to draw a cause-effect graph. Such a cause-effect would serve as a impact chart in latter stages during maintenance.
It would be wonderful if you can share your ideas on this topic (e2e and BP) which is critical for any business application to go live with minimal defects.
The Unified Modeling Language (UML) is a standard language for specifying, visualizing, constructing, and documenting the artifacts of software systems, as well as for business modeling and other non-software systems. The UML represents a collection of best engineering practices that have proven successful in the modeling of large and complex systems.1 The UML is a very important part of developing object oriented software and the software development process. The UML uses mostly graphical notations to express the design of software projects. Using the UML helps project teams communicate, explore potential designs, and validate the architectural design of the software.
The primary goals in the design of the UML were:
- Provide users with a ready-to-use, expressive visual modeling language so they can develop and exchange meaningful models.
- Provide extensibility and specialization mechanisms to extend the core concepts.
- Be independent of particular programming languages and development processes.
- Provide a formal basis for understanding the modeling language.
- Encourage the growth of the tools market.
- Support higher-level development concepts such as collaborations, frameworks, patterns and components.
- Integrate best practices.
For further information please write back to us.
Table of Content
1.Introduction
2.All testers do this…
3.Study the new project
4.Choosing Approach
5.Choosing the Executors
6.3 stages of Exploratory Testing
7.Summarizing and evaluating: merits and demerits
8.Conclusion
Introduction
When you first meet the concept of the exploratory testing you can think that its name fully exposes the sense of this sort of activity. But if you get a little bit deeper – this confidence weakens and disappears. Especially clear it can be seen when balancing between the concepts of the exploratory and wild testing, which in one hand are synonyms in QA industry, and in another (from the good sense point of view) – contrary one to another. It’s because what can have in common the exploration – serious, responsible and well-organized process – and something wild and chaotic, that is wild testing?
And nevertheless I would say that these two types are much interconnected. It is possible even to say that the wild testing is a subset of the exploratory testing, as the last one is very powerful tool, which is used for any software product, as probably no other type of testing is. You can’t start working with any application without it. Even without realizing that, a tester makes first steps in working with a product according to the written and unwritten rules of the exploratory testing.
So what is exploratory testing, when and how it is utilized, what are its goals, approaches, benefits and penalties?
We’ll discuss everything that in this article.
All testers do this…
The exploratory testing is the only type of testing which does not have a certain scenario. There are no documented Test Plan and test cases. All of scenarios are born in the tester’s mind just in the process of testing and are executed at once. Here is the main difficulty of the process – you should have a considerable base of knowledge and experience.
The main purposes of the exploratory testing are study of the software product (if it’s the first time when tester works with it) and search for the cases untaken into account in the test documentation (if it’s not the first session of work). Watching the program, you hypothesize about its possible behavior, the variants of its responses to your actions and then check up these hypotheses.
And however you can’t do it without a plan. To make the testing more effective it is recommended to create the check-list which will guide you. It’s not obligatory to have it written down but it’s strongly recommended to bear it in mind.
The ideal variant of realization of the exploratory testing is drafting and passing user-cases or tests, which are maximally close to the actions usually performed by user.
If it is the first time when the program is tested, then putting yourself on the user’s place you can find out the most critical bugs for the application functioning. And even if it’s not the first time you work with an application it’s useful to think like a user. The more time tester spends with developers the more he adopts their way of thinking and move away from user. The situation is aggravated by the shortage of time for testing. The more systematic character has the execution of test sets and the less time team has for it the less fantasy is shown by the tester. It’s hard to meet the schedule when you find an «interesting» situation and start to experiment. As the result the vision of the project stiffens, associations become very specific and linked to the concrete steps and terms. It is difficult to find something new or do the same in other way with such adaptation.
So, performing the exploratory testing always try to think in the way a user thinks!
During the testing you should not limit yourself by some rational scopes or paradigms – in fact users very often perform very inefficient actions which can have some unpleasant results.
Try to analyze, what functionality is utilized more frequently and what is touched much rarer. This information will help you to distribute your attention among different features.
The important initial stage of the exploratory testing is formulating the goal. Ask yourself: «Why do I do it? For what purpose?» After completing the testing try to measure your success.
The list of the most widespread goals
- Study of the new product;
- Exposure of omissions in the test-plan. The result is to add non-standard tests to the test coverage;
- Performing the testing of the project with incomplete or inexact specification or when the specification is absent;
- Research of the specific defect;
- Improvement of the product quality after completion of the basic (for example, functional) testing.
- Study the new project
Mostly common the exploratory testing is used as the tool for the study of the new project or functionality.
In this situation you should try to learn all accessible information before starting the testing process.
Documentation is on the first place, as usual. If it’s present, of cause… Specifications, requirement specifications will always remain the faithful helpers of tester.
If consider the process of the exploratory testing as the building of a house, then the documentation is its reliable foundation.
The second step is to ask developers and project managers. They always possess necessary knowledge about the project, therefore do not ignore ANY useful information, ask them questions. My experience shows that this meeting can considerably change your priorities and add new test cases.
Choosing approach
The next stage in building of our house is to choose a project. How we will build? What materials will be used? Where will windows and doors be?
Returning to the QA industry terminology we should choose testing approach. There exist a lot of them, we will consider a few basic ones:
Basic functionality. It is especially important when we have tough time constraints. In this case we shouldn’t get into details and get stuck in some little things. First of all it is necessary to check up main functionality and make sure in its capacity for work.
Places where bugs appear the most frequently. Such places can be the new functionality or those parts of the program which are the most frequently rebuilt. Certainly, this approach is applicable only in case if it’s not the first time you work with the program, and already know its thin places.
The simplest use-cases (for the new build validation). If you’ve got the new version, or version after the deep refactoring then don’t be too lazy to check up the most ordinary use-case, and if it fails return this version to the developers with confidence.
Set of the “favorite” bugs. It will work if the tester knows the programmer well. Each developer has the set of the “favorite” bugs – and each software solution has. When there is much experience of testing even the exploratory testing can be performed by the algorithm.
Concentration on everything and on nothing concretely. This approach is slightly philosophical. Begin to work with the application as a user and concentrate your attention on everything and on nothing concretely. It’s just like you watch from the outside what is going on. Don’t try to analyze anything while working but be ready to start the analysis at any moment and reproduce all of the steps.
Vulnerability. Those who know the design and UML will get it easier. The schematic model of the project is created, and the weakest place is determined using different methods and approaches. Analysis can be performed from the position of user, architecture or realization. Unlike the usual test design here we analyze the specific model – mixture of architecture of business processes and human perception.
Choosing the executors
So we chose the project for our house J. Next question is about the workers. Who will be assigned to this responsible task?
The first are QA specialists. When choosing the responsible tester you should estimate his experience. It is not inconceivable that assigning an inexperienced man to perform the exploratory testing of the new product you will spend time for unavailing mouse clicking.
It is good to get not only QA specialists to take part in the exploratory testing but also other team members, for example developers. Frequently a tester thinks deeper than a user usually does and so sometimes he complicates a task while not seeing the most obvious errors. The exploratory testing will be good here with its fresh look to the situation.
After performing the regression testing of the product many companies invite independent experts who perform the exploratory testing, simulating the behavior of end users. Expenses on such testing usually justify themselves, as far as critical bugs will be found and fixed before the release.
3 stages of exploratory testing
Watch Out! Construction Work!
Many sources describe the process of the exploratory testing as the following cycle of actions:
study of the product -> planning of tests -> implementation of tests
And everything this, as already mentioned higher, is not fixed in documents but takes place only in the tester’s mind. It is another argument in behalf on that the exploratory testing should be performed by really experienced specialists: it is the only type of testing, which includes 3 stages of testing of software product simultaneously.
The following questions can help in the process of planning of tests:
- And what will happen if …?
- And how the program will respond on such my actions?
- What data can I get?
Summarizing we can say that the slogan of the exploratory testing is: «Imagine the problem, and then find it».
Summarizing and evaluating: merits and demerits
Walls are ready. Just one small thing is left to do– to construct the roof.
In this section we’ll sum up and, as a result, describe pluses and minuses of the exploratory testing.
Merits of the exploratory testing:
1.Exposes functionality, uncovered by tests;
2.Helps to find the skipped bugs;
3.Allows to look at the product from the end user point of view;
4.Helps to realize the mission of the tested product, improves understanding of it;
5.Requires little preliminary work as far as it’s not too formalized;
6.Helps to find out the majority of critical bugs almost at once;
7.It is acceptable in terms of lack of resources;
8.Discovers usability bugs.
Demerits of the exploratory testing:
1.It is impossible to estimate time for this testing, and, consequently, to set the terms of project completion;
2.It is hard to estimate the project if only exploratory testing is used – it is impossible to make statistics of passed and to failed tests, which is one of basic indexes of product quality.
3.There is no other possibility to check up results but just fully retest the project;
4.It’s impossible to review test cases;
5.The exploratory testing never can be performed identically for several times. Some tests will slip out each time, and so there is a risk to skip some bugs;
6.Results depend much on the experience of tester.
Conclusion
You can read music and you can improvise. Any skills suppose, foremost, that you master some basic techniques and also have certain framework to use as a ground of improvisation.
The same is true for the testing. The exploratory testing is improvisation, the highest degree of tester mastery. It is a complex creative process requiring significant efforts. To perform really qualitative exploratory testing tester should have a lot of experience.
Perform the exploratory testing for the first acquaintance with a product, and it will help to study and understand your application.
Execute the exploratory testing simultaneously with the regressive testing – you will promote the quality of your product, discovering the serious non-standard errors just on the testing stage.
Perform the exploratory testing after basic one and you will be able to discover and remove bugs that slipped out before.
The process metrics questions focus on the degree to which the software engineering process is quantified and measured. Typical metrics concern software quality, the amount of code developed, resources used, and such progress indicators as review coverage, test coverage, and test completion.
- Are any metrics defined and applied for measuring the quality of the software product?
- Are there any metrics to rate the quality of the software process?
- Are quality measures used to identify weak points of the product and the process and to verify quality criteria?
- Are measures used for initiating corrective actions, if a measurement value deteriorates or exceeds given boundaries?
- Are statistics on software design errors gathered?
- Are software staffing profiles maintained of actual staffing versus planned staffing?
- Are software problem reports tracked to closure?
- Are the cost and effort needed to correct errors measured and recorded?
- Are statistics on software code and test errors gathered?
- Are profiles maintained of actual versus planned software units designed, over time?
- Are profiles maintained of actual versus planned software units completing unit testing, over time?
- Are measures used for introducing product and process improvements?
- Are all data collected during the project recorded and evaluated?
- Are process and product metrics (e.g. productivity and effort) used to plan following projects?
- Are there any mechanisms to regularly review whether the defined metrics are still adequate and representative?
- Do you plan for measurement activities?
- Are productivity and effort measured and recorded for each phase of the project?
- Are profiles of software size maintained for each software configuration item?
- Are profiles of software complexity maintained for each software item?
- Are profiles maintained of actual versus planned software units integrated, over time?
- Are target computer memory utilization estimates and actuals tracked?
- Are target computer throughput utilization estimates and actuals tracked?
- Are design and code review coverages measured and recorded?
- Is test coverage measured and recorded for each module tested?
These days a number of web sites are deployed in multiple languages. As companies perform more and more business in other countries, the number of such global multi-lingual web applications will continue to increase.
Testing web sites supporting multiple languages has its own fair share of challenges. In this article, I will share seven tips with you that will enable you to test the multi-lingual browser-based applications in a complete way:
Tip # 1 – Prepare and use the required test environment
If a web site is hosted in English and Japanese languages, it is not enough to simply change the default browser language and perform identical tests in both the languages. Depending on its implementation, a web site may figure out the correct language for its interface from the browser language setting, the regional and language settings of the machine, a configuration in the web application or other factors. Therefore, in order to perform a realistic test, it is imperative that the web site be tested from two machines – one with the English operating system and one with the Japanese operating system. You might want to keep the default settings on each machine since many users do not change the default settings on their machines.
Tip # 2 – Acquire correct translations
A native speaker of the language, belonging to the same region as the users, is usually the best resource to provide translations that are accurate in both meaning as well as context. If such a person is not available to provide you the translations of the text, you might have to depend on automated web translations available on web sites like wordreference.com and dictionary.com. It is a good idea to compare automated translations from multiple sources before using them in the test.
Tip # 3 – Get really comfortable with the application
Since you might not know the languages supported by the web site, it is always a good idea for you to be very conversant with the functionality of the web site. Execute the test cases in the English version of the site a number of times. This will help you find your way easily within the other language version. Otherwise, you might have to keep the English version of the site open in another browser in order to figure out how to proceed in the other language version (and this could slow you down).
Tip # 4 – Start with testing the labels
You could start testing the other language version of the web site by first looking at all the labels. Labels are the more static items in the web site. English labels are usually short and translated labels tend to expand. It is important to spot any issues related to label truncation, overlay on/ under other controls, incorrect word wrapping etc. It is even more important to compare the labels with their translations in the other language.
Tip # 5 – Move on to the other controls
Next, you could move on to checking the other controls for correct translations and any user interface issues. It is important that the web site provides correct error messages in the other language. The test should include generating all the error messages. Usually for any text that is not translated, three possibilities exist. The text will be missing or its English equivalent will be present or you will see junk characters in its place.
Tip # 6 – Do test the data
Usually, multi-lingual web sites store the data in the UTF-8 Unicode encoding format. To check the character encoding for your website in mozilla: go to View -> Character Encoding and in IE go to View -> Encoding. Data in different languages can be easily represented in this format. Make sure to check the input data. It should be possible to enter data in the other language in the web site. The data displayed by the web site should be correct. The output data should be compared with its translation.
Tip # 7 – Be aware of cultural issues
A challenge in testing multi-lingual web sites is that each language might be meant for users from a particular culture. Many things such as preferred (and not preferred) colors, text direction (this can be left to right, right to left or top to bottom), format of salutations and addresses, measures, currency etc. are different in different cultures. Not only should the other language version of the web site provide correct translations, other elements of the user interface e.g. text direction, currency symbol, date format etc. should also be correct.
As you might have gathered from the tips given above, using the correct test environment and acquiring correct translations is critical in performing a successful test of other language versions of a web site.
In a round-robin review, each participant is given an equal opportunity to study and present the evaluation of the product being reviewed.
A round-robin review can be performed during any phase of the product life cycle and is also useful for documentation review. In addition, some variations of the round-robin review incorporate the best features from other peer review forms but continue to use the alternating review leader approach. For example, during a round-robin inspection, each item on the inspection checklist is made the Responsibility of alternating participants.
The usual number of personnel involved in this type of peer review is four to six. The meeting is scheduled by the producer, who also distributes high-level documentation as input. The producer is either the first review leader or assigns this responsibility to another participant. The temporary leader guides the other participants (e.g., implementers, designers, testers, users, and maintenance representatives) through the first unit of work. This unit may be a module, paragraph, line of code, inspection item, or other unit of manageable size. All participants (including the leader) have the opportunity to comment on the unit before the next leader begins the evaluation of the next unit. The leaders are responsible for noting major comments raised about their piece of work. At the end of the review, all the major comments are summarized and the group decides whether to approve the product. No formal mechanism for review follow-up is used.
It is also a company level document and developed by QA people. It defines testing approach followed by testing team.
Components in Test Strategy:
Scope and Objective: About testing need and their purpose
Business Issues:
Budget control
for testing in terms of time and cost
100%—-Project Cost
64% Development & Maintenance and 36% for Testing
Test Approach: It defines mapping between development stages and testing issues
Test Matrix (TM)/ Test Responsibilities Matrix (TRM)
Test Deliverables:
Required documents to prepare during testing of a project
Ex: Test Methodology, Test Plan, Test case etc.,
Roles & Responsibilies: Names of jobs in testing team and their responsibility
Communication and Status Reporting: Require negotiations between two consecutive job in testing team
Automation Testing Tools: Need of automation in our organization level project testing
Testing Measurements and Metrics: QAM, TMM, & PCM
Defect Reporting and Tracking: Required negotiations between testing team and development team
Risks & Mitigations: Possible risks and mitigations to solve( risks indicates a future failure)
Change and Configurations Management: How to handle change requests coming from customers during testing and maintenance
Training Plan : Need of training to tester before starts of every project testing
To define quality S/W, quality analyst defines 15 testing issues. Test factor or issue means that a testing issue to apply on S/W to achieve quality.
The test factors are:
- Authorization: Whether user is valid or not to correct application
- Access Control: Authorized to access specific services
- Audit Trail: Meta data about user operations
- Continuity of processing: Inter process communication (IPC) during execution.
- Correctness: Meet customer requirements in terms of inputs and out puts
- Coupling : Co-existence with other existing S/W
- Ease of Use : User friendliness of screens
- Ease of operate : Installation, un installation, dumping, exporting etc.,
- File integrate: creation of internal files (ex back up)
- Reliability: Recover from abnormal situation
- Portable : Run on different plat forms
- Performance : speed of processing
- Service Levels : order of services
- Methadology : Follow standards
- Maintainable : Long time serviceable to customers
Arun Vijayaraghavan is a Senior QA working for one of the world's largest Investment Banks. Arun has extensive experience in Retail Banking, Investment banking and non-life insurance domains.
