Foreword to Testing Software
Why it's important to test software
There's an idea in software engineering that bugs become more expensive as they age. Sure, bugs and defects in software are not always catasrophic but that is not an absolute. Regardless of the discipline or industry, we should all agree that we want less issues in our software.
Cost of Fixing Bugs - Sketchplanations
A Short View to the Past...
Before we start taking specifics about testing, it would be best to start with a story (or three). The following are multiple instances where a lack or lapse in testing caused critical problems or failures:
The Mars Pathfinder Mission marked the first occurrence of a rover on the surface of Mars. While ultimately sucessful in it's objective of deploying the Sojourner rover on Mars, an irregular exception occured shortly into the mission's timeline. After deploying the Sojourner rover a system reset occurred due to an expiring watchdog timer. According to transcripts by engineers involved in the project, the culprit turnedd out to be a priority inversion issue where a lower priority peripheral task ended up blocking a higher priority system task by claiming a shared resource. The consequences of this issue were premature system shut downs. Fortunately, it appears that any data collected was not lost but the premature shut down does mean that the many of the missions objective's had to be curtailed or adjusted due inactivity and loss of time.
The Mars Climate Orbiter was one of the initial missions to Mars with the objective of studying weather patterns on the surface and acting as a communication relay for future missions. While the spacecraft did reach Mars, an incident occurred during orbital insertion which resulted in loss of the spacecraft and ultimately mission failure. It was discovered later that there was a navigation error that resulted in a loss of communication and the subsequent destruction of the spacecraft due to atmospheric forces. A independent review by the Mishap Investigation Board (MIB) later revealed that the culprit behind the incident was a mismatch in units in ground software between the two development groups, Lockheed and NASA. While there were other aministrative factors influencing the project which likely lead to less ideal design choices, objectively, an aspect that should have been ironed out at the beiginning of the project went unchecked.
A more recent example of a notable issue would be Crowdstrike cybersecurity update that caused many Windows based system to stop working for a brief period.Systems and infrastructure based on the Windows operating system experience irregular and unprecedented blue screens of death (BSOD) which render a significant number of services and tools unusable for an extended period of time. Multiple industries and organizations were directly impacted including but not limited to motorsports, financial institutes (i.e. bank), transportation services, etc. In regard to the cuase of the mass outtage, there were a multiude of factors and findings that have been detailed in Crowdstrike's Technical Root Cause Analysis - Channel File 291 document. The document itself is a interesting read but it indicates that the main culprits behind the events that occurred were a runtime array indexing out of bounds error that did not appear during testing, potential under-coverage of testing criteria and constraints, and possible design flaws in deployment process.To highlight the damages even further, Delta Airlines was reported to have $500 million in damages from the incident.
Morale of the stories, all of these issues could have been caught and prevented with stronger testing techniques.
What Testing is not
It's important to note that testing software is not a silver bullet. Testing shows the presenece of bugs not their absence. This may seem counter-intuitive but finding a bug in testing (in a sense) is a good thing. Once you know that the bug exist, it can be fixed but not knowing it's there is a problem waiting to happen (probably).
So what does testing software mean?
We can define testing software as the process of checking if a program, application, snippet, library, or some code does what it is expected to do. That's a start but it's more nuanced than that which we shall see from the following guides.
Challenges with Properly Testing Software
There are many obstacles, inconveiences, and challenges that can be found from testing. Just to name a few:
- Lack of documentation
- Lack of domain-knowledges
- Poor requirements
- Can be extremely time consuming
- Requires considerable effort and attention
- Lack of a proper testing environment
- Missing equipment or hardware*
This list is only scratching the tip of the iceberg regarding the difficulties you may face during testing, there are ways to mititgate the negative effects these problems can have on your projects. Understanddably, writing and executing test cases can be tiring and most developers.