DHS’s BioWatch program aims to provide early indication of an aerosolized biological weapon attack. Until April 2014, DHSpursued a next-generation autonomous detection technology (Gen-3), which aimed to enable collection and analysis of air samples in less than six hours, unlike the current system (Gen-2), which requires manual intervention and can take up to thirty-six hours to detect the presence of biological pathogens. A GAOreport found that DHS lacks reliable information about BioWatch Gen-2’s technical capabilities to detect a biological attack, and therefore lacks the basis for informed cost-benefit decisions about upgrades to the system.
DHS’s BioWatch program aims to provide early indication of an aerosolized biological weapon attack. Until April 2014, DHS pursued a next-generation autonomous detection technology (Gen-3), which aimed to enable collection and analysis of air samples in less than six hours, unlike the current system (Gen-2), which requires manual intervention and can take up to thirty-six hours to detect the presence of biological pathogens.
DHS is taking steps to address the capability gap which resulted from the April 2014 cancellation of Gen-3 by exploring other technology upgrades and improvements to the Gen-2 system.
A Government Accountability office (GAO) was asked to review the technical capabilities of the currently deployed BioWatch system (Gen-2); the Gen-3 testing effort; and the characteristics of autonomous detection as a possible option to replace the current BioWatch system.
GAO says it analyzed key program documents, including test plans, test results, and modeling studies. GAO assessed Gen-3 testing against best practices, reviewed relevant literature, and discussed the BioWatch program and testing efforts with key agency officials and national laboratories staff.
BioWatch Gen-2: Unreliable information
A Government Accountability office (GAO) report found that the Department of Homeland Security (DHS) lacks reliable information about BioWatch Gen-2’s technical capabilities to detect a biological attack, and therefore lacks the basis for informed cost-benefit decisions about upgrades to the system.
DHS commissioned several tests of the technical performance characteristics of the current BioWatch Gen-2 system, but the department has not developed performance requirements which would enable it to interpret the test results and draw conclusions about the system’s ability to detect attacks. GAO notes that although DHS officials said that the system can detect catastrophic attacks, which they define as attacks large enough to cause 10,000 casualties, they have not specified the performance requirements necessary to reliably meet this operational objective.
In the absence of performance requirements, DHS officials said computer modeling and simulation studies support their assertion, but none of these studies were designed to incorporate test results from the Gen-2 system and comprehensively assess the system against the stated operational objective. Additionally, the GAO report says, DHS has not prepared an analysis that combines the modeling and simulation studies with the specific Gen-2 test results to assess the system’s capabilities to detect attacks.
Finally, GAO found limitations and uncertainties in the four key tests of the Gen-2 system’s performance. Because it is not possible to test the BioWatch system directly by releasing live biothreat agents into the air in operational environments, DHS relied on chamber testing and the use of simulated biothreat agents, which limit the applicability of the results.
“These limitations underscore the need for a full accounting of statistical and other uncertainties, without which decision makers lack a full understanding of the Gen-2 system’s capability to detect attacks of defined types and sizes and cannot make informed decisions about the value of proposed upgrades,” the report says.
GAO notes that the actions and decisions DHS made regarding the acquisition and testing of a proposed next generation of BioWatch (Gen-3) partially aligned with best practices GAO previously identified for developmental testing of threat detection systems. For example, best practices indicate that resilience testing, or testing for vulnerabilities, can help uncover problems early. DHS took steps to help build resilience into the Gen-3 testing, but the report says that future testing could be improved by using more rigorous methods to help predict performance in different operational environments.
DHS canceled the Gen-3 acquisition in April 2014, but GAO identified lessonsDHS could learn by applying these best practices to the proposed Gen-2 upgrades.
According to experts and practitioners, the polymerase chain reaction (PCR), which detects genetic signatures of biothreat agents, is the most mature technology to use for an autonomous detection system. DHS is considering autonomous detection as an upgrade to Gen-2, because according to DHS, it may provide benefits such as reduction in casualties or clean-up costs. But the extent of these benefits, GAO says, is uncertain because of several assumptions, such as the speed of response after a detection, whih are largely outside of DHS’s control. As a result, the effectiveness of the response and the number of lives that could be saved is uncertain. Further, an autonomous detection system must address several likely challenges, including minimizing possible false positive readings, meeting sensitivity requirements, and securing information technology networks.
GAO recommends that DHS not pursue upgrades or enhancements for Gen-2 until DHS reliably establishes the system’s current capabilities. GAO also recommends DHS incorporate best practices for testing in conducting any system upgrades.
DHS has generally concurred with GAO’s recommendations.
— Read more in Biosurveillance: DHS Should Not Pursue BioWatch Upgrades or Enhancements Until System Capabilities Are Established, GAO-16-99 (23 November 2015)