BSP

Improving the communication of uncertainty in climate science and intelligence analysis

by Emily H. Ho, David V. Budescu, Mandeep K. Dhami, & David R. Mandel
February 16, 2017

Supplemental Material

Author Note

Portions of this research were funded by Grants 1049208 and 1125879 from the U.S. National Science Foundation, the Defence Research and Development Canada Joint Intelligence Collection and Analytic Capability Project, and Her Majesty’s Government. This research contributes to the NATO System Analysis and Studies Panel Technical Team on Assessment and Communication of Risk and Uncertainty to Support Decision-Making (SAS-114). We thank Thomas S. Wallsten, professor emeritus at the University of Maryland, College Park, for useful feedback on the manuscript, and Ronald Wulf, formerly of the Canadian Forces School of Military Intelligence, for facilitating the data collection from the Canadian sample in Study 2. Copyright of this article belongs to Her Majesty the Queen in Right of Canada, as represented by Defence Research and Development Canada.

Author Affiliation

Ho, and Budescu, Department of Psychology, Fordham University; Dhami, Department of Psychology, Middlesex University; and Mandel, Socio-Cognitive Systems Section, Toronto Research Centre, Defence Research and Development Canada. Corresponding author’s e-mail: [email protected]

References

1. Morgan, M. G., Henrion, M., & Small, M. (1990). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge, United Kingdom: Cambridge University Press.

2. Budescu, D. V., Por, H.-H., Broomell, S. B., & Smithson, M. (2014). The interpretation of IPCC probabilistic statements around the world. Nature Climate Change, 4, 508–512. doi:10.1038/nclimate2194

3. Brun, W., & Teigen, K. H. (1988). Verbal probabilities: Ambiguous, context dependent, or both? Organizational Behavior and Human Decision Processes, 41, 390–404

4. Wallsten, T. S., Budescu, D. V., Zwick, R., & Kemp, S. M. (1993). Preferences and reasons for communicating probabilistic information in numeric or verbal terms. Bulletin of the Psychonomic Society, 31, 135–138. doi:10.3758/BF03334162

5. Fox, C. R., & Ülkümen, G. (2011). Distinguishing two dimensions of uncertainty. In W. Brun, G. Keren, G. Kirkeboen, & H. Montgomery (Eds.), Perspectives on thinking, judging and decision making (pp. 21–35). Oslo, Norway: Univeristetforlaget.

6. Dhami, M. K. (2008). On measuring quantitative interpretations of reasonable doubt. Journal of Experimental Psychology: Applied, 14, 353–363. doi:10.1037/a0013344

7. Dhami, M. K., Lundrigan, S., & Mueller-Johnson, K. (2015). Instructions on reasonable doubt: Defining the standard of proof and the juror’s task. Psychology, Public Policy, and Law, 21, 169–178.

8. Wallsten, T. S., & Budescu, D. V. (1995). A review of human linguistic probability processing: General principles and empirical evidence. The Knowledge Engineering Review, 10, 43–62. doi:10.1017/S0269888900007256

9. Budescu, D. V., & Wallsten, T. S. (1985). Consistency in interpretation of probabilistic phrases. Organizational Behavior and Human Decision Processes, 36, 391–405. doi:10.1016/0749-5978(85)90007-X

10. Karelitz, T. M., & Budescu, D. V. (2004). You say “probable” and I say “likely”: Improving interpersonal communication with verbal probability phrases. Journal of Experimental Psychology: Applied, 10, 25–41. doi:10.1037/1076-898X.10.1.25

11. Dhami, M., & Wallsten, T. S. (2005). Interpersonal comparison of subjective probabilities: Toward translating linguistic probabilities. Memory & Cognition, 33, 1057–1068. doi:10.3758/BF03193213

12. Mandel, D. R. (2007). Toward a concept of risk for effective military decision making (Technical Report 2007-124). Retrieved from Defence Research and Development Canada website: http://cradpdf.drdc-rddc.gc.ca/PDFS/unc68/p529014.pdf

13. Berry, D., Raynor, D. K., Knapp, P., & Bersellini, E. (2003). Patients’ understanding of risk associated with medication use: Impact of European Commission guidelines and other risk scales. Drug Safety, 26, 1–11. doi:10.2165/00002018-200326010-00001

14. Barnes, A. (2015). Making intelligence analysis more intelligent: Using numeric probabilities. Intelligence and National Security. Advance online publication. doi:10.1080/02684527.2014.994955

15. Mosteller, F., & Youtz, C. (1990). Quantifying probabilistic expressions. Statistical Science, 5, 2–16.

16. Mandel, D. R. (2015). Accuracy of intelligence forecasts from the intelligence consumer’s perspective. Policy Insights from the Behavioral and Brain Sciences, 2, 111–120.

17. Intergovernmental Panel on Climate Change. (n.d.). Organization. Retrieved December 13, 2015, from http://www.ipcc.ch/organization/organization.shtml

18. Swart, R., Bernstein, L., Ha-Duong, M., & Petersen, A. (2009). Agreeing to disagree: Uncertainty management in assessing climate change, impacts and responses by the IPCC. Climatic Change, 92, 1–29.

19. Budescu, D. V., Por, H. H., & Broomell, S. B. (2012). Effective communication of uncertainty in the IPCC reports. Climatic Change, 113, 181–200.

20. Ha-Duong, M., Swart, R., Bernstein, L., & Petersen, R. (2007). Uncertainty management in the IPCC: Agreeing to disagree. Global Environmental Change, 17, 8–11. doi:10.1007/s10584-008-9444-7

21. Wallsten, T. S., Budescu, D. V., Rapoport, A., Zwick, R., & Forsyth, B. (1986). Measuring the vague meaning of probability terms. Journal of Experimental Psychology: General, 115, 348–365. doi:10.1037/0096-3445.115.4.348

22. Dhami, M. K. (2008). On measuring quantitative interpretations of reasonable doubt. Journal of Experimental Psychology: Applied, 14, 353–363.

23. Lundrigan, S., Dhami, M. K., & Mueller-Johnson, K. (2013). Predicting verdicts using pre-trial attitudes and standard of proof. Legal and Criminological Psychology. Advance online publication. doi:10.1111/lcrp.12043

24. Kent, S. (1964). Words of estimative probability. Retrieved from Central Intelligence Agency website: https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/sherman-kent-and-the-board-of-national-estimates-collected-essays/6words.html

25. Mandel, D. R., & Barnes, A. (2014). Accuracy of forecasts in strategic intelligence. Proceedings of the National Academy of Sciences, USA, 111, 10984–10989. doi:10.1073/pnas.1406138111

26. Derbentseva, N., McLellan, L., & Mandel, D. R. (2010). Issues in intelligence production: Summary of interviews with Canadian managers of intelligence analysts (Technical Report 2010-144). Retrieved from Defence Research and Development Canada website: http://pubs.drdc-rddc.gc.ca/PDFS/unc111/p534903_A1b.pdf

27. Adams, B. A., Thomson, M., Derbentseva, N., & Mandel, D. R. (2012). Capability challenges in the human domain for intelligence analysis: Report on community-wide discussions with Canadian intelligence professionals (Contract Report 2011-182). Retrieved from the Defence Research and Development Canada website: http://pubs.drdc-rddc.gc.ca/PDFS/unc118/p536570_A1b.pdf

28. Weiss, C. (2007). Communicating uncertainty in intelligence and other professions. International Journal of Intelligence and CounterIntelligence, 21, 57–85. doi:10.1080/08850600701649312

29. Office of the Director of National Intelligence. (n.d.). National Intelligence Council. Retrieved December 14, 2015, from http://www.dni.gov/index.php/about/organization/national-intelligence-council-who-we-are

30. National Intelligence Council. (2007). Prospects for Iraq’s stability: A challenging road ahead. Retrieved from http://fas.org/irp/dni/iraq020207.pdf

31. Gov.UK. (n.d.). Defence Intelligence. Retrieved December 14, 2015, from https://www.gov.uk/government/groups/defence-intelligence

32. Dhami, M. K. (2013). Understanding and communicating uncertainty in intelligence analysis. Unclassified report prepared for Her Majesty’s Government, United Kingdom. (Available from the author).

33. Defence Intelligence. (n.d). Quick wins for busy analysts. London, United Kingdom: United Kingdom Ministry of Defence.

34. Budescu, D. V., Karelitz, T. M., & Wallsten, T. S. (2003). Predicting the directionality of probability phrases from their membership functions. Journal of Behavioral Decision Making, 16, 159–180.

35. Harris, A. J. L., & Corner, A. (2011). Communicating environmental risks: Clarifying the severity effect in interpretations of verbal probability expressions. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 1571–1578.

36. Fox, C. R., & Irwin, J. R. (1998). The role of context in the communication of uncertain beliefs. Basic and Applied Social Psychology, 20, 57–70.

37. Piercey, M. D. (2009). Motivated reasoning and verbal vs. numerical probability assessment: Evidence from an accounting context. Organizational Behavior and Human Decision Processes, 108, 330–341. doi:10.1016/j.obhdp.2008.05.004

38. Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498.

39. Mandel, D. R. (2015). Communicating numeric quantities in context: Implications for decision science and rationality claims. Frontiers in Psychology, 6, Article 537. doi:10.3389/fpsyg.2015.00537

40. Mandel, D. R., Barnes, A., & Richards, K. (2014). A quantitative assessment of the quality of strategic intelligence forecasts (Technical Report 2013-036). Retrieved from the Defence Research and Development Canada website: http://cradpdf.drdc-rddc.gc.ca/PDFS/unc142/p538628_A1b.pdf

41. Dhami, M. K., Mandel, D. R., Mellers, B., & Tetlock, P. (2015). Improving intelligence analysis with decision science. Perspectives on Psychological Science, 106, 753–757.

42. Friedman, J. A., & Zeckhauser, R. (2012). Assessing uncertainty in intelligence. Intelligence and National Security, 27, 824–847.