Overview

Presently, there is a lack of understanding concerning how task features such as complexity, time-criticality, and unfamiliarity may modulate the nature of expert decision-making. Although substantial research has focused on the ability of experts to make decisions intuitively based on past experience, there is a gap concerning how experts may be able to deal with time limitations in novel tasks where previous experience can't help them.

• Additionally, much of the work in this area has focused on simple laboratory tasks that are far removed from the real-world environment or has relied on observational research of real-world tasks that lack experimental control. Therefore, the present research aims to use methods that approximate the real world more closely without compromising on experimental control.

• Specifically, the research is based on two questions:

  • First, what role does expert-level knowledge play in time-critical, unfamiliar, and complex environments?

  • Second, does having superior situation awareness lead to better decision-making performance in time-critical, complex, and unfamiliar environments?

• We expect that expert-level knowledge will lead to better decision outcomes across all types of decision-making tasks. Importantly, the benefits of expert-level knowledge are expected to be most prominently demonstrated in complex, time-critical, and novel tasks.

• If our hypotheses are supported, then our research can help design better training programs modeled after the knowledge of experts rather than ones focused on rehearsal-based training alone.

• Moreover, we expect expert-level knowledge to lead to a higher level of situation awareness, which could buy time to make decisions in novel and time-critical situations. If this is indeed the case, then it could inform design decisions concerning automation. If situation awareness is the key to making better decisions, then automation aids intended to support decision making should be aimed at increasing operator situation awareness.

Work

Literature Review

• Read 100+ book chapters and journal articles concerning expertise and decision-making.

• Identified functional knowledge of underlying system relationships as the key underlying difference between experts and novices that may affect their decision-making outcomes.

Experimental Design and Method

Constraint

• We do not have access to expert air force pilots to conduct the research.

Problem

• How can you attempt to characterize expert decision-making without experts?

Solution

• As mentioned above, I identified knowledge as the key difference between experts and novices. Experts have domain-specific high-level knowledge structures that enable them to understand underlying system relationships whereas novices have superficial knowledge based on surface-level relationships.

• Based on my reading, I devised a manipulation that will allow me to use naive subjects and simulate the differences between experts and novices by giving half of them expert-level knowledge and the other half novice-level knowledge.

Experimental design

• We are using a 2 (task complexity) x 2 (time-criticality) x 2 (familiarity) x 2 (task-relevant knowledge) mixed experimental design. Task complexity, time-criticality, and familiarity will be within-subject manipulations, and the type of knowledge (expert level or surface level) will be the between-subject manipulation.

• The decision task refers to inflight failures that the subjects will have to diagnose using multiple cues.

• We coded flight scenarios to simulate real-world flight. Using a flight simulation will allow us to do a human-in-the-loop simulation within a laboratory environment thus affording us the benefits of approximating real-world dynamics without compromising on the control a laboratory offers.

Analysis

• We plan to conduct a mixed Analysis of Variance (ANOVA) to investigate the interactions between knowledge and task factors. Using an ANOVA will also help us control for Type I error rates as compared to using multiple t-tests.

Outcomes and Reflection

• Even though the research is still a work in progress, I have already collaborated with my advisor (Dr. Pamela Tsang), who is an industry expert in expertise literature. Additionally, I have worked with a thesis committee member (Dr. Michael Vidulich) who is a well-known figure in the US Air Force when it comes to Human Factors research.

• As the research develops further, I will also be collaborating with an expert on eye-tracking methods as we plan to include eye-tracking to study situation awareness in more detail.

• Developing the present study has taught me a lot:

  • Understanding constraints of doing research and learning various workarounds to circumvent them without compromising on the validity

  • Learning industry best practices to deal with human variability

  • Scoping a research study to work with the time and resource limitations