Kathy Baxter

The journey of the AI assistant is already well underway. We’re seeing the evolution to agent take shape in three phases: assistant, concierge, and agent. This evolution is exciting and promises to make our lives much easier — eliminating mundane tasks and helping us become more productive. But anyone who has ever had an assistant knows that the relationship only works if there is trust. The authors explore what can go wrong with autonomous agents, and how companies can deploy them responsibly.

Generative artificial intelligence (AI) has become widely popular, but its adoption by businesses comes with a degree of ethical risk. Organizations must prioritize the responsible use of generative AI by ensuring it is accurate, safe, honest, empowering, and sustainable. Organizations need to be mindful of the ethical implications and take necessary steps to reduce risks. Specifically, they need to: use zero or first party data, keep data fresh and well labeled, ensure there’s a human in the… Show more

Generative artificial intelligence (AI) has become widely popular, but its adoption by businesses comes with a degree of ethical risk. Organizations must prioritize the responsible use of generative AI by ensuring it is accurate, safe, honest, empowering, and sustainable. Organizations need to be mindful of the ethical implications and take necessary steps to reduce risks. Specifically, they need to: use zero or first party data, keep data fresh and well labeled, ensure there’s a human in the loop, test and re-test, and get feedback. Show less

Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems. Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments? To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability. We show how adversarial attacks can be reframed for this goal, via a framework for… Show more

Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems. Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments? To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability. We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests. We argue that reliability testing — with an emphasis on interdisciplinary collaboration — will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards. Show less

The study of fairness in machine learning applications has seen significant academic inquiry, research and publication in recent years. Concurrently, technology companies have begun to instantiate nascent program in AI ethics and product ethics more broadly. As a result of these efforts, AI ethics practitioners have piloted new processes to evaluate and ensure fairness in their machine learning applications. In this session, six industry practitioners, hailing from LinkedIn, Yoti, Microsoft… Show more

The study of fairness in machine learning applications has seen significant academic inquiry, research and publication in recent years. Concurrently, technology companies have begun to instantiate nascent program in AI ethics and product ethics more broadly. As a result of these efforts, AI ethics practitioners have piloted new processes to evaluate and ensure fairness in their machine learning applications. In this session, six industry practitioners, hailing from LinkedIn, Yoti, Microsoft, Pymetrics, Facebook, and Salesforce share insights from the work they have undertaken in the area of fairness, what has worked and what has not, lessons learned and best practices instituted as a result. Show less

When it comes to conducting online user studies, bots have become that one jerk that ruins it for everyone! The Internet allows researchers to conduct user studies online 24-hours a day with nearly anyone in the world but bots are increasingly “participating” in these studies and corrupting the data. Here are some tips for how researchers can distinguish human from bot and how to stop the bad guys from contaminating everyone’s data.

Ten years is an eternity in the tech world. But the speed of change makes the classic “decade of reflection” even more valuable for assessing which changes really count and why. We had a chance to reflect systematically on the last decade of user experience research for the second edition of our book “Understanding Your Users,” which was first published in 2005. Since then, user research has become more widespread and more sophisticated. It has also responded to challenges of faster development… Show more

Ten years is an eternity in the tech world. But the speed of change makes the classic “decade of reflection” even more valuable for assessing which changes really count and why. We had a chance to reflect systematically on the last decade of user experience research for the second edition of our book “Understanding Your Users,” which was first published in 2005. Since then, user research has become more widespread and more sophisticated. It has also responded to challenges of faster development cycles while simultaneously contributing more, not just to product development, but to the bigger picture of strategic innovation.

So among the noise of trendy terms and fashionable phrases we found five key trends that really do matter. Show less

This new and completely updated edition is a comprehensive, easy-to-read, "how-to"​ guide on user research methods. You'll learn about many distinct user research methods and also pre- and post-method considerations such as recruiting, facilitating activities or moderating, negotiating with product developments teams/customers, and getting your results incorporated into the product. For each method, you'll understand how to prepare for and conduct the activity, as well as analyze and present… Show more

This new and completely updated edition is a comprehensive, easy-to-read, "how-to"​ guide on user research methods. You'll learn about many distinct user research methods and also pre- and post-method considerations such as recruiting, facilitating activities or moderating, negotiating with product developments teams/customers, and getting your results incorporated into the product. For each method, you'll understand how to prepare for and conduct the activity, as well as analyze and present the data - all in a practical and hands-on way.

Each method presented provides different information about the users and their requirements (e.g., functional requirements, information architecture). The techniques can be used together to form a complete picture of the users'​ needs or they can be used separately throughout the product development lifecycle to address specific product questions. These techniques have helped product teams understand the value of user experience research by providing insight into how users behave and what they need to be successful. You will find brand new case studies from leaders in industry and academia that demonstrate each method in action.

This book has something to offer whether you are new to user experience or a seasoned UX professional. After reading this book, you'll be able to choose the right user research method for your research question and conduct a user research study. Then, you will be able to apply your findings to your own products. Show less

Experience Sampling Methodology (ESM) is a type of longitudinal diary study that allows one to understand a person's experience in the moment. It combines the qualitative richness of longitudinal diary studies, artifacts of a field study, and quantitative data of a large-scale survey or app tracker. Using a free, open-source mobile app called "PACO (Personal Analytics COmpanion)," we can conduct ESM studies with participants anywhere in the world. These studies can be conducted after… Show more

Experience Sampling Methodology (ESM) is a type of longitudinal diary study that allows one to understand a person's experience in the moment. It combines the qualitative richness of longitudinal diary studies, artifacts of a field study, and quantitative data of a large-scale survey or app tracker. Using a free, open-source mobile app called "PACO (Personal Analytics COmpanion)," we can conduct ESM studies with participants anywhere in the world. These studies can be conducted after qualitative studies (e.g., ethnography, interviews) to ascertain how broadly your observations apply to your user population or they can be done in advance to identify insights you want to study in-person, in-depth. By combining these methodologies, you create a, deeper, more holistic understanding of your users. This workshop will give attendees the skills to design, conduct, and analyze data from ESM studies at any scale. Show less

In the last few years we’ve all heard amazing stories of how “big data” can make uncanny predictions about users (e.g., Target knowing when someone is pregnant based on their purchases). However, there are just as many stories of when analytics gets it wrong (e.g., Google Flu Trends overestimating flu cases in 2013). It shouldn’t be at all surprising when predictions based strictly on click analysis (sometimes with demographic data thrown in) gets it wrong because analytics tells us only the… Show more

In the last few years we’ve all heard amazing stories of how “big data” can make uncanny predictions about users (e.g., Target knowing when someone is pregnant based on their purchases). However, there are just as many stories of when analytics gets it wrong (e.g., Google Flu Trends overestimating flu cases in 2013). It shouldn’t be at all surprising when predictions based strictly on click analysis (sometimes with demographic data thrown in) gets it wrong because analytics tells us only the WHAT, not the WHY. In the case of Google Flu Trends, the massive media coverage influenced people to search on the keywords that previously indicated one was coming down with the flu but now simply meant people were curious about it. Our logs were missing the WHY behind the keywords.

At Google, we conduct an annual Experience Sampling Methodology (ESM) study with a large sample of our users recording their needs, context, and experience to capture the WHY we cannot see in our query stream. Over a five day period, participants tell us about their experiences throughout their day and submit photos to explain things words alone cannot capture. Doing this over a several month period every year allows us to monitor changes in user’s subjective and objective behavior for a clearer picture than analytics alone. Show less

Many people know that Google's motto is "Focus on the user and all else will follow" but what does that really mean. People in different roles at Google apply this to their jobs in different ways. This talk will discuss how the User Experience Research team specifically applies this in their daily work. Kathy will discuss the research methods employed and examples of how they have been applied.

There is much interest in the topic of Big Data. However, many researchers are unclear how to combine rich or “thick,” ethnographic data with large-scale quantitative data. This workshop offers an approach, which does not require a computer science degree: ESM via PACO. Experiential Sampling Methodology (ESM) is a type of longitudinal diary study that allows one to understand a person’s experience in the moment. Using a free, open-source mobile app called “PACO (Personal Analytics Companion),”… Show more

There is much interest in the topic of Big Data. However, many researchers are unclear how to combine rich or “thick,” ethnographic data with large-scale quantitative data. This workshop offers an approach, which does not require a computer science degree: ESM via PACO. Experiential Sampling Methodology (ESM) is a type of longitudinal diary study that allows one to understand a person’s experience in the moment. Using a free, open-source mobile app called “PACO (Personal Analytics Companion),” we can conduct large-scale ESM studies with users anywhere in the world. These studies can be conducted after ethnographic studies to ascertain how broadly your observations apply to your user population or they can be done in advance to identify insights you want to study in-person, in-depth. By combining these methodologies, you create a more holistic understanding of your users.

In this three-hour workshop, we will introduce attendees to ESM, discuss ways we have used this methodology at Google, help attendees create their own ESM study, and discuss data analysis. The goal is for every attendee to leave the session equipped with the knowledge to design and create their own ESM study, analyze the data, and make actionable recommendations. Attendance will be capped at 25 attendees to ensure 1:1 attention and good group discussions. Show less

The Experience Sampling Method (ESM) enables researchers to capture information about participants' experiences in the moment. Adding an end-of-day retrospective survey also allows participants to elaborate on those experiences. Although the use of photos in retrospective interviews and surveys for memory elicitation is well known, little research has investigated the use of photos in ESM studies. As smartphone adoption increases facilitating ESM studies and making photo sharing easier… Show more

The Experience Sampling Method (ESM) enables researchers to capture information about participants' experiences in the moment. Adding an end-of-day retrospective survey also allows participants to elaborate on those experiences. Although the use of photos in retrospective interviews and surveys for memory elicitation is well known, little research has investigated the use of photos in ESM studies. As smartphone adoption increases facilitating ESM studies and making photo sharing easier, researchers need to continuously evaluate the method and investigate the role of photos in such studies. We conducted a large-scale ESM and retrospective survey study via Android smartphones with more than 1,000 US participants, and analyzed participants' photo submissions, including how photo use correlated with participants' data quality and what, if any, value photos added for researchers. Our study sheds light on the role of photos in ESM and retrospective studies that researchers can reference when constructing future study designs. Show less

Search malware redirects nearly 100% of infected users' clicks on web search results to unintended websites. Most published research details how web-based malware works and technological interventions to stop it before users ever see it; however, the constant evolution of obfuscation techniques makes it difficult to prevent infection altogether. User interventions in the form of toolbars, dialogs, and user education have seen limited success. Previous research has focused on a prototypical type… Show more

Search malware redirects nearly 100% of infected users' clicks on web search results to unintended websites. Most published research details how web-based malware works and technological interventions to stop it before users ever see it; however, the constant evolution of obfuscation techniques makes it difficult to prevent infection altogether. User interventions in the form of toolbars, dialogs, and user education have seen limited success. Previous research has focused on a prototypical type of malware; a sophisticated program that conceals itself (e.g., surreptitious download onto a host computer) or tries to fool the user by mimicking known, trusted websites (e.g., phishing attacks). The goal of our research is to understand users' experience, understanding of and response to search malware. The present research shows that even when confronted with blatantly unusual search behavior, people are unlikely to attribute blame to malware or to engage in behavior that may remedy the situation. Show less

User Experience (UX) leaders and managers are required to continually adapt to changes in: organizational strategies and re-structuring, resources, technology, economic pressures, and other factors. Simultaneously, more companies are realizing that they need UX expertise to ensure that they are competitive in today's marketplace. This panel is comprised of UX leaders who have created strategies and tactics to succeed both in spite of and with the aid of the past and current trends. The panel… Show more

User Experience (UX) leaders and managers are required to continually adapt to changes in: organizational strategies and re-structuring, resources, technology, economic pressures, and other factors. Simultaneously, more companies are realizing that they need UX expertise to ensure that they are competitive in today's marketplace. This panel is comprised of UX leaders who have created strategies and tactics to succeed both in spite of and with the aid of the past and current trends. The panel will focus on the current trends, what strategies and tactics have and have not worked in addressing these trends, and also discuss which future trends they think will impact UX departments, companies, and the field, and how they are preparing for these future trends.

The panel will be of interest to managers, practitioners and those who work closely with these teams, including developers, project managers, market researchers, test managers, and executives. Show less

Increasingly large and complex data sets, fewer resources, and short timelines pose unique challenges to researchers. To over come obstacles of scale, complexity, and velocity, we propose two techniques” data triangulation and process scaling. We will explain how we've used these techniques, discuss lessons learned, & through interactive exercises, participants will learn how to apply these techniques to their own work.

Ethnographic researchers are often more at home in the field than in organizational settings
and designers in the open studio. We often see competing internal goals trump insights from
effective research-based design proposals, presentations and reports. The Strategic
Dialogue workshop prepares participants with tools for organizing collaborative stakeholder
workshops that help you establish joint ownership of the meaning of research.

Agile development is being adopted by many leading software companies, such as those represented by this panel. Though many instructional resources exist to guide companies through a change to Agile Development, there are few resources available on the subject of Agile development and User Centered Design (UCD). As a result, user experience practitioners have had to develop their own tactics and strategies for maintaining sound UCD practices within their organizations when moving to… Show more

Agile development is being adopted by many leading software companies, such as those represented by this panel. Though many instructional resources exist to guide companies through a change to Agile Development, there are few resources available on the subject of Agile development and User Centered Design (UCD). As a result, user experience practitioners have had to develop their own tactics and strategies for maintaining sound UCD practices within their organizations when moving to Agile.

This panel consists of six practitioners who actively work with development teams using Agile. Panel members will share the challenges and successes they face while championing UCD within their respective development organizations. Panelists will focus on adaptations to research methodology and strategy that make UCD possible to attain within Agile cultures. Show less

Today many companies are employing a user-centered design (UCD) process, but for most companies, usability begins and ends with the usability test. Although usability testing is a critical part of an effective user-centered life cycle, it is only one component of the UCD process. This book is focused on the requirements gathering stage, which often receives less attention than usability testing, but is equally as important. Understanding user requirements is critical to the development of a… Show more

Today many companies are employing a user-centered design (UCD) process, but for most companies, usability begins and ends with the usability test. Although usability testing is a critical part of an effective user-centered life cycle, it is only one component of the UCD process. This book is focused on the requirements gathering stage, which often receives less attention than usability testing, but is equally as important. Understanding user requirements is critical to the development of a successful product.

"Understanding Your Users" is an easy to read, easy to implement, how-to guide on usability in the real world. It focuses on the user requirements gathering stage of product development and it provides a variety of techniques, many of which may be new to usability professionals. For each technique, readers will learn how to prepare for and conduct the activity, as well as analyze and present the data -all in a practical and hands-on way. In addition, each method presented provides different information about the user and their requirements (e.g., functional requirements, information architecture, task flows). The techniques can be used together to form a complete picture of the users' requirements or they can be used separately to address specific product questions. These techniques have helped product teams understand the value of user requirements gathering by providing insight into how users work and what they need to be successful at their tasks. Case studies from industry-leading companies demonstrate each method in action. In addition, readers are provided with the foundation to conduct any usability activity (e.g., getting buy-in from management, legal and ethical considerations, setting up your facilities, recruiting, moderating activities) and to ensure the incorporation of the results into their products. Show less

This tutorial will provide attendees with the skills to conduct three effective and low cost user requirements techniques (Wants and Needs Analysis, Group Card Sort, and Group Task Analysis). Each method provides different information requirements information (functional requirements; information architecture and task flows, respectively).

Participants will learn how to:
· Conduct each method, collect the data, analyze, and present the results
· Recruit the “right” participants… Show more

This tutorial will provide attendees with the skills to conduct three effective and low cost user requirements techniques (Wants and Needs Analysis, Group Card Sort, and Group Task Analysis). Each method provides different information requirements information (functional requirements; information architecture and task flows, respectively).

Participants will learn how to:
· Conduct each method, collect the data, analyze, and present the results
· Recruit the “right” participants for these activities
· Be aware of cultural differences that can impact user requirements collection
· Be an effective group moderator Show less

This tutorial will provide participants with the skills to conduct a Wants and Needs Analysis, Group Task Analysis and Card Sort. In addition, participants will learn how to interpret the data from these techniques and present the results. The tutorial will consist of hands-on exercises, group discussion, and lecture. At the end of this tutorial participants will be able to go back to their workplace armed with the skills to con- duct three user requirements activities. In addition to practical… Show more

This tutorial will provide participants with the skills to conduct a Wants and Needs Analysis, Group Task Analysis and Card Sort. In addition, participants will learn how to interpret the data from these techniques and present the results. The tutorial will consist of hands-on exercises, group discussion, and lecture. At the end of this tutorial participants will be able to go back to their workplace armed with the skills to con- duct three user requirements activities. In addition to practical tips and tricks, they will learn to:
• Facilitate a Wants and Needs Analysis, collect the data, and ana- lyze the results
• Facilitate a Group Task Analysis, collect the data, and analyze the results
• Facilitate a Group Card Sort, collect the data, and analyze the results
• Turn the results of these methods into design solutions that customers will be able to understand and implement
• Recruit the participants for these activities. Show less

Usability and design professionals know that user requirements gathering is critical to the development of quality products. However, development teams often skip formal user requirements gathering, fearing it will take too long. Certain techniques can be employed by design professionals that are easily learned and implemented with little overhead. They can help designers by providing insight into how users work and what they need to succeed at their task. Through lecture, case studies, and an… Show more

Usability and design professionals know that user requirements gathering is critical to the development of quality products. However, development teams often skip formal user requirements gathering, fearing it will take too long. Certain techniques can be employed by design professionals that are easily learned and implemented with little overhead. They can help designers by providing insight into how users work and what they need to succeed at their task. Through lecture, case studies, and an emphasis on hands-on activities, participants will learn how to conduct three user requirements gathering techniques and incorporate the results into design solutions. Show less

Two commercially available, computerized vocabulary games were assessed for their
teaching benefits to users. The games were independently tested against three criteria, not
against each other. The Matching Game used a rehearsal teaching strategy and required
participants to match words to their meanings. The Analogy Game used a semantic strategy
and required participants to determine the relationship between the meanings of three
words. Game scores were used as the… Show more

Two commercially available, computerized vocabulary games were assessed for their
teaching benefits to users. The games were independently tested against three criteria, not
against each other. The Matching Game used a rehearsal teaching strategy and required
participants to match words to their meanings. The Analogy Game used a semantic strategy
and required participants to determine the relationship between the meanings of three
words. Game scores were used as the dependent measure because they reflected the ... Show less

This experiment examined changes in performance using an ambiguous set of system displays and controls and the mental models formed. The independent variables were control type (three levels) and number of trials (17 levels). The three control types were Legitimate Control (affected the system output level as well as the displays for other controls), Indirect Distractor (affected the displays for other controls but not the system output level), and Blatant Distractor (affected nothing when… Show more

This experiment examined changes in performance using an ambiguous set of system displays and controls and the mental models formed. The independent variables were control type (three levels) and number of trials (17 levels). The three control types were Legitimate Control (affected the system output level as well as the displays for other controls), Indirect Distractor (affected the displays for other controls but not the system output level), and Blatant Distractor (affected nothing when activated). The dependent variables were the amount of time it took to learn the difference between control types, the number of mouse clicks on each control, the system output level maintained, the importance ranking given to each control, and the participant's drawing of the relations between the various controls and the system output. The participants were told what output level to maintain but they were not told how to accomplish that task nor were they given clear feedback. Based on the rankings and drawings, it was concluded that the participants developed an accurate mental model of the system. They learned the differences between the three control types and correctly identified which controls were most important for manipulating system output. However, performance data (output level maintained and number of mouse clicks per control type) suggest that the participants did not learn the differences between the three control types. The participants continued to respond to distractor controls and were not able to maintain the target system output level. Show less