Ethical considerations of AI in retail | Talkdesk (2024)

Retail and Consumer Goods

Artificial intelligence (AI), undoubtedly, is helping build stronger connections for brands and create easier customer experiences. It is increasingly used to personalize product recommendations and power voice/chatbots. However, with all emerging technologies, including the most recent development of generative AI (GenAI), it’s all about the data — “garbage in, garbage out” remains as accurate as ever. Consequently, if AI models are not monitored and kept up to date, incorrect responses, biases, and hallucinations can creep in.

Retail brands that want to take advantage of AI are wrestling with keeping their products accurate and unbiased, as evidenced by the recent Talkdesk Bias & Ethical AI in Retail survey. We asked 1,000 consumers about current personalization experiences and their expectations for retailers to deploy ethical and transparent AI applications. The results of this survey should be an eye-opener for every retailer, whether they use AI or not.

Why consumers have issues with AI in retail.

Biases, stereotyping, and inaccurate personalization are some common threads of frustration among the survey respondents:

• 64% have received an AI-powered product recommendation that did not match their preferences, interests, or previous shopping behaviors. The number is even higher for Hispanic (72%) and Black (69%) respondents.
• 79% refrain from purchasing product recommendations because they are not tailored to their interests but appear to be the top-selling products that retailers are pushing for sales purposes.
• 60% avoid AI recommendations because they believe recommended products are biased or stereotypical. Hispanic (75%) and Asian (67%) consumers are most concerned about the problem.

Shoppers want AI in retail to reflect DEI principles.

Developing, assessing, and deploying responsible AI involves ensuring safety, trustworthiness, and ethics while including people from diverse ethnic, racial, and cultural backgrounds. A perceived neglect of diversity, equity, and inclusion (DEI) in AI has many consumers suspicious and concerned. Less than half (46%) of shoppers believe AI-powered recommendations include diverse shoppers, and only 39% believe retailers consider diverse communities when using or developing AI products. As a result:

• 53% believe that AI-powered facial recognition software will lead to increased instances of racial discrimination. This concern is not unfounded when you consider the example we shared earlier regarding AI image generators perpetuating dangerous race and gender stereotypes.
• 44% worry that AI will make the shopping experience less inclusive because brands aren’t deploying it responsibly, with Hispanic (50%) and Asian (52%) consumers being the most concerned.
• 45% are fearful their online shopping experience will no longer be personalized because retailers are not practicing responsible AI. For example, they worry that they’ll no longer receive personalized products inclusive of all demographics or that AI will recommend irrelevant products.
• It is important for brands to note that 31% would stop shopping with a brand if they felt their AI products did not represent or include diverse communities, with 34% of Gen Z and 35% of Millennials willing to go elsewhere
• 69% of men (compared to 53% of women) avoid AI recommendations because of bias and stereotypes.
• 71% have never purchased a recommended product because it makes them feel like a brand is tracking them — i.e., listening to their in-person conversations, tracking their browsing history, and other concerns.

GUIDE

ChatGPT prompts for retail customer service leaders

Download our deck of ChatGPT prompts, specifically designed for retail CX leaders.

Customers expect transparency and accountability.

Regarding AI and customer data, customers are abundantly clear about their expectations. Brands must pay attention to the fact that customers expect transparency at every point of the journey. The report found that:

• 90% of shoppers believe retailers should be required to openly disclose how they use customer data in applying AI usage
• 87% believe customers should have the right to access and review the data retailers have collected about them.
• 80% of shoppers want retailers to seek explicit consent to use their data for AI.
• 46% review a retailer’s customer data privacy statement before shopping.

Sadly, only 28% of shoppers are confident retailers have the proper security measures to protect data used by AI technology. As AI is quickly becoming a pillar of modern customer experience, brands must ensure they launch safe and ethical AI solutions and be transparent in how their AI models leverage and use customer data.

How can retailers promote the ethical use of AI?

Consumers believe the responsibility for improving AI experiences falls on the brands deploying the technology. They highlighted three main focus areas for brands:

1. 88% agree that brands must develop internal policies to guide the ethical use of AI.
2. 86% state that brands are responsible for monitoring, controlling, and mitigating AI bias.
3. 83% express that brands should develop proactive communications to customers on their use of AI and any potential biases in their services.

Data governance teams are essential to ensure the data used to train AI systems is consistent, accurate, trustworthy, and up to date. Do you know what your brand is doing?

Ethical considerations of AI in retail: a top priority for 2024.

Addressing the ethical considerations of AI in retail will be an ongoing activity. Some technologies can monitor and refine AI models to identify and remove discriminatory patterns. This safeguard helps detect and mitigate instances where AI may generate inaccurate or misleading information. However, this checks and balances system shouldn’t entirely rely on technology. Humans should also monitor AI systems to track for biases or inaccuracies, as people can do something even the most powerful AI technology can’t: review information with context and empathy.

Guide

Creating a culture of compassion in retail contact centers

Imagine customer service where everyone leads with humanity and the burden of kindness doesn’t rest solely on the shoulders of customer service agents. What if leaders, customers, and agents focused on extending compassion in each interaction?

Building loyalty with customers is a top priority for almost every retailer. Acquiring a new customer can be five times higher than retaining an existing one. Plus, just one negative review can cost more than $15,000 annually, depending on the business. Even the biggest brands in the world can be severely damaged by neglecting factors that impact customer satisfaction, including the ethical considerations of deploying AI technologies. 80% of shoppers would have more trust in product recommendations if retailers were transparent about their AI use.

Creating anticipatory and hyper-personalized experiences is quickly becoming table stakes for brands. AI is essential in doing so, but this survey reveals that it can’t just be AI in a black box. AI requires supervision to accurately reflect the reality of a brand’s customer base and ensure personalized experiences for individual customers. In 2024 and beyond, ensuring responsible, unbiased AI use must be a top priority for retail brands.

Methodology:

The Talkdesk Bias & Ethical AI in Retail Survey was fielded in December 2023 to a sample of 1,000 consumers based in the U.S. All respondents regularly shop and purchase items online weekly or monthly and have some basic knowledge of AI.

Shannon Flanagan

Shannon Flanagan has been retailing since college. She’s worked in-stores and as a merchant, consultant, sales leader, technologist and strategist. She’s been an executive with Gap Inc., Lands’ End, and Macy’s, leading strategic initiatives, with expertise in omnichannel transformation. She’s partnered with hyper-growth companies to Fortune 500 companies during her time with Accenture, Infor, Slalom and now Talkdesk. In her current role, as the VP|GM of Retail & Consumer Goods, she helps create happy path experiences for customers and agents alike. In addition, she is a ReThink Top Influencer, DE&I board member, Mental Health advocate and speaker and founder of Retail Women in Tech. Retail is never boring and she loves it!

Other blog posts.