A 2025 study found that AI-generated summaries influenced users to make purchase decisions 84 per cent of the time, even though the summaries contained hallucinated or altered facts in up to 60 per cent of cases.
This is not just a technical flaw. It’s a product liability risk.
If your AI changes the sentiment of reviews and invents product features, it nudges users toward purchases. Then you are no longer just building an AI tool—you are shaping consumer behaviour in ways that may be misleading or even legally questionable.
AI Bias here is not just unfair—it’s conversion distortion.
Every dataset used to train AI systems is essentially a snapshot of the real world. But it’s a snapshot that comes with all the imperfections, prejudices, and historical inequalities of that world.
Let’s explore a few real-world cases where it has gone wrong
AI’s bias in selecting resumes for hiring
For instance, imagine you train an AI to recognise job applicants based on resumes.
If the data that the AI is trained on predominantly includes resumes from a certain demographic, the system may learn to favour that demographic only by reproducing and even amplifying existing biases.
This was precisely the problem with an AI used by a Google in the past to screen resumes.
Google’s AI hiring tool was trained on resumes that were submitted to the company over several years, which unfortunately had an overwhelming bias toward male candidates.
As a result, Google’s AI learned to favour male-associated words and traits, like “aggressive” or “competitive,” and ended up filtering out resumes from women.
The AI had simply learned the pattern of who was hired, not the traits that would have led to success for any candidate, regardless of gender. The algorithm did not have the nuance to recognise gender inequality and instead perpetuated it.
This example demonstrates that AI isn’t immune to the biases inherent in human decision-making. In fact, because it operates based on historical data, it often amplifies those biases.
Whether it’s racial, gender-based, or socio-economic bias, AI can end up supporting societal inequalities if not carefully controlled.
Also Read: The rise of invisible businesses: Why the most powerful companies may be built by one person and AI
Risk of over-optimisation
Another major problem in AI pattern recognition is over-optimisation.
This happens when an algorithm is trained too thoroughly on a specific dataset and ends up “memorising” the data rather than learning the underlying pattern.
As a result, the AI performs well on the data it was trained on but poorly when exposed to new, unseen data. This lack of generalisation can be particularly dangerous when AI is deployed in the real world, where data is constantly changing.
Take the example of an AI model trained to predict stock market movements. If it is trained on historical stock data that covers a period of rapid economic growth, the AI might learn to associate certain market behaviours with positive economic conditions.
However, if the economy shifts and a recession begins, the AI might not recognise the new patterns and could make disastrously inaccurate predictions. This is an issue of over-optimisation. The AI has learned patterns specific to one period in time, but cannot extrapolate useful information for a new scenario.
For example, Wealthfront, a robo-advisor that uses AI to manage investment portfolios, had an incident where its algorithm predicted a market correction and advised its clients to sell off stocks in anticipation of a downturn. However, the correction didn’t materialise as expected, and the stocks that were sold off ended up increasing in value.
AI was reacting to market indicators that pointed to a correction, but it failed to account for other factors, such as market sentiment and long-term trends. It was a case of model overfitting, where the algorithm focused too narrowly on historical patterns rather than adapting to evolving market conditions.
AI’s bias in healthcare at IBM
Imagine an AI that has been trained on a specific subset of medical data that doesn’t account for all possible patient conditions.
If that AI is used to make medical diagnoses in the real world, its inability to adapt to new conditions could result in missed diagnoses or, worse, fatal errors.
IBM’s Watson AI for Oncology was designed to help doctors diagnose and treat cancer by analysing medical data. However, it was revealed that the system was providing unsafe and inaccurate treatment recommendations, as it was trained on limited and biased data. In some cases, Watson made recommendations that didn’t align with clinical standards, and it struggled with real-world data complexity.
Lack of contextual learning
While AI systems are excellent at recognising patterns within the scope of the data they are trained on, they lack the ability to understand the context in which these patterns occur.
Humans have the capacity for empathy, ethical reasoning, and a broader understanding of the world, which is something that AI simply cannot replicate yet.
Also Read: The art of AI integration: Growing your business with chatbots and human expertise
AI’s bias in criminal justice
A glaring example of this is AI’s use in criminal justice, particularly in predictive policing. Predictive policing algorithms use historical crime data to forecast where crimes are likely to occur, in an attempt to optimise law enforcement resources.
However, these algorithms are prone to problematic outcomes because they don’t understand the socio-economic or political context behind why crimes are committed in certain areas.
For instance, if an AI system identifies a pattern where certain neighbourhoods have higher crime rates, it might suggest that police patrols be concentrated in those areas. But it may fail to account for systemic issues such as poverty, lack of education, or historical over-policing, which contribute to these higher crime rates in the first place.
Instead of addressing the root causes of crime, the AI ends up reinforcing a cycle of surveillance and criminalisation that disproportionately affects marginalised communities.
For example, the COMPASS (Correctional Offender Management Profiling for Alternative Sanctions) algorithm was used in the US criminal justice system to predict the likelihood of recidivism (repeat offences) and inform parole decisions. Investigations found that the system was biased against Black defendants, giving them higher risk scores compared to white defendants with similar criminal histories.
In essence, the AI has no moral or ethical compass to guide its decisions. It simply follows the data, leading to outcomes that may perpetuate harm rather than reduce it.
The risk of “invisible” bias
One of the more deceptive aspects of AI’s bias is that it’s not always obvious. Often, AI systems are seen as impartial or objective because they aren’t influenced by human emotions, subjective opinions, or personal experiences.
However, the reality is that human biases are embedded in the design and deployment of these systems in ways that may be invisible to users.
Consider facial recognition Software in China. Chinese facial recognition technology has come under fire for disproportionately misidentifying certain ethnic groups. A recent study showed that in regions with minority populations, facial recognition models had higher error rates, leading to false arrests and discrimination.
While these issues might seem specific to the technology or country, they highlight a larger trend: AI systems built without local context or inclusive data can fail spectacularly when deployed at scale.
These biases often remain hidden because, to the untrained eye, the system “seems” to work fine when tested on a homogenous group.
This issue of invisible bias is compounded by the fact that the vast majority of AI models, especially those used in industry and business, operate as “black boxes.”
The decision-making processes of many AI systems are not transparent, meaning the users of these systems may have no idea how or why the AI made a particular decision.
When these decisions have real-world consequences, such as who gets approved for a loan or who gets hired for a job, there’s little accountability or recourse for those affected.
So, how to tackle these AI’s bias? Let’s find out some interesting solutions explored by a few startups here.
Also Read: AI and ethics in digital marketing: Building trust in the tech era
Pymetrics
A startup focusing on AI-driven recruitment tools introduced an ethical AI framework by using neuroscience-based games and algorithms that assess candidates’ cognitive and emotional abilities rather than relying on resumes or biased historical data.
They also partnered with the Fairness, Accountability, and Transparency community to ensure their models are regularly audited for fairness, ensuring that their system doesn’t perpetuate bias.
Impact: This approach provides a more equitable hiring process and has led to a more diverse and inclusive workforce for companies using their platform.
Truera
An AI explainability startup developed an AI model monitoring and auditing tool that not only explains model decisions but also helps identify and mitigate bias in machine learning models. The platform uses visualisations and diagnostics to show if certain demographic groups are disadvantaged by a given model.
Impact: By identifying hidden biases in complex AI models, Truera helps companies correct these issues before they impact real-world outcomes, promoting fairness in automated decisions.
Zest AI
It focuses on making AI-driven lending fairer by using an alternative credit scoring model that analyses a wider variety of factors, including behaviour and transaction history, instead of just traditional credit scores. They also continuously test their models for bias against different groups to ensure equitable access to financial services.
Impact: Zest AI’s methods have led to more accurate credit assessments, increasing loan approvals for underrepresented groups without increasing risk for lenders, thus reducing financial inequality.
H2O.ai
A startup known for its open-source machine learning tools introduced an automated tool that integrates with its platform to detect and mitigate bias. Their solution uses fairness constraints during training to ensure that models do not favour one group over another, regardless of sensitive attributes like race, gender, or age.
Impact: Their tool, “Fairness.ai,” has been adopted by companies looking to build more transparent and accountable models that are less prone to bias, enhancing trust in AI-powered decision-making.
One of the most important things to remember is that while AI has immense potential, it’s not inherently neutral or infallible.
Its power and effectiveness are entirely dependent on the way it is designed, trained, and used.
In nutshell
As AI continues to evolve, its ability to recognise and predict patterns will only improve.
The key lies in ensuring that the humans who design and deploy these systems are aware of these risks and work to make AI a force for fairness, equity, and progress. In the end, the true power of AI will be in its ability to enhance human capabilities, not replace them.
—
Editor’s note: e27 aims to foster thought leadership by publishing views from the community. You can also share your perspective by submitting an article, video, podcast, or infographic.
The views expressed in this article are those of the author and do not necessarily reflect the official policy or position of e27.
Join us on WhatsApp, Instagram, Facebook, X, and LinkedIn to stay connected.
The post The hidden dangers of AI bias: Where it can go wrong appeared first on e27.
