What Is Ethical Debt?
The concept of ethical debt is starting to become a rigorous conversation among enterprise architects. The simple definition is an accumulation of risks from compromising proper ethical standards during development. Organizations are familiar with technical debts; we know they create lower-quality products that are hard to scale, leading to increased overall costs. Similarly, the consequences of ethical debt are unavoidable in the long run. The compromises made during development can manifest in multiple ways such as bad press coverage, loss of data and clashes with government officials and regulators often ending in stiff penalties and reputational damage to the brand.
Today, there is more pressure on organizations to thoughtfully incorporate ethical standards into software development. This ensures there is strong protection against consumer data exploitation, guarantees that data is unbiased and contains diverse and inclusive considerations, and helps meet all regulatory requirements such as General Data Protection Regulation (GDPR).
The Challenge with AI
When it comes to AI, the problem of ethical debt can be magnified because there are added layers of obscurity dealing with the “unknown” outcomes. AI solutions are built to learn from existing data and if the base data hides biases, the action will propagate throughout the code and possibly be compounded with more complex logic. What’s worse is that large, modularized, code bases are reused in other solutions, often making it harder to identify root causes. The immense pressure on companies to go to market with products as quickly as possible creates this hard scenario. Consequently, data ethics, which rely on peer review and extensive testing, are often conceded.
How Can You Accumulate Ethical Debt?
As stated before, internal deadlines and intense competition to build the first solution puts significant pressure on companies to release AI solutions quickly. This hurry-up mode is one example that pushes management to make difficult choices, sometimes opting for speed with development, knowing there will be defects to address post-deployment. Other ways organizations can accumulate ethical debt are presented below, along with some notable examples.
1. Not Building Enough Safeguards for Data Misuse
Data misuse can come in various forms. One example is insufficient design or insufficient testing to ensure there is strong protection built into the system. Consequences can lead to:
- Phishing attacks – deceptive tactics are used to compel the user to share sensitive information.
- Malware attacks – malware code can infiltrate the system, gaining access to customer data. It can also spread quickly from one machine to another or allow trojans to steal personal data
Remember the Yahoo breach? This was one of the largest data breaches that happened with Yahoo between 2013 and 2016. Russian hackers targeted Yahoo’s database using backdoor access and stolen backups, then latched on to access cookies. They stole a myriad of personal information including name, phone numbers, birth date and passwords.
Another example of data misuse is misinformation. It can start with inaccurate data that is collected and disseminated, either inadvertently or intentionally, to mislead the public. Misinterpretation of this information can lead to sever consequences including public harassment, hate speech and damage to reputation or brand.
According to data from Misich & Cross, “8 in 10 Americans get their news on digital devices.” This is a stark change from the 1960’s when news was broadcast nationally as a public service. Today, news is accompanied by “expert” opinions from pundits and packaged for consumption in small bits on social media. There is a lot of pressure to be first with a headline. Sometimes, there are valid reporting errors; and other times, the facts are inflated for effect.
One example of misinformation was related to the AI Chatbot from Elon Musk’s xAI. The app falsely blamed NBA star Klay Thompson for throwing bricks through windows of multiple houses. The theory is the AI app hallucinated, which occurs when the large language model (LLM) powering the app generates incorrect or misleading information. In basketball, the term “throwing bricks” is commonly used to refer to missed shots.
2. Compromise on Privacy Protection
During the coronavirus, the Zoom app was a popular option for corporate and casual video conferences. At the same time, there was an increase in cyber harassment. In one instance an unidentified person dialed into a high school teacher’s class and yelled profanity. In another example, an uninvited person started waving swastika tattoos during a virtual high school class.
Zoom acknowledged it needed to fix the app and started making the meetings private, requiring passwords and allowing guests to remain in a waiting room before they are invited to the meeting.
A core practice, especially with AI solutions, is to include privacy in every stage of the lifecycle development of the software, starting with project inception. Teams are using the “shift left” approach to discuss and identify privacy considerations during the initial design phase when requirements are fleshed out and the initial architecture is conceptualized. This allows them to develop effective test scripts that can execute at each phase, during development Sprints, testing, deployment and post deployment for ongoing operations and maintenance.
3. Bias and Discrimination with Algorithm
The truth is systems perform based on how the developer codes them. The logic that runs the algorithm is not always transparent; in fact, it’s shielded from the public in proprietary systems or black boxes behind corporate boundaries. This makes it difficult to know where and when the bias occurs. For example, we don’t know fully how ads are targeted or how insurance companies come up with our auto insurance premiums. As consumers, we see the outcomes or results of the algorithm. Examples include the ads that pop up in our social media feed, or the bill for our insurance.
Some of the biggest pitfalls with algorithms can result from the makeup of the dataset. A dataset may be incomplete- for example, missing seasonality data or having two years worth of data for an operation that ran five years. The data or sample may have been selected poorly; for example, disproportionally skewed towards one gender, one age group or a particular race over the population. In scientific studies, populations are carefully selected, categorized and peer reviewed. It’s unclear if this level of rigor is consistently applied to AI datasets.
One example that’s highlighted in multiple places is the flop with the Amazon resume screening tool leveraging AI. The company was hoping to improve efficiency with resume screening and selection, so they developed an algorithm based on resumes it collected for a decade. They forgot to consider that most resumes in this dataset came from men. It was later clear that the algorithm preferred selecting men over women, effectively discriminating against women applicant. Amazon claims the system was never used.
In a healthcare example, an algorithm was used by hospitals and insurance companies across the U.S. to identify patients that needed high-risk care to prevent serious complications. It turns out, the model was biased towards White patients. A closer look revealed that one factor giving Black patients a lower score was their spending on healthcare. In reality, income levels contributed to less visits by Black patients, who were less likely to access medical care even when they had insurance.
There’s a separate argument that an algorithm will not necessarily be fair in the future, even if it follows good practices today. The model is built with past data or training data and can create data drift, which is a degradation of the model performance over time attributed to changes in data and changes in relationships between variables. In other words, the assumptions we made about the dataset today may not be valid tomorrow. Fortunately, we are learning to measure data drift and account for changes. Summary statistics are a good approach; for example, calculating the mean, variance, median and other measures. We can also look at data distribution and monitor outliers. The appearance of new outliers are indicators that the data needs a closer review.
Accelerate with Caution
The advancement of technology is exciting and we are moving at a faster pace with AI compared to other innovations in the past. Companies are spending more money to get it right, learning from their mistakes and sharing the lessons learned with the broader community. There is also increased awareness among consumers demanding that systems be transparent with how companies use our information.
The risks remain the same. Companies are making decisions on how much ethical debt they will accumulate and where they will compromise good values for profit. Consumers are the ones impacted by it and we pay that debt with privacy incidents, most commonly losing our identities to bad actors. We also see worse cases where consumers are victims of bad AI solutions, for example not getting the job we want or missing out on good health care.
Understanding ethical debt should be a priority, perhaps a mandate, for all software development projects. This is the only sure way to gain our customers’ trust and satisfaction in our products. If you’re committed to making responsible data ethics and governance core principles within your organization, let’s connect for a Strategy, Vision and Roadmap session to outline the best path forward.
