Why We Need to Audit Algorithms

Thoth_Adan/Getty Images

Algorithmic decision-making and artificial intelligence (AI) hold enormous potential and are likely to be economic blockbusters, but we worry that the hype has led many people to overlook the serious problems of introducing algorithms into business and society. Indeed, we see many succumbing to what Microsoft’s Kate Crawford calls “data fundamentalism” — the notion that massive datasets are repositories that yield reliable and objective truths, if only we can extract them using machine learning tools. A more nuanced view is needed. It is by now abundantly clear that, left unchecked, AI algorithms embedded in digital and social technologies can encode societal biases, accelerate the spread of rumors and disinformation, amplify echo chambers of public opinion, hijack our attention, and even impair our mental wellbeing.

Ensuring that societal values are reflected in algorithms and AI technologies will require no less creativity, hard work, and innovation than developing the AI technologies themselves. We have a proposal for a good place to start: auditing. Companies have long been required to issue audited financial statements for the benefit of financial markets and other stakeholders. That’s because — like algorithms — companies’ internal operations appear as “black boxes” to those on the outside. This gives managers an informational advantage over the investing public which could be abused by unethical actors. Requiring managers to report periodically on their operations provides a check on that advantage. To bolster the trustworthiness of these reports, independent auditors are hired to provide reasonable assurance that the reports coming from the “black box” are free of material misstatement. Should we not subject societally impactful “black box” algorithms to comparable scrutiny?

Indeed, some forward thinking regulators are beginning to explore this possibility. For example, the EU’s General Data Protection Regulation (GDPR) requires that organizations be able to explain their algorithmic decisions. The city of New York recently assembled a task force to study possible biases in algorithmic decision systems. It is reasonable to anticipate that emerging regulations might be met with market pull for services involving algorithmic accountability.

So what might an algorithm auditing discipline look like? First, it should adopt a holistic perspective. Computer science and machine learning methods will be necessary, but likely not sufficient foundations for an algorithm auditing discipline. Strategic thinking, contextually informed professional judgment, communication, and the scientific method are also required.

As a result, algorithm auditing must be interdisciplinary in order for it to succeed. It should integrate professional skepticism with social science methodology and concepts from such fields as psychology, behavioral economics, human-centered design, and ethics. A social scientist asks not only, “How do I optimally model and use the patterns in this data?” but further asks, “Is this sample of data suitably representative of the underlying reality?” An ethicist might go further to ask a question such as: “Is the distribution based on today’s reality the appropriate one to use?” Suppose for example that today’s distribution of successful upper-level employees in an organization is disproportionately male. Naively training a hiring algorithm on data representing this population might exacerbate, rather than ameliorate, the problem.

An auditor should ask other questions, too: Is the algorithm suitably transparent to end-users? Is it likely to be used in a socially acceptable way? Might it produce undesirable psychological effects or inadvertently exploit natural human frailties? Is the algorithm being used for a deceptive purpose? Is there evidence of internal bias or incompetence in its design?  Is it adequately reporting how it arrives at its recommendations and indicating its level of confidence?

Even if thoughtfully performed, algorithm auditing will still raise difficult questions that only society — through their elected representatives and regulators — can answer. For instance, take the example of ProPublica’s investigation into an algorithm used to decide whether a person charged with a crime should be released from jail prior to their trial. The ProPublica journalists found that the blacks who did not go on to reoffend were assigned medium or high risk scores more often than whites who did not go on to reoffend. Intuitively, the different false positive rates suggest a clear-cut case of algorithmic racial bias. But it turned out that the algorithm actually did satisfy another important conception of “fairness”: a high score means approximately the same probability of reoffending, regardless of race.  Subsequent academic research established that it is generally impossible to simultaneously satisfy both fairness criteria.  As this episode illustrates, journalists and activists play a vital role in informing academics, citizens, and policymakers as they systematically investigate such tradeoffs and evaluate what “fairness” means in specific scenarios. But algorithm auditing should be kept distinct from these (essential) activities.

As this episode illustrates, journalists and activists play a vital role in informing academics, citizens, and policymakers as they investigate and evaluate such tradeoffs. But algorithm auditing should be kept distinct from these (essential) activities.

Indeed, the auditor’s task should be the more routine one of ensuring that AI systems conform to the conventions deliberated and established at the societal and governmental level. For this reason, algorithm auditing should ultimately become the purview of a learned (data science) profession with proper credentialing, standards of practice, disciplinary procedures, ties to academia, continuing education, and training in ethics, regulation, and professionalism. Economically independent bodies could be formed to deliberate and issue standards of design, reporting and conduct. Such a scientifically grounded and ethically informed approach to algorithm auditing is an important part of the broader challenge of establishing reliable systems of AI governance, auditing, risk management, and control.

As AI moves from research environments to real-world decision environments, it goes from being a computer science challenge to becoming a business and societal challenge as well. Decades ago, adopting systems of governance and auditing helped ensure that businesses broadly reflected societal values. Let’s try replicate this success for AI.

3 Split Tests to Try for Your E-Commerce Store

Split tests are by far one of the best ways to improve your conversion rate on your e-commerce store. A split test is essentially a small change made within your business model to see if it produces an increase in profit.

We are going to look at some of the ways you can implement these small changes in your day-to-day business routine in order to boost your revenue, get more click-throughs, and grow your business.

The first thing you should know is not all A/B tests produce insane amounts of revenue beyond your wildest dreams. When you run these types of tests on your business page, the goal is to find small changes that work.

As long as you keep an open mind and remember that not every change is going to profitable, you'll eventually start learning what split tests work well within your business model.

Now that we have that caveat out of the way, let's look at some of the most successful types of split tests and how you can start using them today.

Editor's note: Need an e-commerce website design service for your business? Fill out the below questionnaire to have our vendor partners contact you with free information.

Change a suggestion to a CTA (call to action)

The way you use your words is the single most important defining factor when you build an e-commerce brand. Your goal is to convince the customer that you have a product worth buying and that your brand is trustworthy, high quality and dependable.

Confidence is key in all areas of our life, and running an online store is no different. You have to go into it with the mindset that the customer is there because they are 100 percent ready to make a purchase. Your language has to reflect this confidence. One company found that when they changed a suggestion to a call to action, the CTA outperformed the original by a staggering 127 percent.

This is what it looked like:

Original: Get a Free Trial

Variation 1: Start Free Trial

Variation 2: Get Free Trial

Variation 3: Free 30-Day Trial

Variation No. 1 is the one that increased revenue by such a wide margin. The theory is because "Get a Free Trial" sounds like a suggestion. As in, "Would you like to get your free trial?" Variant No. 2 is similar to the original, and variant No. 3 has no real suggestion. The first variation oozes confidence and assumes the customer is there to take advantage of the trial. You can use this call to action whether you're offering a discount, a free gift (e-book, physical item, etc.), or even just if you're trying to get someone to sign up for your newsletter.

Which one looks better? "Would you like to sign up for our newsletter?" or "Start getting our newsletter and awesome deals now!" The choice is obvious. Words are everything!

Try different images

The way you display images can have a huge impact on your conversions. There are some people who like to see images of the product they are purchasing at all angles, much like you would see on an Amazon listing.

Other people like to see people using the product in a practical way. For example, if you're selling a vacuum, you might show an image of someone using your vacuum in a sparkly clean home. This image can change how people view your product. They might think, "I want my home to look that clean," and that can initiate the purchase.

When you're trying to decide what images work best for your product or brand, make sure you are testing both images over a set period of time so you can see what kind of images attract your audience.

Change pricing options

There are multiple split tests you can use when changing your pricing. The first method involves deciding whether you want to use even or odd numbers when pricing your products.

Odd number prices, such as 19.99, can encourage conversions, because people feel like they are getting a deal. They feel as if they are only paying 19 dollars instead of 20, even if there is only a 1 cent difference.

On the other hand, most companies that advertise their products as being high quality use even-number pricing, like $100.00. The even numbers lead customers to believe that they are investing in a product that is top of the line.

Another way you can use split tests with your pricing and possibly reduce abandoned e-commerce carts is by changing the price of your product and seeing how customers respond. One company had a product that had an initial cost of $19.95. They did a split test to see if raising the product to $29.95 to see if the same percentage of customers still made purchases on the website.

Their findings? At both prices, 1.1 percent of customers followed through with their purchase. In other words, this business owner increased revenue by 36.48 percent just by split testing the price. You can also add coupon codes to see if customers respond well to a "discount," even if the base price is the same after the coupon.

In the previous example, if the business included a 36 percent discount, but 5 percent of people who viewed the page converted, that's a 4x increase with the addition of a coupon code and a slightly higher price!


Split tests offer entrepreneurs nearly infinite possibilities when it comes to increasing and adjusting sales. It's important to keep in mind that every business is different, so what works for one company may not work for another.

If you want to continually grow your business, it's vital that you continue testing variables and find ways to make your e-commerce store more appealing. Learn from your failures and continue making changes.

Even if you think you've found an award-winning formula, continue split testing. There is infinite room for growth in a free market. If you're willing to continue making small changes and measure how it impacts your business, you're constantly tweaking your business for success.

There is no "end goal." As your business grows, you should continue making changes and growing alongside your online store.

MIT Sloan Management Review

It’s no surprise that many large multinational corporations are paying increased attention to sustainability-oriented innovation (which we’re broadly defining as improvements for social good, not just “green” initiatives).1 Faced with mounting challenges and pressure from governments, nongovernmental organizations (NGOs), investors, and employees to be more aware of the environmental and social impacts of business activities, companies are searching for ways to do things differently while also seeking opportunities for growth.2 As a result, many are attempting to tap into the creativity and entrepreneurial potential of their employees, encouraging them to develop new products, services, or business models that create value for both the company and society.3

Sustainability-oriented innovation isn’t new — researchers have been studying it for more than a decade.4 However, as traditional organizations adopt new technologies and business models, some are finding it difficult to get employees to think like entrepreneurs, which is essential to building an innovative culture that’s committed to solving environmental and social problems.5 To learn what leading companies are doing to address that challenge, we conducted interviews with managers at seven multinational companies recognized for their sustainability activities: AkzoNobel (a Dutch paint and chemicals company), Interface (a U.S. carpet-tile manufacturer), Johnson & Johnson (a U.S. medical products, pharmaceuticals, and consumer goods company), Pearson Education (a British education and publishing company), Koninklijke Philips (a Dutch company involved in health care, consumer electronics, and lighting),6 UBS (a Swiss financial services company), and Unilever (a Dutch-British consumer goods company). (See “About the Research.”) In this article, we integrate the experiences of these companies with findings from prior studies to illustrate what effective sustainability-oriented innovation looks like and to describe how companies can cultivate the entrepreneurial thinking and behavior that will spark such efforts.

What Sustainability-Oriented Innovation Culture Looks Like

Sustainability-oriented innovation takes many forms — everything from the development of new or improved products or services to the creation of new processes and business models that bring benefits to the environment or the society at large.7 While the innovations themselves may or may not be disruptive or radical, the idea is to mitigate the negative impacts of existing solutions or, even better, make a positive impact.8

In describing environments that are successfully fostering sustainability-oriented innovation, most of the executives and managers we interviewed highlighted the close connection between individual purpose and corporate purpose and the importance of linking the company’s long-term interests with the good of society. (Those themes are prevalent in other studies, as well.9) There is a segment of employees, interviewees noted, who deliberately choose their employer on the basis of its commitment to sustainability. Having a common purpose, many said, helps set the direction for what individuals and corporations actually do.10 And when individual and corporate purposes are aligned, managers observe that employees are willing to go above and beyond what is required to express their values.

Building the Right Culture

Our research is designed to help corporations motivate and incentivize employees to change their behaviors and actively support new initiatives. So what can executives and managers do to promote sustainability-focused innovation in their organizations? Based on the experiences of the seven companies we studied (which reinforce findings in other research),11 we have identified five essential elements. First, companies need to have a clear direction and be able to articulate the goals to employees. Second, they need to provide an adequate budget and other resources (including the space, time, and training) to pursue the goals. Third, they must have what we call “room for collaboration” — that is, they need to allow people to work with other parts of the organization or partner with suppliers, customers, NGOs, and other third parties to address gaps in skills and resources. Fourth, companies should provide positive reinforcement to employees who get involved in sustainability-oriented innovation projects. And finally, they need to institute measures of accountability that establish social and environmental value creation as a priority.

A clear direction. To translate a corporation’s purpose into action, it’s important for managers to articulate how sustainability fits into the company’s broader goals. That involves explaining how sustainability-oriented innovation supports strategy and is embedded in day-to-day operations. At Unilever, for example, CEO Paul Polman has made sustainable growth a strategic priority and an essential part of the company’s long-term growth model. With Polman’s backing, the company created a road map that aims to increase Unilever’s positive social impact while strengthening its competitive position. Unilever employees we spoke with said the plan is motivating, and many see it as an invitation to seek out their own ways of pursuing sustainability-oriented innovation.

Yet we also found that managers need to be willing to go beyond merely voicing encouragement for sustainability-oriented innovation efforts. They must actively guide and support employees who are involved in such activities. For example, at Interface, the carpet-tile company, one of late founder Ray Anderson’s most far-reaching goals during the early 2000s was to achieve zero waste. Interface’s executive board continues to encourage employees to create new sustainability-oriented products, processes, and materials, and identifies steps people can take to carry out Anderson’s vision. In one employee-led initiative, Interface is collaborating with local fishing communities in the Philippines and Cameroon to recycle discarded nylon fishing nets and turn them into material for new carpets. To support the business model, executives actively and publicly get behind sustainability-oriented initiatives like this one, thereby legitimizing the activities internally.

We found that having a coherent set of corporate sustainability values can serve as a behavioral and decision-making compass for employees. Johnson & Johnson’s credo — putting “the needs and well-being of the people we serve first” — doesn’t just describe the company’s commitment to society. It also gives individuals a clear context within which to act. Recently, a Johnson & Johnson employee initiated an effort to improve access to mental health care in sub-Saharan Africa. Once she was inspired and empowered by the company’s mission, she took steps to innovate accordingly, and the company identified ways to provide training for mental health care professionals in the region and give local residents access to less expensive medications.

Budget and resources. In addition to setting a clear direction, companies seeking to promote sustainability-oriented innovation need to make sure employees have the necessary funding, expertise, and corporate support to pursue related projects. Each of the companies we studied recognizes the importance of providing financial resources and helping employees connect with the right experts. In many cases, supportive business units and people in various roles throughout the organizations lend their support to sustainability-oriented projects and raise the profile of the initiatives. Pearson, the education company, for example, established what it calls Tomorrow’s Market incubator in 2016 to explore possible new products and ventures focused on expanding educational opportunities for people in low-income communities. Employees with ideas they think are promising can apply for funding, receive support from executives, and get time off to execute their business plans.

Like Pearson, many of the companies we studied give employees permission and flexibility to pursue the sustainability-oriented activities they are interested in. Philips, for example, gives scientists in its innovation department time on Fridays to work on their own projects and lets them apply for three-month research fellowships to pursue ideas of interest. In addition, the company gives sustainability projects leeway on how fast they are expected to show returns.

Obviously, a company’s ability to innovate successfully takes more than ambition, time, and financial resources.12 It also depends on being able to attract the right talent and helping employees develop the right capabilities and skills. Recognizing that it wasn’t attracting enough entrepreneurially minded employees, Unilever altered the way it recruits recent graduates and added a new program that focuses on hiring individuals with a sustainability mindset. Johnson & Johnson, for its part, has created what it calls the Social Intrapreneurship Program, which is modeled after a program developed by the Aspen Institute and is specifically aimed at helping employees learn how to develop initiatives focused on sustainability. Consultants teach participants how to attract resources, identify partners, and work collaboratively to achieve common goals.

Among the companies we studied, Interface stands out in its commitment to training. Every new employee goes through a mandatory training program detailing the importance of sustainability for Interface, describing activities underway and challenging people to seek out new opportunities for sustainability-oriented innovation. The company also provides optional advanced training aimed at helping employees become more effective in societal problem-solving. Employees say the training helps orient them to Interface’s philosophy and strategy, making them feel more positive about the company.

About the Research

This article is based on 63 semistructured interviews with top, senior, and midlevel managers responsible for sustainability-oriented innovation activities at seven multinational corporations. The companies were selected based on their rankings on the Dow Jones Sustainability Index and the GlobeScan/SustainAbility survey, which surveys thought leaders on sustainability issues. We asked interviewees to identify factors that stimulate, support, discourage, or hinder employees in relation to sustainability-oriented innovation projects inside their organization and to provide examples of best practices. We then used the most commonly mentioned projects and practices to construct our overview.

Room for collaboration. Many of the managers we interviewed stressed the importance of collaborative relationships both within their companies and with groups on the outside. Several interviewees talked about the negative effects that internal competition and silo thinking can have on innovation and the benefits of collaboration across disciplines. Some organizations are reinforcing that idea through behavioral incentives. Consider UBS, which has placed greater emphasis on collaboration over the past several years. Not only does top management regularly tout the value of collaboration, it has made it part of employees’ performance evaluation and the basis for recognition awards.

Other companies have redeployed internal capabilities by developing sustainability-oriented programs as offshoots of their traditional businesses. Philips, for example, wanted to leverage its expertise as a lighting specialist to develop business models that produce both business and environmental benefits. Through conversations with one of its clients, it came up with the idea of selling “light as a service” to commercial customers. Rather than selling lighting equipment in a traditional way, Philips designs an energy-efficient system and charges a fee — with a commitment to adapting the system as the customer’s needs change and, eventually, recycling it.13

Having external partnerships, interviewees told us, enabled their companies to access capabilities they didn’t have internally while also establishing helpful new ties.14 Pearson, for example, established a strategic relationship in 2015 with Save the Children to provide education resources to child refugees. The immediate goal of the program, called Every Child Learning, is to increase educational opportunities for Syrian child refugees living in Jordan. However, Pearson treats it as a pilot program that could be adapted to other conflict-torn areas.

Several managers we interviewed emphasized the importance of having close relationships with community-focused stakeholders in supporting sustainability-oriented innovation. Such relationships enable companies to interact with local philanthropic groups on a regular basis, thereby allowing employees to identify potential entrepreneurial opportunities that can benefit society.15 For example, AkzoNobel, the Dutch paint and specialty coatings maker, developed a relationship with an NGO in the United Kingdom named Community RePaint. It collects tens of thousands of gallons of leftover paint from British households each year and remanufactures it for reuse by communities and charities, thus diverting the paint from landfill.

In studying how companies encouraged sustainability-oriented innovation, we found that while it is important for employees to have expertise in their own field, they also need to be interdisciplinary because the economic and societal value their initiatives generate transcends functional boundaries. Cross-functional teams and committees play a big role in bringing stakeholders to the table and fostering collaboration. Interface, for example, has a sustainability ambassador program. Its goal is to not only educate employees about sustainability and challenge them to innovate, but also cultivate a cadre of enthusiastic and committed employees from different parts of the company.

Positive reinforcement. A proven way to motivate employees to go beyond initial involvement in innovation activities is to offer positive reinforcement. Managers do this through both informal compliments and formal recognition. Although many of the employees who take part in sustainability-oriented innovation projects are intrinsically motivated to participate in these activities, companies should look for ways to expand participation and show how the personal rewards exceed the costs. Although UBS, as a financial services company, operates in a highly regulated industry, it tries to encourage its employees to think entrepreneurially. At a recent event, the company’s top and senior managers spoke about the need to praise and support the employees who are championing initiatives from inside the firm.

The need for accountability. Organizations that want to promote sustainability-oriented innovation need to define employee accountability in ways that extend beyond profit maximization for shareholders and adopt targets and appraisal systems that include sustainability. Having such incentives encourages employees to work toward ambitious objectives for sustainable growth.16 For its part, AkzoNobel introduced a bonus system that rewards top and senior managers for meeting long-term sustainability targets. To encourage investment in sustainability-oriented innovation, 30% of the top managers’ long-term incentive bonus is tied to how well the company performs in an index developed by RobecoSAM, a Swiss investment firm that focuses on sustainability.

Multiple interviewees noted that transparency about sustainability goals and performance helps companies meet new standards of disclosure, raises awareness among customers, and shows what the company sees as important. At Pearson, for example, management made a commitment to advancing quality education and reducing inequities. In keeping with these goals, the company has begun reporting on the “efficacy” of its products and services for students, with results audited by PricewaterhouseCoopers. Such practices can help companies become more accountable and set even higher targets and standards.17

Sustainability-oriented innovation can help companies become more competitive and identify new markets while addressing the world’s needs. However, unless management provides a clear sense of purpose, the right resources, a collaborative environment, positive reinforcement, and a commitment to accountability, companies will find it difficult to get employees to participate in sustainability-oriented initiatives. And without the enthusiasm and commitment of motivated employees, companies aspiring to become innovators in sustainability will risk losing their edge.

No comments