Technology & Innovation

Credit check: Using advanced analytics to leverage a trillion dollars of data

October 07, 2015

North America

October 07, 2015

North America

With over one hundred million credit cards in the world accounting for over $1 trillion in charge volume every year, American Express deals with vast quantities of data.  The company has recently moved beyond data warehousing to an enterprise-wide Hadoop-based big data platform in order to maintain a competitive edge in analytics and decision science.

A commitment to the holy grail of digital commerce

A global financial services giant like American Express (Amex) holds massive quantities of data by the nature of its business. But until recently, the sheer volume was a daunting challenge and had not been fully leveraged. Amex data specialists and heads of lines of business recognised that the company was not taking advantage of a huge proprietary asset.

In 2010, leaders at American Express understood that technology was reaching a point where an enterprise-wide commitment to deep-dive analytics could change their business. Storage was becoming cheaper and access faster, and machine learning was evolving at an exponential rate. The company saw a chance to parlay its data into what the head of their Enterprise Growth Group called "the holy grail of digital commerce"—smarter fraud protection, greater customer satisfaction and new market-facing initiatives.

Three visionary objectives for data analytics

The company had three distinct objectives when it embarked on its big data journey. First, it wanted to refine its existing routines and processes using machine learning techniques. Second, it wanted to experiment with new techniques such as real-time analysis of item-level transaction data that had previously been impractical using traditional tools. Third, and maybe most ambitious, it wanted to explore innovative possibilities and new ways of thinking that hadn't occurred to others in the field.

Challenges faced

The central challenge to all of these objectives was intelligent management of the volume of data. Consolidating and prepping the constant inflow of both internal and external data proved to be a major ongoing challenge. Business priorities had to be set that balanced return against complexity of implementation. Privacy and regulatory constraints governed which data could be used, how it was used, and what the end-product would be.

Hoarding the data within the Chief Data Officer's office was not going to work. Dumping the information into a 'data landfill' and allowing general open access was not going to be the answer. The Amex team needed a way forward that balanced requirements for internal control with an ability to experiment and innovate.

The big solution for Amex

Amex found that a strategy of strict governance combined with individual flexibility was the solution. First, it established a formally-governed central databse directly under the authority of the Chief Data Officer (CDO). The CDO's office managed a strict policy of security, defined access, and set rules for internal curation and management of data. Access was given under strict read-write privileges, allowing Amex to exercise controls demanded by regulators and privacy experts.

The second step was establishing integrated, sanctioned areas where analysts could "take [the data] out and play with it" without endangering the sanctity of the database or the privacy of customers. This allowed the qualified business user to test the data, determine its usability, and conduct his or her own experiments to adanvce the business.

Putting the solution into action

A critical area addressed was fraud. Over time, the need to detect and manage fraud has become increasingly critical to its customers and larger parts of the business. By managing the anti-fraud operations through a big data environment and using machine learning, Amex has allowed more transactional volume to flow through for improved customer service and satisfaction while at the same time identifying trouble spots much more quickly. A second accomplishment was being able to measure customers' behaviour patterns in real-time. Rather than taking weeks to create large 'one size fits all' offerings, Amex is able to craft and extend individual offers for customers in minutes.

According to Manish Gupta, Executive Vice President and Head of Global Information Management and Data Products, Amex's digital prospecting models have been radically overhauled, resulting in responsive market-facing initiatives that would have been impossible a few years ago. "A good example would be when we suggest Amex Offers to provide merchant offers to card members," he says. "Our machine learning algorithm assesses things like spending preferences and which offers have been redeemed in the recent past. Using all the data in our big data set, a group of offers are prioritised for each individual and it keeps evolving as the algorithm learns and improves constantly. This process ensures we are providing the most relevant offers customised uniquely for every card member."

What was learned, what was accomplished, and the road ahead

The transition to the big data environment and the ongoing pursuit of strategic data objectives has resulted in a transformation of how American Express interacts with its customers. Every single transaction taking place through American Express—in excess of $1 trillion annually—is analysed in real-time using machine learning.

Company experts foresee a fast-approaching day when effectively every model in the company will contain at least some elements of big data optimisation. "We analyse each and every data point to figure out how we can get incremental benefit," says Mr Gupta. "The benefits go across the whole value chain, from prospecting to customer management to risk management, and they are very large—probably the largest that I've seen in my career."

Enjoy in-depth insights and expert analysis - subscribe to our Perspectives newsletter, delivered every week