Big Data in Supply Chains: Separating Hope from Hype


This blog is aimed at Supply Chain leaders and their business teams. It is intended to help you understand where supply chain and big data come together as practitioners in these disciplines currently have limited understanding of the other side. 

What is big data?

While the term big data is hyped big time by technology vendors, most supply chain leaders and their business teams do not understand the concepts and opportunities well given the misdirection from technology vendors. Most technology vendors attempt to dazzle their audience with the ever growing data volume and the speed with which they can sift through the volume. What they don't know or forget to tell you is that most supply chains do not have an issue with volume. The real challenge lies with the variety and velocity of data for which you require solution providers that get the supply chain problem and its vertical context, not vendors with a one size fits all approach to big data platforms and solutions. 

Why should you care about big data?

Most enterprises struggle today with getting quality insights & visualization of their structured data trapped in ERP, Supply Chain, CRM, or legacy BI solutions. They are right to wonder who cares about big data when getting quality & timely insights from small data remains unfinished business. The short answer to why should I care is that's the way for enterprises going forward to survive. Supply chains can no longer be about driving costs down, they are increasingly about enabling top line growth. This requires a precise understanding of the volatile demand signal, proactive and micro-targeted shaping of the demand signal, optimization of inventory buffers across all echelons to cover for demand volatility, and a supply chain value network that can turn manufacturing, procurement, and distribution plans on a dime. Supply chains need to be proactive, agile, and outside-in to disrupt or risk being disrupted. The balance, strength, and resiliency needed from supply chains are not possible without having the full holistic view  of demand and supply that only a big data analytics based solution can enable.

Why can't you get needed insights with current solutions?

The current set supply chain solutions are based on linear, deterministic optimization and are built largely for managing the enterprise on-premise, not the entire value network in the cloud. The depth of optimization available is simply not sufficient for today's global multinational company in a setting of many-to-many value network of trading partners. Supply chains are a non-linear complex system with variable lead times that are changing all the time. Concurrent optimization and cognitive learning are needed to have the flexibility needed to generate quality insights. Yesterday's systems applied to today's supply chains is akin to taking a snapshot of the freeway every minute and driving based on it. You need a fundamentally different architecture: one that combines cloud, excel like intuitiveness, big data prescriptive analytics solutions with cognitive learning capabilities to learn with and from humans and get smarter over time.  

In traditional supply chains, business processes and software largely designed in nineties focus on processing linear and structured data. Master data like cycle times and yields are stale averages of historical performance, and bear no resemblance to current reality or future trends of actual values. These simplistic systems have neither confidence intervals nor the understanding of external factors like competition's pricing/other actions, economic trends, consumer confidence, and other data like weather and natural disasters. Planners often get frustrated at the poor solution quality of these systems and their inability to make changes to input parameters or enhance the processing logic in the engine with external factors to produce a meaningful projection of plans & future performance. It's no surprise that in most organizations, planners are building models and planning in spreadsheets, draining needed actual shipments & current backlog data from ERP/APS/BI solutions into these spreadsheets, and then figure out creative ways to overwrite system generated plans with spreadsheet based output for link & signal to the ERP system. This is largely transparent to the technology vendors who continue to believe and tout the usage of their planning solutions to the next set of gullible customer prospects. 

How should you look at the Big Data Opportunity?

Make no mistake. This is a business performance improvement initiative and should be led by the CFO or Chief Supply Chain Officer. It is NOT a technology initiative to be run by the CIO and his team. Make sure you are not doing it for the technology cool factor, resist the technology vendor sales & marketing machine. They have little clue about how to deliver the value they are promising and can hurt adoption of  big data and the value it can deliver by taking a technology centric approach to a business problem. Becoming data driven is a bigger cultural shift than many can imagine. Treat technology as a mere enabler in this cultural shift, not a panacea. 

For Supply Chain Leaders, the challenge is to rewire the brain of the organization to imagine how these new technologies and approaches can transform business processes and deliver a step change in financial performance. The opportunity is not in processing the volume faster (the processing speed argument of crunching last 10 years of history with HANA in seconds, ask the so what question). The real opportunity is understanding how you can predict the future demand and supply situation better, faster, which requires leveraging more type of data (the variety side - prescriptive analytics with cognitive learning capabilities) and generate actionable insights in a timely way (the velocity side - streaming analytics). 

There is a lot of hype around Internet of Things and Predictive Maintenance or Driver less trucks type scenarios in the market, but that's not low-hanging fruit by any means given the huge capital expenditure associated with replacing current manufacturing or transportation equipment with machines that have the necessary sensors required to stream valuable temperature or RPM data or the capability to drive themselves in the field.  

We see real hope with Demand Sensing / Shaping and Supply Chain Visibility as the two key areas where big data in supply chain can help right away - without a huge capital expenditure. 

  1. Demand Sensing / Shaping: Chemical or Semi-conductor companies often use the argument that they are too far upstream in the value network to care about sensing end consumer demand and attempt to deal with the demand signal erratic movements and bull-whip with Assemble To Order Push/Pull techniques and setting inventory buffers lower down in their Bill of Material as Die Bank stock in the case of Semi-conductor companies. At the same time these companies have many horror stories to share when major OEM customers have canceled or severely curtailed demand last minute. Given the long lead times in manufacturing and the global back and forth in these heavily outsourced supply chains, these demand shocks can be crippling to the business performance in more ways than one. Treating next tier customer's forecast or orders as a demand signal is no longer enough and is indeed a very risky strategy. Executives in these companies often guess the appropriate supply response level against a stated customer forecast (build 60% of the number they are being asked to deliver) to limit exposure. Supply chains can benefit greatly if these seat of the pants based decisions can be supported or replaced by a data driven culture of understanding and influencing downstream demand better. 
  2. Supply Chain Visibility: Demand signal management while absolutely necessary is not sufficient given you get to a point where more squeeze is not worth the juice. Demand sensing projects done well can cut forecast error in half, but they cannot eliminate it entirely. Supply chain visibility enabled by big data can deal with the remaining volatility by combining pieces of precise visibility with predictive and prescriptive response management engine that can react profitably to the inevitable shocks or minimize collateral damage. Today the Spanish garment manufacturer Inditex has an RFID reusable tag to each and every one of their garments. Imagine the precise visibility this enables in their supply chain resulting in much more efficient operations (inventory counting in stores, stock visibility across channels down to color/size level). Supply chain visibility enabled by RFID tags (one per discrete unit, picking tote bag, or delivery truck) is a far better ROI than smart machines in the manufacturing world. Combine this level of visibility with AI enable cognitive learning software that can sense demand-supply disconnects proactively and figure out profitable responses to actual disruptions in the moment. This can deliver the step change in financial performance through supply chain excellence built on a 20/20 line of sight and adaptive agile response.

The possibilities for disruption are endless once you can sense demand real-time and react to it profitably. Think Uber over regular taxi service. Now you can harness insights from data that does not fit neatly in rows and columns (small data world). Choice is yours: do you want an army of excel experts sweating the spreadsheets to get some after the fact insights or do you want to leverage big data analytics systems to detect patterns and benefit from these insights at the speed of light. 

Key takeaways: 

  1. Listen deep, listen often: Know your customer. Really well. Most companies get direct feedback in terms of reviews & ratings. You also have blog comments and feedback through social media. Listen cross-functionally to end consumer sentiment as a value network. Test and learn from market response in real-time in the new world where digital marketing becomes the predominant marketing vehicle. Today, sentiment data is largely contained within the digital marketing team today, it needs to be pervasively understood throughout the organization. Set capabilities to answer questions you are not asking today. Know what the gap is in performance vs. expectations, not performance vs. specifications. Do you know how long it takes your company to learn about end-product / service failures in the market and how can you adapt plans to recover from broken product/service faster to deliver superior customer experiences more often. Listen better, respond faster. 
  2. Sense before responding: Ready, shoot, aim is the world of most supply chains today. They believe in: why sense when you can respond. Traditional supply chain responses are often late and inappropriate. They are based on history, not current market data. They are poor at sensing both the demand and supply volatility. Relying on order and shipment data increases latency and eats into valuable time that is needed to respond effectively to market shifts. Move from a fire fighting mindset to an orchestrating mindset.
  3. Supply Chain Solutions need to get much smarter: Today's supply chains are hard wired. They are inflexible. The response is based on average values (average lead time) and simple if-then-else logic. Design your supply chain for flexibility and resiliency. This requires cognitive learning that can deal effectively with non-linear trends. You need rules based ontologies (multiple ifs to multiple thens and systems that learn from a range of outcomes due to interactions of multiple variables in a real-time sensing way). 
  4. Demand Sensing and Supply Chain Visibility are the low hanging fruit: With cloud based network wide solutions, it is finally possible to flowcast a demand signal through the multiple tiers of the network based on end consumer POS signal or a better understanding of all the market drivers impacting demand from your next tier customer. Geo-location, mapping data, and visualization along with supply sensing transmission (e.g., sensors on items, totes, trucks, and rail cars) transform supply chain visibility to real-time location awareness and ability to predict delays much more accurately. This is transformational, you must evaluate it.  

Over the last 3 years, companies are making little progress on adopting big data analytics despite all the hype technology vendors have created. Technology without business context is a bull in a china shop, watch out. Focus on teasing benefits from the data variety and velocity side, don't fall for the sexy comments on the volume side (e.g., world will have gazillion petabytes of data created in next 5 years, so what?). Unstructured, mobile, streaming, and geolocation data holds great promise to improve supply chain processes/KPIs through precise visibility, but you cannot get there with current structured data based solutions. Leaders will differentiate themselves with art of the possible in big data, while laggards remain stuck with rows/columns of structured data: driving 60 mph based on traffic snapshots every minute and having frequent accidents. You can have a medical insurance policy that covers all accidents, but that's not the smart way to drive. Start carefully, start small, and start with the right partners who get the intersection of supply chain as a functional domain and big data technology layer that can deliver valuable insights and boost your business performance.