Data Lifecycle of a Supply Chain Distribution Center
Tags: datascience, datalifecycle, dataanalysis, dataengineering
Published: 2021-12-16 04:38:27

**Introduction**

Considering the data lifecycle is a crucial skill for data professionals.  Surely we've all heard the adage "garbage in, garbage out" - models are only as good as the data fed to them.  To build reliable and successful data solutions, we must understand how the data exists at each point in its lifecycle. 

Much of the content in the data space revolves around tech companies that deal almost entirely in digital products, if not completely so. One could practically make a second YouTube just for "how to get a job at a FAANG company" videos.  However, the supply chain - the complex network of businesses that deals in the procurement and sale of physical goods - also benefits from the work of data professionals, and there are considerations in the data lifecycle that may not be immediately apparent to those who are immersed in the tech side of the data.

I've worked in supply chain distribution (henceforth SCD) for the better part of a decade, and I want to collect some insights I've gained here to help folks with less experience in the domain think about the data in a practical way. This discussion outlines how data flows through a supply chain distribution center at three levels - the physical level, the operations level, and the solutions level. It also presents some questions to consider at each level when designing data solutions in an SCD environment. 


![Diagram of the levels of data in a SCD environment](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2n5vz6tqn5hu9elk2l1l.png)

**Data at the Physical Level**

When data professionals think about the source for data, it's often in terms of a click on a web page, or an object landing in an S3 bucket, or perhaps a database update or model refresh. The reality in SCD is that the source for a given point of data is almost always a physical action or product. 

When you consider the 3 V's - velocity, variety, and volume - these are almost entirely dependent on the people moving the products. Velocity and volume are dependent on the number of orders that can be shipped and received in a given unit of time.  Variety is based largely on the variety of products and services the company provides, with a base amount required for operations (HR, sales, procurement, and inventory control, for example).  

As an aside, e-commerce is a slightly different animal given that the velocity and volume of online orders can be decoupled from the physical operation, but many of the principles still apply. 

Questions to consider:  
 
- How are the employees reporting their work? (hand-written forms, ERP systems, warehouse management systems, etc)
- What are the possible opportunities for error, and which among those are the most common? (shipping or receiving incorrect items, incorrect item counts, data entry errors)
- What are the physical bottlenecks for your data, and how does this influence your paradigm? (real-time may not be possible or practical, given the rate at which orders are processed)

**Data at the Operations Level**

In this context, the 'operations level' refers to the volume and variety of data necessary for a business to function, as well as the means of moving and storing that data.  This is where we would typically find our databases, lakes, and warehouses containing ERP data that reflects the physical actions and entities described earlier.  Data at this level informs business operations, but doesn't necessarily support or drive initiatives.  

The diagram below provides a high level order of operations for data at this level:

- Items get sold, triggering a shipment
- Items get shipped, triggering a replenishment 
- Items get purchased, leading to a receipt
- Items get received to eventually be sold

![Diagram outlining the flow of data in a SCD operation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2t12viw83j7hl2xbyqfy.png)

There are more nuances to the process that are not being outlined here, but this roughly illustrates how the data works at this level.  Each of the employees involved in the steps above needs to know about the changes in the data caused by the other steps in order to perform their functions. This manifests as lines on sales orders, packing lists, purchase orders, and receiving documents - quantities, costs, descriptions, unique identifying numbers, addresses, etc.  Data at this level of granularity allows the business to operate but does not provide further insight the way the next level does. 

Questions to ask:
- How and where is the operational data being captured and stored? (hand-written orders, data entry, databases, data lakes, etc)
- What does a point of data at this level represent at the physical level? (Units of measure are big here - these represent units sold and convert to extended costs and extended sales)
- Who is accessing which data, how, and why? (Sales, procurement, and inventory control teams need to see each others' data. Analysts may need access to a wide variety of the data, including payroll and financial data not outlined above. Executives and administrators may have access to all of it)
- Which operational data may be most useful for supporting business initiatives? (Sales, costs, and customer demographics are the biggest ones)

**Data at the Solutions Level** 

The solutions level is where most data professionals spend the bulk of their time. Physical events and activities have been recorded, and that raw data has been shaped to address specific questions or problems.  Data at this level represents a transformation and interpretation of data at the previous levels, and as such it is subject to the quality of those levels. 

This is the core of why it is important to have a nuanced understanding of the SCD data lifecycle.  A whole team of data engineers, analysts, and scientists could be spending hundreds of hours (not to mention thousands of company dollars) developing solutions that won't perform because the data is being captured poorly at the physical level or managed improperly at the operations level.  
 
Questions to ask:

- Who is responsible for data management at the lower levels? (who are the operations managers, the DBAs, etc)
- What level error is tolerated, or not tolerated? (clearly establish the level of human error and determine how much improvement is necessary)
- What are the performance expectations for the solutions being developed? (timelines increase if data at lower levels needs more improvement; in some cases, the project may not be feasible)
- How involved are data professionals in defining the needs of and potential improvements to the data at previous levels? (the company should be willing and able to take on technologies and processes that serve the data strategy)

**Conclusion**

For data analysts and scientists, it is worth getting to know key team members at each of these levels, as ensuring data quality will involve coordination with them.  Learn who your leads are in shipping, receiving, and inventory control. Talk to your operations managers regularly. Get to know your DBAs.  Your work depends on theirs.  

For business leaders, know that getting real value out of your data starts at the physical level.  No amount of codified procedures for engineers, administrators, or analysts will correct errors at the physical level.  Make sure your operations teams have the resources they need to minimize the opportunity for error.  This will pay dividends across departments in your organization, not just for your data professionals. 

Thank you for reading!