Solution | Markdown Optimization#

Overview#

Business Case#

A data-driven approach to markdown optimization can yield improved margins by 400 to 800 basis points . This level of impact on margin is surely welcomed in today’s environment, where retailers and CPG companies alike are feeling the margin squeeze from inflation, overstocked positions, and shifting purchase behavior — just to name a few. As a result, getting markdowns right is now a necessity.

The right Markdown Optimization solution leverages machine learning models to go beyond historical data to predict the impact of price changes at the product level. This requires forecasting weekly sales with all possible discounts and training a demand forecast model on the historical data by product in order to test the discounts and estimate price elasticity.

In turn, companies can eliminate erroneous assumptions from fragmented data and short-sighted timeframes in order to equip business teams with the insights needed to better inform pricing decisions for the right products, at the right time.

With this Dataiku solution, retailers will achieve automated pricing and promotion insights that result in improved margins, increased sales revenue, greater sell-through rates, and optimized inventory levels. Companies will be well on their way to consistently delivering value to a changing consumer, while realizing profitable growth.

Installation#

The process to install this solution differs depending on whether you are using Dataiku Cloud or a self-managed instance.

Dataiku Cloud

Dataiku Cloud users should follow the instructions for installing solutions on cloud.

The Cloud Launchpad will automatically meet the technical requirements listed below, and add the Solution to your Dataiku instance.
Once the Solution has been added to your space, move ahead to Data Requirements.

Self-managed

After meeting the technical requirements below, self-managed users can install the Solution in one of two ways:

On your Dataiku instance connected to the internet, click + New Project > Dataiku Solutions > Search for Markdown Optimization.
Alternatively, download the Solution’s .zip project file, and import it to your Dataiku instance as a new project.

Technical Requirements#

To leverage this solution, you must meet the following requirements:

Have access to a Dataiku 12.3+* instance.
The Dataiku permission “May develop plugins” is needed to modify the Dataiku Application post-installation.
A working instance of the Demand Forecast solution is required to provide the necessary data inputs for this solution.
A Python 3.8 code environment named solution_markdown-optimization with the following required packages:

cloudpickle>=1.3,<1.6
lightgbm>=3.2,<3.3
scikit-learn>=0.20,<1.1
scikit-optimize>=0.7,<0.10
scipy>=1.5,<1.6
statsmodels==0.12.2
xgboost==0.82
gluonts>=0.8.1,<=0.10.4
pmdarima>=1.2.1,<1.8.5
mxnet==1.8.0.post0
prophet==1.1.1
holidays>=0.14.2,<0.25
click==8.1.3
Flask==2.2.2
Flask-Caching==2.0.2
importlib-metadata==5.0.0
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.1
Werkzeug==2.2.2
zipp==3.8.1

Data Requirements#

The Dataiku Flow was initially built using publicly available data. However, this project is meant to be used with your own data, which can be uploaded using the Dataiku Application.

Dataset	Description
demand_forecast_features (mandatory)	Is provided from the Dataiku Demand Forecast solution instance that is linked to the project. It will automatically be updated when the original dataset is modified.
product_balance_parameters (optional)	The parameters are used in the products with a custom optimization strategy: (profit_parameter profit_uplift + volume_parametervolume_uplift)/(profit_parameter+volume_parameter).
product_markdown_constraints (optional)	The solution will only explore discounts between min_markdown and max_markdown (minimum and maximum markdown should be between 0 and 1).
product_strategy (optional)	Specifies what metric is chosen for the optimization: profit, volume, or custom.
product_inventory (optional)	Defines the available stock for sales of the product (at the forecast_granularity level if defined).
product_cost (optional)	The base price is the reference public price of a product. The product cost is defined as the cost incurred to create, transport, and sell a given product. The difference between the selling price and the product_cost is the profit margin of a given product.
product_to_markdown (optional)	Lists the products to include in markdown optimization; if not used, all product will be optimized.
product_override (optional)	When you don’t want a product to go through the optimization process, put the override_flag to true, and the override_value will be used instead. If override_flag is false, the value is ignored.

Workflow Overview#

You can follow along with the solution in the Dataiku gallery.

The project has the following high-level steps:

Aggregate the configuration inputs to form an optimization strategy.
Prepare the demand forecast features to match the configuration.
Run the prediction for the new demand based on all possible markdowns.
Compare results to other strategies.
Select the best-performing discount.
Review and customize the constraints using the webapp.

Walkthrough#

Note

In addition to reading this document, it is recommended to read the wiki of the project before beginning to get a deeper technical understanding of how this Solution was created and more detailed explanations of Solution-specific vocabulary.

Plug and play with your own data and parameter choices#

To begin, you must configure the solution in the project landing page by using PROJECT SETUP. You will land on the configuration page, where you can select the connection used by the project and the Demand Forecast instance used as input.

The input dataset and models will be automatically loaded from the Demand Forecast instance. Still, the page should be refreshed after your Flow has been rebuilt so that subsequent parameters for selection can be updated with the relevant options.

After establishing your data connections, you can also select the optional datasets in this session, and you can fill in the dataset path or table name for the ones you use.

Including optional datasets can greatly enhance the accuracy, impact, and complexity of your markdown strategies, but it is sufficient to build a markdown optimization strategy with your demand forecast data alone.

With all of our data established in the Flow, we can use the default parameters section of the Project Setup to configure the default optimization parameters. Products that don’t have parameters specified in the optional datasets will use these default parameters for optimization and their margin. In this section, you can also select the step for the discounts and the used currency of your data.

The final two sections of the Project Setup allow us to very simply build the entire Flow with our selected data and parameters, and notify us when the build has been completed. Additionally, from this screen, we can easily link to the output dataset and output dashboard where our strategies can be analyzed.

If you’re only interested in the outputs of the Flow, you can skip the following section of this article, which will briefly explain how the solution Flow functions to take your input data and parameters and turn them into actionable insights.

Under the Hood: What happens in the solution’s underlying Flow?#

The size of the preprocessing Flow zone will depend on the optional datasets that have been connected. Regardless of how many optional datasets are used, the steps are the same. Here we aggregate the different configuration parameters, and use the project variables as default values.

Updates from the interactive webapp will override previous configurations to allow for post-build tuning and experimentation without needing to rebuild the entire Flow. Additional preprocessing is done in the markdown_preprocessing Flow zone in order to take the data from our Demand Forecast solution, use the features, and create new ones based on the configuration parameters. Our optimization will be based on the resulting features from this preprocessing step.

Once our features are ready, we apply the Demand Forecast model to the data that includes the modified markdowns. We also run two reference scenarios (Constant Markdown, considering that the markdowns stay constant from the reference period, and Zero Markdown, where all products are at full price).

The performances of the predictions are directly dependent on the Demand Forecasting models, especially the impact of price and discount over the demand. Verifying this before using the Markdown Optimization in a production environment is good practice.

Optimization is a result of the Flow adjusting price and discount parameters of the products to predict their effect. If those parameters positively impact the predicted demand, we can more accurately represent the price elasticity.

It’s also recommended to perform exploratory analysis on your data to ensure you have good coverage of markdowns on your historical data. Without it, the models will not be able to perform well for unknown markdowns.

Understanding our Products and Markdown Strategy#

From the Markdown Optimization dashboard, we have two ways of understanding and interacting with our solution outputs. In the first tab of the solution, we can interact with the webapp in order to check the aggregated results of the optimization and perform the customization of the optimization parameters for specific products.

Results Summary

The Key Performance Indicators are available on the summary page of the webapp. They compare the predicted optimization using the current pricing strategy “Current Markdown” and the prediction of whether all products are at their full price “No Markdown”.

Product Customization

If we want to improve a given product’s performance or change the optimization context, we have to go through the product optimization page.

We first select the products we want to modify. Then, we can filter and sort the available products based on several parameters.

We can then modify the product attributes (product cost, product reference price) or the constraints for the optimization (Markdown Range, the strategy, and the balance parameters). Once you have made your modifications, you can save and run a new optimization batch, and review the results on the next page.

Business Insights

The second tab of the dashboard presents users with pre-built charts for consulting the historical values of the KPIs across all stores if available. We also represent the best and worst performing products (in terms of uplift).

Reproducing these Processes With Minimal Effort For Your Own Data#

The intent of this project is to enable pricing teams to understand how Dataiku can be used to find the best discount for a product in various contexts.

By creating a singular solution that can benefit and influence the decisions of a variety of teams in a single retail organization, smarter and more holistic strategies can be designed in order to maximize profit and sales, while limiting the depth of discount and remaining stock.

We’ve provided one way of finding and defining the best markdown, but ultimately the “best” approach will depend on your specific needs and your data. If you’re interested in adapting this project to the specific goals and needs of your organization, roll-out, and customization services can be offered on demand.