About MySales

Overview

MySales is an app that helps you:

Analyse your sales
Use different regression models for your product groups, items, and stores
Regress your sales to the influencing factors
Understand why your sales change, growing or dropping
Understand the impact of prices, discounts
Understand other influencing factors, including your competitor's prices, currency exchange rate
Build sales forecast based on different models
Plan your promotion campaigns better
Order goods based on sales forecast and forecast to sales deviation
Optimize retail prices based on statistics and competitor prices
Analyze the price image using the dynamics of price indices

In this documentation you can find a description of the methods and algorithms of MySales, as well as practical instructions and a user guide.

Machine learning algorithms

In MySales, there are several basic machine learning algorithms used for forecasting:

A set of regressions and a decision tree used to build a multi-factor forecast
Additive and multiplicative seasonality calculation algorithms
Neural networks for predicting promotional activities sales
Algorithm for evaluating the similarity and comparison of promotional items

It should be noted that almost all MySales algorithms are proprietary, since the use of well-known methods and libraries of machine learning does not provide a stable result in the conditions of a huge number (millions) of forecasting positions that differ significantly from each other. However, they have much in common with well-known algorithms.

The set of regressions is not limited by simple and multiple, linear and non-linear regressions. The gradient descent method is also used to build a forecast based on more factors than classical multiple regression allows. In addition, neural networks are used for both sales and promotional forecasting. As well as an algorithm which automatically evaluate the similarity of positions in promos, and other machine learning algorithms.

The key difference and know-how of all MySales algorithms is their predictable behavior under various conditions and a low tendency to the effect of retraining, which is also known as the “fit effect” or “over fit”. This is achieved due to the fact that machine learning in MySales is carried out within the framework of logical rules and restrictions set by both system parameters and individual settings for the customer.

Regression analysis

MySales uses both simple and multiple regression analysis to decompose your sales to the influencing factors like seasonality, trend, prices and discounts, stock, etc.

Description

In statistics, regression analysis is a statistical process for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables when the focus is on the relationship between a dependent variable and one or more independent variables. More specifically, regression analysis helps one understand how the typical value of the dependent variable (or 'criterion variable') changes when any one of the independent variables is varied, while the other independent variables are held fixed. Most commonly, regression analysis estimates the conditional expectation of the dependent variable given the independent variables – that is, the average value of the dependent variable when the independent variables are fixed. Less commonly, the focus is on a quantile or another location parameter of the conditional distribution of the dependent variable given the independent variables. In all cases, the estimation target is a function of the independent variables called the regression function. In a regression analysis, it is also of interest to characterize the variation of the dependent variable around the regression function which can be described by a probability distribution.

Applying

Regression analysis is widely used for forecasting, where its use has substantial overlap with the field of machine learning. Regression analysis is also used to understand which among the independent variables are related to the dependent variable, and to explore the forms of these relationships. In restricted circumstances, regression analysis can be used to infer causal relationships between the independent and dependent variables. However this can lead to illusions or false relationships, so caution is advisable; For example, correlation does not imply causation.

Efficiency

Regression analysis is widely used in many areas such as aerospace and defence, aviation, artillery, astronomy, for prediction and forecasting, or understanding impact of external/internal factors. The performance of regression analysis methods in practice depends on the form of the data generating process, and how it relates to the regression approach being used. Since the true form of the data-generating process is generally not known, regression analysis often depends to some extent on making assumptions about this process. These assumptions are sometimes testable if a sufficient quantity of data is available. Regression models for prediction are often useful even when the assumptions are moderately violated, although they may not perform optimally.

Neural networks

An artificial neural network is a mathematical model, as well as its software or hardware implementation, built on the principle of organization and functioning of biological neural networks - networks of nerve cells of a living organism. This concept arose in the study of processes occurring in the brain, and in an attempt to simulate these processes.

Neural networks are used in MySales both for building a multi-factor sales forecast, for those positions where it is important to determine the synergy effect of two or more factors, for example, seasonality and discounts, as well as for predicting promotional growth for those promotional positions for which it's impossible to find comparable and similar promo positions in the past.

The key advantage of neural networks over regression analysis methods in sales forecasting is precisely the ability to determine the effect of synergy. However, the key disadvantages of neural networks also follow from this same key advantage: increased tendency to retrain or over-fit, low transparency and poor interpretability of the results generated by the neural network.

The key difference between the proprietary algorithms of MySales neural networks is that these algorithms allow training the neural network within the framework of the rules, which makes the results generated by the neural network more understandable and predictable to the user.

Model

Model is the set of predictors, the original or calculated variables that impact your sales.

MySales chooses the best model for each product Group/Store, SKU/Store, Group/Whole network Group/Region, SKU/Whole network, SKU/Region level, also you can set your own models on these levels.

Choosing model

You can choose your own models for every product group, SKU, or even every product group/store level, or SKU/store level.

MySales already chooses the best model that fits your sales, and overriding this means you agree for a compromise to have less forecast accuracy for better forecast sensitivity for the specific predictors you include into your models.

And remember, including too many predictors into the model doesn't always mean a better forecast, the more smart approach is to include the most important predictors and to exclude the others, instead of including everything that can potentially impact.

Adding models

You can add more models for MySales to choose. The number of models is only limited by the performance of the calculations. But if you add a model, then check how many items are covered by this model. If MySales doesn't choose this model for any item, then it means the model doesn't make the forecast any better.

To add a model, ask your MySales administrator to do this, as it currently requires editing 'options.js' file in the application root directory. Always keep a backup copy of 'options.js' file when adding models.

Predictors

A predictor is an independent variable loaded into MySales from the source system, or calculated for a specific model.

Typical predictors in retail are:

Seasonality – calculated based on sales history
Trend – calculated based on non-seasonal sales
Price, discount – loaded from the source system
Regressed price, discount – calculated using non-linear regressions based on price
Price change – percent of price change in a specific week to an average price of the last several weeks
Item count – count of items in a product group
Stock – loaded from the source system
Rate – currency exchange rate, downloaded from an external source
Rate change – percent of currency exchange rate change in a specific week to an average exchange rate for the last several weeks
Volume share – a share of a specific product in a group sales, percentage

Default predictor set

The system automatically sets a default set of predictors. This set can be modified according to customer needs.

The following predictors are default:

Seasonality (seas) - calculated based on sales history (additive seasonality);
Base (avg) - base level of sales (as average sales for a period);
Trend (inc) - calculated based on non-seasonal sales;
Stock (stock) - loaded from DB;
Price (price) - loaded from DB;
Actual price (a_price) - calculated as price after discount;
Discount (disc) - loaded from DB (numerical value);
Discount in % (disc_p) - percentage discount calculated on the basis of price and discount;
TRX (trx) - loaded from DB;
Price change (price_chg) - percent of price change in a specific week to an average price of the last several weeks;
Group sales (parent_group) - loaded from DB;
Region sales (parent_reg) - loaded from DB;
Number of SKUs in group (cnt) - the number of SKUs in the group at the Group level;
Number of stores in region (stores) - the number of stores at the Region level;
Multiplicative seasonality (seas_mult) - calculated based on sales history;
The presence of SKU in the assortment (assrt) - takes the value 0 if the product is not in the assortment matrix and 1 if in the assortment matrix;
Season type (seas_type) - a value of 0 for the normal season, -1 for the low season and 1 for the high;
Promo ratio (promo_ratio) - percentage of days per week for which a promo was held;
Availability (avail) - the value of the availability of goods in stores (takes the value 0 or 1);
Rate (rate) - currency exchange rate, downloaded from an external source;
Rate change (rate_chg) - percent of currency exchange rate change in a specific week to an average exchange rate for the last several weeks;
Average group price (grp_price) - calculated as the average price for all products in the group;
Average group discount (grp_disc) - calculated as the average discount for all products in the group;
Product price to group price ratio (grp_price_r) - calculated as the ratio product price to the average price in the group ratio;
Number of SKUs in group (grp_cnt) the number of SKUs in the group at the SKU level;
Temperature (tempm) - average weekly air temperature from an external source;
Temperature change (temp_chg) - the difference between the average temperature for a particular week and the average temperature for the last 13 weeks;
Rain (rain) - proportion of raining days of the week;
Snow (snow) - proportion of snowy days of the week;

Choosing predictors

MySales checks the predictors you've included into the model for every product group and SKU, in case the impact of any particular predictor differs from the expected, or multicollinearity detected between predictors, MySales removes this predictor from the model automatically.

To avoid this, after you have fixed the set of predictors in the model, recalculate the forecast for this group or SKU and check whether MySales still uses the original set of predictors you've chosen, and the forecast is still accurate.

In general, you should remember the following rules while choosing the predictors for your models:

Avoid including correlated predictors into the model, e.g. don't include 'Discount' and 'Discount %' as they have a strong correlation
Avoid including the predictors with non-linear (exponential, logarithmic, power) impact to your sales for a specific SKU or product group, instead include their regressed values, e.g. if the price has non-linear, but the exponential impact to your sales, include 'Regr price' instead of 'Price' predictor.
Exclude the predictors that do not impact your sales or their impact is too low, this will improve the model accuracy

Adding predictors

You can add more predictors, but this requires more work than just adding more models based on the combinations of already existing predictors. In order to add a predictor, first, you should prepare the data in the source system for download, add it into your 'sql.js' file and set it up in 'options.js' file. Often it also requires some coding to convert the predictor into the appropriate form of data.

Cannibalization

The cannibalization effect is one of the most pressing analytical tasks in retail. The concept of cannibalization appeared in the 1970s, and meant the effect of a decrease in demand for one product as a result of complementing the assortment with another product. This usually happens when a product is introduced that easily replaces other products. For example, when introducing into the assortment a new position of eggs that are cheaper than all the others, you can see how much sales of all old eggs from the assortment drops. Cannibalization occurs when a new product is similar (or a substitute) and has the same target audience.

The predictor of cannibalization is very difficult to consider in the forecast. And user should do it in specific cases. The forecast of cannibalization is based on three predictors - the average price of the group, the number of positions in the group and the price of a position to the average price of the group ratio. The cannibalization predictor can be used only if all the predictors on the basis of which its forecast is built are well forecasted. In other case, cannibalization forecast can be inaccurate.

You also need to understand that the more accurate the forecast, the more prone to errors. Therefore, even if using cannibalization in the forecast gives better accuracy, sometimes it is better not to take it into account. The forecast will become less accurate, but the safety stock will rise.

There is a sense to use cannibalization in the forecast in two cases:

If the groups have a lot of similar positions and there is a clear understanding of the cannibalization effect presence;
If you conduct the assortment management using assortment matrices (but this can only be done with a good forecast for the average price of the group and the number of positions).

In any case, using cannibalization, the user must understand that it is impossible to fully take into account cannibalization and use of it will only be beneficial in the case of a visible improvement in the forecast.

is also taken into account in price optimization module in cross-elasticity. In the price optimization module, can see the dependencies between specific products. Cross-elasticity is described in detail in the Cross-elasticity section.

Using a Key Element in a Forecast

The system can take into account the effect of cannibalization of interchangeable and similarly priced positions in a direct way.

To do this, it is possible to combine several items that are interchangeable and close in price into one key element (KE). This join is done by adding the KEY_ID field to the PRODUCTS table where the list of SKUs is loaded. Also, it is possible to set up maintaining a directory of key elements in the MySales DB Editor. If several positions have the same key element (usually, this is the SKU_ID of the earliest position), then the forecast is built as follows:

Initially, the system consolidates sales and inventory data for all key element (KE) positions
Next, a forecast is built for the KE, i.e. forecast based on consolidated data
For those weeks when each item of the KE was included in the assortment matrix (AM), or if the item was withdrawn from the AM, before the forecast week of the end of the balance, the system builds an independent forecast
Further, the forecast of KE positions is adjusted so that the sum of the forecasts of all KE positions is equal to the total forecast for the KE. Thus, the KE forecast is redistributed to the forecast of each CI position.
Those weeks where the “Daily adjustment” predictor worked are not adjusted
Thus, if, for example, there were 3 KE positions in AM and sales were distributed between them, and one of the positions is withdrawn from AM, then the forecast of the remaining two will be increased. And vice versa, if there were 2 KE positions in AM, and a third one is entered, then the forecast of the previous two will be reduced.

Note 1: combining KE positions does not cancel the recommendation to put down an analogue. If there is no analogue for the new KE position, then the forecast for it will be built as an average for the group, adjusted for the price.

Note 2: It is recommended to test the forecast for positions combined by one KE, since such a combination may not always make a better forecast. For example, even if positions are close in price and consumer purpose, but differ in consumer preferences and each of them has its own target audience, i.e. in fact, one position cannibalizes the other only partially.

Price elasticity

Price elasticity of demand - this is a measure of the change in the product demand, which is caused by a change in the price of the product.

The price elasticity of demand shows how much the percentage of demand for a particular product will change when its price changes by 1%.

Demand can be elastic and inelastic:

For elastic demand when the price of a product changes by 1%, the demand for it will change by more than 1%. For example, when bananas go up by 10%, the demand for them falls by 15% or when the price decreases by 25%, the demand increases by 40%, so the demand for this product will be elastic in price.
For inelastic demand when the price of a product changes by 1%, the demand for it will change by less than 1%. For example, if all varieties of oil rise in price by 49%, then demand decreases by only 19%, so we can conclude that the demand for oil is inelastic in price.

Information on the elasticity or inelasticity of demand is very important for entrepreneurs.

If the demand for the product is elastic, as in the considered case with bananas, then the following scheme can be drawn up: if P (price) grows, then Q (quantity of products sold) decreases more than price increases, or if P decreases, then Q increases more than the price drops.

If the demand for the product is inelastic, as is the case with oil, then if P decreases, then Q grows less than the price decreases, or if P grows, then Q decreases less than the price increases.

Therefore, in order to increase the income from the sale of bananas, it is necessary to lower their prices, and for oil, on the contrary, to increase them.

MySales automatically calculates the effect of changes in product prices on demand. And it calculates theoretically optimal prices at which sales in pieces, money or margin will be maximum.

To calculate theoretically optimal prices using price elasticity, the module "Price Optimization" has been designed (for more details, see the section Price Optimization).

Forecasting new items

New items are products that have been recently added to the assortment and have no sales history.

Our system automatically detects new positions and builds a forecast based on them. The system makes a different forecast for the following types of new positions:

"New items with analogues;"
The system calculates their sales forecast based on sales data for similar products. When building a forecast, the system takes into account the coefficient of influence of each of the analogues, which is specified when adding a new position. (details of adding analogs are described in the section New).
"New items which analogs are not specified or which have no sales history or analogs at all;"
In this case system automatically automatically finds the SKUs, which are in the assortment matrix, have less than 4 weeks of sales history and there are no analogs of them. Then algorithm forecasts them as an average position for the group, taking into account the price of the new position (if it is present if not, the forecast is calculated using the average value of the group). For this it takes sales of Group/Store level and divides them by the number of positions on the stock, and then recounts the group's forecast, taking into account the price of the new position, and then applies it to a new position.
"New positions for which analogues are not specified, but there is a history of sales over the regions;"
For forecasting this positions system uses "Expansion of distribution" algorithm. This option is used to predict a new position in the store more accurately, If new position has already been used in other region's stores assortment, then this option can be used.
Expansion of the distribution is used if the number of weeks of historical sales for a new position in the store is less than a previously set number of weeks (the number of weeks is set using the forecast configuration file), but there are sales in the region for a new position. In this case, the system does not consider this position as completely new, but as an extension of the store's assortment.
For positions that fall under the expansion of distribution, the system calculates the new position forecast as a forecast for the position from the region and applies it to the store level using the following proportion: Sales/Forecast value on SKU/Region level is multiplied by Sales/Forecast on Group/Store and divided by Sales/Forecast from Group/Region level.

How MySales works

As an input MySales downloads your historical weekly sales per group and SKU (stock keeping unit), stock data, and additional predictors (influencing factors, variables) you set.

MySales performs calculations of more variables to decompose your sales:

Additive seasonality based on moving averages,
Trends based on linear and non-linear regressions for non-seasonal sales
Linear and non-linear regression on prices, discounts, and other optional predictors you set up
Linear (or optionally non-linear) regression for a count of items in product groups

Then MySales puts these variables into different regression models you can see on SETTINGS page, tests each model and chooses the model that fits best to your sales.

Fix models

There is a parameter 'Fix Models' in the section '10. Processing Parameters'. Currently, its value set to undefined. It means that MySales will only fix the selected model if the available history is greater than or equal to undefined weeks, or if you fix it manually. If not enough available history, MySales will select the best model every time.

You can observe this parameter on SETTINGS page.

Forecast evaluation to choose the best model

When testing each model, MySales builds the forecast for the past 52 weeks, and calculates Mean Absolute Percentage Error (MAPE) for the forecast on each model. Then based on your model settings, MySales adds the penalty for each model and chooses the model with minimum error including Penalty. The penalty is a percentage that you can use to reduce sensitivity for a particular model.

You can see Penalty percentage for each model on SETTINGS page. To see the tested models and the selected models, use Models button on MY SALES page.

Using past undefined weeks to evaluate the forecast

This is a parameter and it can be observed on SETTINGS page, section '6. Other Forecast Parameters'.

This parameter can be decreased to 3-6 weeks when building a short-term forecast only and increased to 9-13 weeks for a mid– or long-term forecast.

How to order goods based on the forecast

There are multiple options, but the more generic solution is to take the forecasted quantity for the delivery period (from the nearest delivery day to the next delivery day), add Mean Absolute Deviation (MAD) or Mean Absolute Percentage Error (MAPE) as a safety stock. Your target stock quantity is ready, then just subtract the current stock and your quantity to order is ready. If the forecast accuracy is high, using MAPE is preferred when you need to increase the safety stock for the forecasted peaks. Using MAD is preferred when the forecast accuracy is less, or sales are low, or the sales expected to be more smooth without high peaks.

Examples

Here is an example 1:

You order goods for 1 week, every Monday
The forecast for the next week is 10 pieces
MAPE is 20%, MAD is 1.2 peaces
Your order for the next week is 12 peaces (10 + 20%) minus the current stock

Here is an example 2:

You order goods for 1 week, every Monday
Your forecast for the next week is 0.5 peaces
MAPE is 50%, MAD is 0.3 peaces
Your order for the next week is 1 peace ( round(0.5 + 0.3) ) minus the current stock