Sparse data can impact the effectiveness of machine learning models. As students and experts alike experiment with diverse datasets, sparse data poses a challenge. The Leeds Master’s in Business Analytics teaches students how to manage different types of data and use the learnings from models to make business decisions. In order to make the best decisions from data, the predictive models used must be accurate, which means that learning how to handle sparse data is critical.
What is sparse data?
Sparsity occurs in matrices where most of the cells in the matrix are empty. Here’s a simple example. You asked every person who came into a movie theater one day to rate each movie they saw on a scale from 1 to 5 or to mark a zero if they hadn’t seen the movie. You would expect to have a data set with lots of empties because most people only saw and rated one movie at the theater. Because of the high number of missing data in the dataset, this would be considered a sparse dataset.
This sort of sparse data often occurs in scenarios when zero or a null (a word that denotes a “vacuum”) is used to represent information in the dataset. We simply do not know what they would think of the movies they have not seen.
This also demonstrates the difference between sparse and missing data. Sparse data is still representing something within the variables. Missing data, however, means that the data points are unknown.
Challenges in machine learning with sparse data
There are several problems in using sparse data to train a machine learning model. If the data is too sparse, it can increase the complexity of the machine learning model. This means that the model will take up more storage space and take longer to process.
Beyond this, there are several other challenges that appear with sparse data. First is lack of representativeness. Sparse data might not accurately represent the distribution of the data, which can lead to worse performance in the trained model. Overfittting is another issue. A model trained on sparse data is more likely to overfit to the limited data. This means that the model will struggle to generalize new data when given new data.
Likewise, the model can behave in unexpected ways when provided a sparse dataset. Certain models might underestimate the importance of sparse variables and give preference to denser variables, in other words, variables with more non-zero datapoints. This can happen even when the sparse data might be more predictive.
Best machine learning model for sparse data
To help combat these issues that arise with sparse data machine learning, there are a few things to do.
First, because of the noise in the model, it’s important to limit variables with sparse data before “training” the training set. Second, it’s helpful to find ways to make it denser. For example, using principal component analysis or feature hashing. Both of these practices help to remove unnecessary variables in the training set to help the model perform better.
Most importantly, it helps to pick a machine learning model that will work better with sparse data. Linear regression and tree-based models are both at risk for needing more memory space and taking longer to process. Linear regression runs into this issue because it needs to fit more coefficients, while tree-based models need a greater depth to incorporate all of the variables.
Tree-based models also struggle to equally weigh denser data with sparser data. This means that a tree-based model might leave out a highly-predictive variable because it gives preference to the denser variables.
If tree-based models and linear regression struggle, then what is the right fit? While no one model is designed to be the fix for sparse data, studies suggest that there are a few models that are performing well with limited data.
For example an entropy-weighted k-means algorithm performed better with sparse data than its standard k-means counterpart. It does this by weighting different variables to ensure that the most predictive aren’t excluded because of being sparse. Ensemble SINDy works by reducing noise in the data to prevent overfitting. And the Lasso algorithm has been shown to better apply sparse datasets to real world samples because it is able to work through which variables are impacting each other within the model.
As machine learning evolves, more variations of machine learning models will be developed and tested on sparse data. Students and professionals using the skills learned in a master’s in business analytics can help to discover and refine these models, and ultimately use them to inform critical business decisions.