DATA CLEANING

     Data cleaning is the process of fixing or removing incorrect, inaccurate, outdated information, and duplicates, called dirty datawithin a database or data set. Data cleaning, also called data cleansing or data scrubbing is believed to be a very important part of maintaining quality of the data. Without data cleaning, results are unreliable, especially in analyses. Cleaned data results in high data integrity, more saved memory space, better results, and saved money! Data cleaning can be done with a single datum if a cell is the “error,” or it can be completed in multiple data sets.

     In different software platforms, data cleaning methods may vary. Data cleansing may be easy to identify the need of if the data is formulated and the results show as “error” or easier, if there is conditional formatting used. It can also be more sophisticated like if there is a large data set to scroll through, only to find that one cell preventing necessary results! Datanami (2020) estimates that data scientists spend about 45% of their time cleaning data (https://www.datanami.com/2020/07/06/data-prep-still-dominates-data-scientists-time-survey-finds/). Failure to include data cleaning can cause adverse events like hindering the process, wasted memory, misinformation, possible misdiagnoses [in health care], and misspelled or mispronounced names. 

     Some things utilized by data, like pivot tables or data visualizations, require data cleansing to even be accessed, but they are more visibly understandable since not everyone has the patience to stare at numbers to find their meanings or differences. Data cleaning benefits companies that have a lot of clientele and revenue because it keeps account information updated and would make it easier to identify older, inactive accounts to remove. Data cleansing also benefits companies that have item numbers of their inventory in a database; it not only prevents items receiving the same item numbers, but also assists in maintaining the inventory, including the discontinued merchandise.

     

     There are different ways and strategies for data scrubbing or data cleaning, whichever term you prefer. Data cleaning can be done by correcting a single datum cell, it may require deleting a whole column of cells, or may even be as sophisticated as typing in a formula to copy to locate where the issue(s) are.  

Requirements for data cleaning:

  • Attention to detail;
  • Patience

     While data cleansing is time-consuming, the accuracy of the results can lead to more entrust from authorities for business decisions or even to bonuses! As previously mentioned, common dirty data may include duplicated data, outdated accounts, or typos or errors, but another kind of dirty data usually recommended to remove is outliers. Removing outliers while data cleaning can keep the numbers together that are close so forecasting or predicting from results would be more reliable and easier, especially if regression analysis is what is being completed. If the outliers are included, there may be a huge difference in what would be of normalcy compared to that of the outliers. For business decisions, cleaning data would include removing outliers, unless the company is going to use the outliers for root cause analysis.  “Was it seasonal?” “Was it during a special event or bad weather?” Questions like that may be asked if outliers are left in a database.

Conclusion

     Data cleaning improves the quality of the data which is why so much attention is needed and why so much time is spent on it. Too little time spent for data cleaning depletes the efforts put into acquiring all the data used since the results will be considered inaccurate. Would you rather take the extra time to ensure accuracy by cleaning data, or would you rather risk submitting something that lacks quality and credibility?