All companies and individuals are constantly creating data. According to a study by the Mapfre Foundation, every minute, the estimated 4.5 billion people with Internet access in the world, send almost 42 million WhatsApp messages, install the TikTok application 2,704 times, upload 500 hours of video to YouTube, join 319 new followers on Twitter, publish almost 348,000 posts on Instagram and upload more than 147,000 photographs to Facebook.
The world of transport engineering has been not different; we have a wealth of data from numerous sources, cameras, sensors, mobile phones, number plate reading devices, traffic lights, etc.
This data could not be analyzed in its entirety due to the number of resources it used, but thanks to the use of technologies such as Big Data, it has been possible to analyze it, not only to planning transport more efficiently, but also to generate mobility patterns more effectively or to establish predictive models necessary for traffic management.
All this data must be transformed into useful information to generate the value and to do so, we must follow the five main phases of any Big Data project.
1. Data import: Data is usually scattered in numerous sources, such as databases, files, ERPs. The Big Data specialist must identify the information to be included and the best methods to achieve it.
2. Exploratory analysis of the data: At this stage it is essential to clean the source data, eliminate nulls, normalize data, eliminate wrong information, or make transformations to the data with our goal in mind.This phase can represent 80% of the estimated time for this type of project, so take the necessary time, the result depends on it.
3. Analysis: In this phase we can use different strategies or tools to analyze the data:
3.1 Statistical modelling is to understand the data we have, for example looking for outliers that can distort the results or interpolate certain data to be able to “fill in” certain gaps in the initial data.
3.2 Artificial intelligence.
3.3 Data mining to detect possible patterns.
3.4 Machine learning
Each of these tools will allow us to solve a specific challenge, so it is important to select the right analysis according to our final objective.
4. Data visualization: Representing data in a visual and attractive way is a very efficient communication mechanism to the human mind that allows us to see large amounts of data.
For this we can use BI tools such as Tableau or Power BI, Qlik, or more complex tools such as Python’s matplotlib or Seaborn libraries.
5. Decision making: This is the goal to use data as an objective source to make decisions, or even identify new opportunities.
These phases should be understood as a continuous life cycle in which you can return to later phases as many times as necessary.
In short, to carry out the project successfully, we must carry out a complete study of the data, which will involve a compilation exercise of all the available data, a cleaning of the data to eliminate null data, or carry out the appropriate transformations, and select the appropriate visualizations to be able to transmit the appropriate information.
This process, which at first sight may seem long and tedious, entails a series of benefits that in the long run are very interesting for a company:
1.Decision-making is more agile, efficient, and objective.
2.It is possible to detect new sources of income or business that were previously unknown.
3.In-depth knowledge of customers, which enables customer loyalty or to offer a better user experience.
4.Elimination of information silos within the company.