Using Big Data to Predict the Future

Microsoft recently announced that it is using big data to try to predict traffic jams up to an hour before they happen. The plan is to use Microsoft's expansive cloud computing capabilities to store and crunch data from multiple sources in "real time."

"The Traffic Prediction Project uses data available by social networks, department of transportation, and data that the users create themselves while they move around the city," Juliana Salles of Microsoft Research said in a promotional video.

According to Peter Keen of Digital Traffic Systems in an interview with Marketplace Tech, those data points aren't new and neither is work to predict traffic jams. 

"It is being done now. It can always be done better," Keen says, because right now prediction models depend on past traffic information.

"You have historical trends of what the volume's going to be at a given day of the week, at a given time of day," says Keen, which can help make predictions that are accurate much of the time, especially about typical traffic patterns on major passageways.

Keen says predictions can be better if both past and current data is combined. But such data is often in multiple formats and multiple databases, and hard to combine.

That is exactly what the Traffic Prediction Project is trying to overcome. Microsoft, in collaboration with researchers at a university in Brazil, is studying whether it's possible to predict traffic congestion 15 minutes to an hour before it happens. Microsoft will be putting its Azure cloud-computing platform to the test for the project because of the immense processing power needed to crunch multiple terabytes of data, including historical traffic numbers, data from transport departments, road cameras, and Microsoft’s Bing traffic maps – and even drivers’ social networks – to help foresee traffic jams.

Beyond the convenience of knowing when the George Washington will back up before you are actually stuck on it, the implications of predicting human behavior using big data for small business are enormous. What if you could combine what your customers historically purchase during a certain month with real-time data about what they are purchasing right now and what they are saying about what they want on social media, so you can give them what they will want tomorrow? What if using big data to predict the performance of your employees was possible? If this Microsoft project works (it already reports 80% accuracy), and it can be delivered to small business cost effectively, the question is not if, but when historical and real-time data can be used to help small businesses predict the future.