What is Big Data?

Big data is a term that describes the large volume of data – both structured and unstructured beyond the ability of commonly used software tools to capture, curate, manage, and process data within a tolerable elapsed time.

It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.

Big data is characterized by 4Vs as shown in the figure.

Applications of Big Data

Companies use big data to better understand and target customers by bringing together data from their own transactions as well as social media data and even weather predictions.

Businesses optimize their processes by tracking and analyzing their supply chain delivery routes and combine that data with live traffic updates. Others use machine data to optimize the service cycles of their equipment and predict potential faults.

Big Data is used in healthcare to find new cures for cancer, to optimize treatment and even predict diseases before any physical symptoms appear.

Big Data is used to analyze and improve the performance of individuals (at sports, at home or work) where data from sensors in equipment and wearable devices can be combined with video analytics to get insights that traditionally were impossible to see.

Police forces and security agencies use big data to prevent cyber-attacks, detect credit card fraud, foil terrorism and even predict criminal activity.

Big Data is used to improve our homes, cities and countries by optimizing the heating or lighting in our homes, the traffic flow in our cities, or the energy grid across the country.

Application in agriculture and food 

Seed Selection – Big-data businesses can analyse varieties of seeds across numerous fields, soil types, and climates and select the best.

Crop disease – Similar to the way in which Google can identify flu outbreaks based on where web searches are originating, analysing crops across farms helps identify diseases that could ruin a potential harvest.

Irrigation – Precision agriculture aids farmers in tailored and effective water management, helping in production, improving economic efficiency and minimising waste and environmental impact.

Weather – Advanced analytics capabilities and agri-robotics such as aerial imagery, sensors help provide sophisticated local weather forecasts can help increasing global agricultural productivity over the next few decades.

Climate change –  Since, climate change and extreme weather events will demand proactive measures to adapt or develop resiliency, Big Data can bring in the right information to take informed decisions.

Food processing – They help in streamlining food processing value chains by finding the core determinants of process performance, and taking action to continually improve the accuracy, quality and yield of production. They also optimise production schedules based on supplier, customer, machine availability and cost constraints.

Loss control – In India, every year 21 million tons of wheat is lost, primarily due to scare cold-storage centres and refrigerated vehicles, poor transportation facilities and unreliable electricity supply. Big Data has the potential of systematisation of demand forecasting thus reducing such losses.

Pricing – A trading platform for agricultural commodities that links small-scale producers to retailers and bulk purchasers via mobile phone messaging can help send up-to-date market prices via an app or SMS and connect farmers with buyers, offering collective bargaining opportunities for small and marginal farmers.

Big Data in India

With a population of 1.2 billion, the relevance of Big Data becomes all the more pronounced for India.

Recently, NITI Aayog also echoed the idea of evidence-based policymaking guided by Big Data.

Discoms in India are using data from last mile sensors to implement measures of cutting down aggregated technical and commercial losses.

Challenges

• Inefficient infrastructure (data management centers) for data collection and management
• Constant evaluation of feedbacks generated from new data required: To use Big Data effectively for policymaking, the government must adopt a dynamic approach and be willing to be flexible regarding its policy structure and processes.
• Anonymization of data or invasion of privacy and Data security: Lack of proper virtual safety raise grave concerns as most of Data today is generated online.

Way Forward

In order to effectively analyze the large chunks of data available, the government must establish well-equipped data centres. It is essential to segregate the relevant data from the irrelevant.

It must strengthen its cybersecurity in order to make the large pool of data available virtually safe.

It must also address the ethical issues regarding big data analytics and formulate a policy regarding data privacy.