BIG DATA and HADOOP AbstractIn thisworld of information the term Big Data has emerged with new opportunities andchallenges to deal with the massive amount of data. Big Data has earned a placeof great importance and is becoming the choice for new researches. Big Datadescribes techniques and technologies to store, distribute, manage largesized datasets with high-velocity and different structures. Big data can be structured, unstructured orsemi-structured, resulting in incapability of conventional data managementmethods. The volume and the diversity of data it is generated withspeed, makes it difficult for the present computing infrastructure to manageBig Data. Traditional data management, warehousing and analysis systems fallshort of tools to analyze this data.
Due to its specific nature of Big Data, itis stored in distributed file system architectures. Hadoop is widely used forstoring and managing Big Data. Hadoop is the core platform for structuring BigData, and solves the problem of making it useful for analytic purposes. Hadoopis an open source software project that enables the distributed processing oflarge data sets across clusters of commodity servers.IntroductionBIG DATA is a vague topic and there is no exact definition which is followed by everyone. Data that has large Volume, comes from Varietyof sources, Variety of formats and comes at us with a great Velocity isnormally refer to as Big Data.
All this data is coming from smartphones, socialnetworks, trading platforms, machines, and other sources. One of the largesttechnological challenges in software systems research today is to providemechanisms for storage, manipulation, and information retrieval on largeamounts of data. Web services and social media produce together an impressiveamount of data, reaching the scale of petabytes daily. To harness the power of big data, youwould require an infrastructure that can manage and process huge volumes ofstructured and unstructured data in real time. 50% to80% of big data work is converting and cleaning the information so that issearchable and sortable. Only a few thousand experts on our planet fullyknow how to do this data cleanup. These experts also need very specializedtools, like Hadoop, to dotheir craft 1.
There are two ingredients that are driving organizations intoinvestigating Hadoop. One is a lot ofdata, generally larger than 10 Terabytes. The other is high calculationcomplexity, like statistical simulations. Any combination of those twoingredients with the need to get results faster and cheaper will drive yourreturn on investment. Big data Right now we have some problemsthat is called Big Data so what is that problem that we have big data isnothing but simple a huge data that we are putting together that is called BigData. As you know on this world of internet enableddevices we are generating a lot of data and we are getting data from different data sources so existingsystems are not able to handle this Big Data .Big data is having two issues thedata is coming from the different sources continuously so the data we are notable to handle it using our existing computational techniques so what are thebig data related issues, we are getting the tons of data from different devicesand we are not able to store the data on time and the second thing isprocessing work somehow we are able to manage the data how to store the data byadding different and many servers to our system we are able to manage somehowbut in many times when it comes to process the data we are not able to do it ontime.
The 3 Vs that define Big Data are Volume, Velocityand Variety :- volumeof data – volume of data is the amount of data that we are adding fromdifferent data sources is called volume of data example probably 100 byteshundred megabytes or 100 terabytes or 1 petabytes – velocity of data – Velocity is the speed at which data is generated and processed 3. – variety of data – what kind of data that we are gettingfrom the different data sources tothe warehouse or what type of data that weare trying to add from differentdata sources to the warehouse HadoopHadoop is fundamentally infrastructure software for storing andprocessing large datasets it’s an open source project under Apache. Hadoop isinfluenced by Google’s architecture, Google File System and MapReduce. Hadoopprocesses the large data sets in a distributed computing environment 3.
Hadoop is an open source frameworkwhich is written in Java completely it does not mean that you have to know Javain order to use Hadoop. To understand Hadoop you have to understand two fundamentalthings about it you have to understand one how it stores files how it storesdata and two how it processes data. Hadoop also can store both structured andunstructured data because it fundamentally is just a file system now. Hadooparchitecture Hadoop works based on master/slavearchitecture so what I mean by master/slave architecture you have one mastercomputer that computer is going to take care of all slave computers within thenetwork topology whenever you have more than one computer together within thenetwork where they can talk to each other we call it as a cluster.Whenever you are going to process terabytes ofdata using single computer it’s going to take a lot of time to find out your endresults and it is going to take long time to process.My entire terabytes of data for example if iget my data from data source called amazon.com and i put the data onto my clusterthen what I am going to do instead of storing this entire data together myHadoop cluster master computer is the one who is going to divide this data intochunks of data instead of storing my entire data into one machine my mastercomputer is the one who is going to be in touch with some of the configuration filesand it’s going to come up with the mechanism saying that instead of storing myentire terabytes of data into single machine I am going to divide into small soinstead of just storing the entire data into the one computer I am going todivide into pieces and that request will go to the master computer and master computer is the one who is going to beallocating each slice that has to go into what machine . The way we distribute the data and divide the data intomultiple computers that concept we call it as a HDFS (Hadoop distributed filesystem).
Whenever I want to increase thestorage capacity simply I am going to add one more computer to my cluster thatis possible that is how we address storage problem.With the processing way the programthat we are going write to the master computer. Master computer is the one whois going to send the same kind of logic to each computer wherever I have storedthe data wherever each slice is that program will be sent and that will beprocessed there. This is different techniques or paradigms that we use fromHadoop methodology so we are not going to pull all of the entire data togetherto process in a single computer instead of pulling the data all together andprocessing your data in a single machine, Hadoop methodology it is going tosend the program to individual machines. The broken-down file however wedivided our file into chunks of data on each sync and we can go ahead andprocess the data the way we are doing it is called MapReduce.
Instead of processingthe ten terabytes of data out of single computer we are going to process thedata in individual computers. If you want more storage capacity we add one computerto the Hadoop cluster one slave computer, if you want more processing capacity weadd one more computer to the requester. HDFS (Hadoopdistributed file system) HDFS as part of Hadoop that standsfor Hadoop distributed file Hadoop lets you store files bigger than what can bestored on one particular node or particular server or a computer so you canstore very large files, imagine your same PC could only store 50,000 files butyou had a million files so with the help of Hadoop you can store them.
HDFS system supports parallelreading and processing of data it supports read, write, rename operations itdoesn’t support random write operations the other key important thingwith HDFS is it is fault tolerant and hence easy to manage the data hasbuilt-in redundancy typically multiple replicas of the data is kept in thesystem and it tolerates disk and node failures. The cluster manages addition and removal ofnodes automatically without requiring any operational Intervention,one operator can support up to 3,000 nodes in a cluster and that’s a lot of nodessupported by a single person. In HDFS files are broken into blocks and these blocksare typically large and they are of the size 128 megabytes the blocks are storedas files on the data nodes on the local storage after data nodes the blocks arereplicated for reliability so typically the block replication factor is threein HDFS cluster there is a node called name node that manages the file system,namespace are directories and files of the file system ,the name node alsomanages the mapping of file to theblocks that belong to it it keeps the entire namespace in memory and and thisis what makes name CIMMYT simpler design and any changes to the namespace suchas addition our files deletion of files are general and they are stored onpersistently on a disk name node also periodically checkpoints Mapreduce MapReduce is the execution engineof Hadoop its duty is to get the jobs executed.
There are two main components of MapReduce job tracker and thetask tracker.The job tracker is hosted insidethe master computer and it receives the job execution request from the clientits main duties are to break down the received job that is big computations insmall parts allocate the partial computations that is tasks to the slave nodesmonitoring the progress and report of task execution from the slave the unit ofexecution is job as tracker this is the MapReduce component on the slavemachine as there are multiple slave machines many task trackers are availablein a cluster its duty is to perform computation given by job tracker on thedata available on the slave machine the task tracker will communicate theprogress and report the results to the job tracker/ how all these variouscomponents are located and activated in the nodes the master node contains thejob tracker and name node whereas all slaves contain the task tracker and datanode Advantagesof Hadoop for the companies Probably you might be thinkinghere instead of just using or having one computer here I use differentcomputers so obviously instead of buying one machine here I will buy more thanone computer here that is the obvious question that we’ll be getting in yourmind. There is a more expensive for the organizationof Hadoop features.
If we use existing legacy systems techniquesthe server should be always up and running if the server is going down the datathat was there will be inaccessible that is the reason we have to use always theenterprise hardware.Enterprise Hardware server arehighly reliable hardware they will not go down very frequently so if we want tohave the higher level hardware you have to spend a lot of money but Hadoop frameworkit can store and process the data using the commodity hardware. Commodityhardware server is cheap hardware in terms of price if you want to buy oneterabytes of hard drive in commodity Hardware you may have to spend very little,whereas if you are going with the enterprise hardware in order to buy one terabyteof hard drive space you have to spend thousands of dollars because this is avery high level hardware that is what you have to use for legacy systems in orderto make sure they are up and running in our production. Then how about if one of the machinesis going down how I am going to recover the data? Because of my Hadoopframework the nature itself it says is a commodity hardware the hardware metalitself is a commodity hardware is a cheap hardware that it may go down at anytime then how we are going to recover the data. Using the concept called faulttolerance using the concept called replication factor we are going to make sureour data availability, we are going to increase the data availability using ourreplication factor Whenever your organization is tryingto adopt any methodology or a framework or any technology what they are goingto do first and foremost thing how much money do I need to spend in order to maintainthis one so if you are going with Hadoop you don’t have to pay too much for thesoftware as well as we don’t have to buy very high reliable machines you can goand buy the normal machines whatever we are using generally in our day to day activitiesyou can just go ahead and use them and the data that you are getting from yourdifferent sources you can go ahead and store the data in Hadoop framework.Hadoop framework is going to likethe data from different data source it can be satellite data it can be the sensordata it can be trucks data it can be servers data whatever you take it can easilycollect the data and put it into that database. Conclusion Over the long run, Hadoop willbecome part of our day-to-day information architecture. We will start to see Hadoopplaying a central role in statistical analysis, ETL processing, and businessintelligence.