{"id":48040,"date":"2018-01-28T11:21:23","date_gmt":"2018-01-28T05:51:23","guid":{"rendered":"http:\/\/blog.odango.com\/?p=48040"},"modified":"2023-02-04T09:31:02","modified_gmt":"2023-02-04T04:01:02","slug":"introduction-to-mapreduce-in-big-data","status":"publish","type":"post","link":"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/","title":{"rendered":"Introduction To MapReduce in Big Data"},"content":{"rendered":"\n[et_pb_section bb_built=”1″ next_background_color=”#000000″][et_pb_row][et_pb_column type=”4_4″][et_pb_post_title _builder_version=”3.0.106″ title=”on” meta=”off” author=”on” date=”on” categories=”on” comments=”on” featured_image=”off” featured_placement=”below” text_color=”dark” text_background=”off” title_font=”Titillium Web||||||||” title_font_size=”47″ title_font_size_tablet=”40″ title_font_size_phone=”35″ title_font_size_last_edited=”on|desktop” title_text_color=”rgba(0,0,0,0.8)” title_text_align=”left” text_shadow_horizontal_length=”0.08em” text_shadow_vertical_length=”0.08em” text_shadow_blur_strength=”0.08em” title_text_shadow_horizontal_length=”0.08em” title_text_shadow_vertical_length=”0.08em” custom_margin=”|||10%” \/][\/et_pb_column][\/et_pb_row][et_pb_row][et_pb_column type=”4_4″][et_pb_text _builder_version=”3.12.2″ text_font=”Titillium Web|300|||||||” text_font_size=”20″ text_font_size_last_edited=”on|desktop” text_text_color=”rgba(0,0,0,0.8)” text_line_height=”1.6em” max_width=”800px” custom_margin=”|||10%” text_line_height_last_edited=”off|phone” max_width_last_edited=”off|phone” custom_margin_last_edited=”off|desktop”]\r\n\r\n

\r\nMapReduce is a Programming pattern for distributed computing based on java<\/a>.<\/p>\r\n

MapReduce algorithm has two main jobs:<\/p>\r\n

1) Map<\/p>\r\n

2) Reduce.<\/p>\r\n

In Map method, it uses a set of data and converts it into a different set of data, where individual elements are broken down into tuples (key\/value pairs).<\/p>\r\n

The second method is Reduce task, it gets the input data from the map, (means output of map is input to reduce).<\/p>\r\n

It joins certain data tuples into a smaller set of tuples.<\/p>\r\n

MapReduce\u2019s<\/a> main advantage is easy to scale data processing over multiple computing nodes.<\/p>\r\n\r\n[\/et_pb_text][et_pb_image _builder_version=”3.17.6″ src=”https:\/\/asha24.net\/blog\/\/wp-content\/uploads\/2018\/03\/download-117.png” align=”center” \/][et_pb_text _builder_version=”3.12.2″ text_font=”Titillium Web|300|||||||” text_font_size=”20″ text_font_size_last_edited=”on|desktop” text_text_color=”rgba(0,0,0,0.8)” text_line_height=”1.6em” max_width=”800px” custom_margin=”|||10%” text_line_height_last_edited=”off|phone” max_width_last_edited=”off|phone” custom_margin_last_edited=”off|desktop”]\r\n\r\n

\r\n\u2022\u00a0<\/strong>\u00a0<\/strong>\u00a0Input Phase \u2212<\/strong>\u00a0Here we have a Record Reader that translates each record in an input file and sends the parsed data to the mapper in the form of key-value pairs.<\/p>\r\n

\u2022\u00a0\u00a0<\/strong>\u00a0\u00a0Map \u2212<\/strong>\u00a0Map is a user-defined function, which uses a series of key-value pairs and processes each one of them to generate zero or more key-value pairs.<\/p>\r\n

\u2022\u00a0\u00a0<\/strong>\u00a0\u00a0Intermediate Keys \u2212<\/strong>\u00a0the key-value pairs generated by the mapper are known as intermediate keys.<\/p>\r\n

\u2022\u00a0\u00a0<\/strong>\u00a0\u00a0Combiner \u2212\u00a0<\/strong>A combiner is a type of local Reducer that groups similar data from the map phase into identifiable sets. It takes the intermediate keys from the mapper as input and applies a user-defined code to aggregate the values in a small scope of one mapper. It is not a part of the main MapReduce algorithm; it is optional.<\/p>\r\n

\u2022\u00a0\u00a0<\/strong>\u00a0\u00a0Shuffle and Sort \u2212<\/strong>\u00a0the Reducer task starts with the Shuffle and Sort step. It downloads the grouped key-value pairs onto the local machine, where the Reducer is running. The single key-value sets are sorted by key toward a larger data list. The data list groups the equal keys together so that their values can be iterated technical terms in the Reducer task.<\/p>\r\n

\u2022\u00a0\u00a0<\/strong>\u00a0\u00a0Reducer \u2212<\/strong>\u00a0The Reducer takes the grouped key-value joined data as input and runs a Reducer function on each one of them. Here, data will be aggregated, filtered, and blended in a several ways, and it needs a wide range of processing. Once the execution is finished, it gives zero or more key-value sets to the final step.<\/p>\r\n

\u2022\u00a0\u00a0\u00a0 Output Phase \u2212<\/strong>\u00a0In the output phase, we have an output format that sends the final key-value pairs from the Reducer function and writes them to a file using a record writer.<\/p>\r\n

Algorithm:<\/strong><\/p>\r\n

The MapReduce algorithm<\/a> having two important tasks, namely Map and Reduce.<\/p>\r\n

\u2022\u00a0<\/strong>\u00a0\u00a0 The map task is done by Mapper Class<\/p>\r\n

\u2022\u00a0<\/strong>\u00a0\u00a0 The reduce task is done by Reducer Class.<\/p>\r\n

Mapper class takes the input information, tokenizes it, maps and sorts it.<\/p>\r\n

The output of Mapper class is used as input to Reducer class, which searches matching pairs and decreases them.<\/p>\r\n

MapReduce implements several arithmetical algorithms to divide a task into little parts and assign them to multiple systems.<\/p>\r\n

In technical terms, MapReduce algorithm assists in transferring the Map & Reduce tasks<\/a> to appropriate servers in a cluster.<\/p>\r\n

These arithmetical algorithms may include the following \u2212<\/p>\r\n

\u2022\u00a0\u00a0<\/strong>\u00a0Searching.<\/p>\r\n

\u2022\u00a0<\/strong>\u00a0 Sorting.<\/p>\r\n

\u2022\u00a0\u00a0\u00a0<\/strong>\u00a0Indexing.<\/p>\r\n

\u2022\u00a0\u00a0<\/strong>\u00a0 TF-IDF.<\/p>\r\n

Sorting:<\/strong><\/p>\r\n

Sorting is one of the primary MapReduce algorithms to operate and analyze data<\/a>.<\/p>\r\n

Sorting methods are performed in the mapper class itself.<\/p>\r\n

In the Shuffle and Sort stage, after tokenizing the values in the mapper class,<\/p>\r\n

The Context class gets the matching valued keys as a collection.<\/p>\r\n

To collect similar key-value pairs, the Mapper class takes the help of Raw Comparator class to order the key-value pairs.<\/p>\r\n

The set of intermediate key-value pairs for a given Reducer is automatically sorted by Hadoop to form key-values (K2, {V2, V2,})<\/p>\r\n

Before they are presented with the Reducer.<\/p>\r\n

Searching:<\/strong><\/p>\r\n

Searching performs a significant task in MapReduce algorithm. It supports in the combiner phase and in the Reducer phase.<\/p>\r\n

Indexing:<\/strong><\/p>\r\n

Indexing is utilized to point to a particular data and its address. It does batch indexing on the input files for a particular Mapper.<\/p>\r\n

The indexing technique that is commonly used in MapReduce is known as an inverted index. Search engines like Google and Bing utilize inverted indexing technique.<\/p>\r\n

TF-IDF:<\/strong><\/p>\r\n

TF-IDF is a document processing algorithm which is brief for Term Frequency \u2212 Inverse Document Frequency.<\/p>\r\n

It is one of the traditional web analysis algorithms. Here, the term ‘frequency’ refers to the no: of times a term arrives in a document.<\/p>\r\n

Term Frequency (TF):<\/strong><\/p>\r\n

It estimates how frequently a particular term happens in a document.<\/p>\r\n

It is measured by the no:of times a word shows in a document divided by the total number of words in this document.<\/p>\r\n

Inverse Document Frequency (IDF):<\/strong><\/p>\r\n

It measures the importance of a term.<\/p>\r\n

It is calculated by the number of documents in the text database divided by the number of documents where a specific term appears.<\/p>\r\n

While computing TF, all the phases are considered equivalently important.<\/p>\r\n

These, are MapReduce algorithms and installation<\/a>.<\/p>\r\n\r\n[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section bb_built=”1″ prev_background_color=”#000000″][et_pb_row][et_pb_column type=”4_4″][et_pb_team_member _builder_version=”3.0.106″ name=”Naveen E” position=”Author” facebook_url=”” twitter_url=”” google_url=”” linkedin_url=”” image_url=”https:\/\/asha24.net\/blog\/\/wp-content\/uploads\/2018\/03\/1234.gif” background_layout=”light” header_font=”Titillium Web|700|||||||” body_font=”Titillium Web||||||||” body_font_size=”16″ body_font_size_last_edited=”on|phone”]\r\n\r\n

\r\nHola peeps! Meet an adventure maniac, seeking life in every moment, interacting and writing at Asha24.<\/p>\r\n\r\n[\/et_pb_team_member][\/et_pb_column][\/et_pb_row][\/et_pb_section]\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":4,"featured_media":48045,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"on","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[33],"tags":[],"class_list":["post-48040","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-big-data"],"yoast_head":"\nIntroduction To MapReduce in Big Data - Asha24 Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Introduction To MapReduce in Big Data - Asha24 Blog\" \/>\n<meta property=\"og:url\" content=\"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/\" \/>\n<meta property=\"og:site_name\" content=\"Asha24 Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-28T05:51:23+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-04T04:01:02+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/asha24.net\/blog\/wp-content\/uploads\/2018\/03\/introduction-to-mapreduce-in-big-data-2.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"865\" \/>\n\t<meta property=\"og:image:height\" content=\"649\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mahesh\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/asha24.net\/blog\/#website\",\"url\":\"https:\/\/asha24.net\/blog\/\",\"name\":\"Asha24 Blog\",\"description\":\"Dedication Towards Learning\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/asha24.net\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/asha24.net\/blog\/wp-content\/uploads\/2018\/03\/introduction-to-mapreduce-in-big-data-2.jpg\",\"contentUrl\":\"https:\/\/asha24.net\/blog\/wp-content\/uploads\/2018\/03\/introduction-to-mapreduce-in-big-data-2.jpg\",\"width\":865,\"height\":649},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/#webpage\",\"url\":\"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/\",\"name\":\"Introduction To MapReduce in Big Data - Asha24 Blog\",\"isPartOf\":{\"@id\":\"https:\/\/asha24.net\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/#primaryimage\"},\"datePublished\":\"2018-01-28T05:51:23+00:00\",\"dateModified\":\"2023-02-04T04:01:02+00:00\",\"author\":{\"@id\":\"https:\/\/asha24.net\/blog\/#\/schema\/person\/f6167b78bbaddfc399ae1154cd5b6759\"},\"breadcrumb\":{\"@id\":\"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/asha24.net\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Introduction To MapReduce in Big Data\"}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/asha24.net\/blog\/#\/schema\/person\/f6167b78bbaddfc399ae1154cd5b6759\",\"name\":\"Mahesh\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/asha24.net\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/085c03e75ffb51af2509c1cfad9c7d78b30236d43a008db2e46f96e2f40c67fc?s=96&d=wavatar&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/085c03e75ffb51af2509c1cfad9c7d78b30236d43a008db2e46f96e2f40c67fc?s=96&d=wavatar&r=g\",\"caption\":\"Mahesh\"},\"url\":\"https:\/\/asha24.net\/blog\/author\/mahesh\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Introduction To MapReduce in Big Data - Asha24 Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/","og_locale":"en_US","og_type":"article","og_title":"Introduction To MapReduce in Big Data - Asha24 Blog","og_url":"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/","og_site_name":"Asha24 Blog","article_published_time":"2018-01-28T05:51:23+00:00","article_modified_time":"2023-02-04T04:01:02+00:00","og_image":[{"width":865,"height":649,"url":"https:\/\/asha24.net\/blog\/wp-content\/uploads\/2018\/03\/introduction-to-mapreduce-in-big-data-2.jpg","path":"\/home\/reviews981\/public_html\/asha24.net\/blog\/wp-content\/uploads\/2018\/03\/introduction-to-mapreduce-in-big-data-2.jpg","size":"full","id":48045,"alt":"","pixels":561385,"type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_misc":{"Written by":"Mahesh","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebSite","@id":"https:\/\/asha24.net\/blog\/#website","url":"https:\/\/asha24.net\/blog\/","name":"Asha24 Blog","description":"Dedication Towards Learning","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/asha24.net\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"ImageObject","@id":"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/#primaryimage","inLanguage":"en-US","url":"https:\/\/asha24.net\/blog\/wp-content\/uploads\/2018\/03\/introduction-to-mapreduce-in-big-data-2.jpg","contentUrl":"https:\/\/asha24.net\/blog\/wp-content\/uploads\/2018\/03\/introduction-to-mapreduce-in-big-data-2.jpg","width":865,"height":649},{"@type":"WebPage","@id":"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/#webpage","url":"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/","name":"Introduction To MapReduce in Big Data - Asha24 Blog","isPartOf":{"@id":"https:\/\/asha24.net\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/#primaryimage"},"datePublished":"2018-01-28T05:51:23+00:00","dateModified":"2023-02-04T04:01:02+00:00","author":{"@id":"https:\/\/asha24.net\/blog\/#\/schema\/person\/f6167b78bbaddfc399ae1154cd5b6759"},"breadcrumb":{"@id":"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/asha24.net\/blog\/introduction-to-mapreduce-in-big-data\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/asha24.net\/blog\/"},{"@type":"ListItem","position":2,"name":"Introduction To MapReduce in Big Data"}]},{"@type":"Person","@id":"https:\/\/asha24.net\/blog\/#\/schema\/person\/f6167b78bbaddfc399ae1154cd5b6759","name":"Mahesh","image":{"@type":"ImageObject","@id":"https:\/\/asha24.net\/blog\/#personlogo","inLanguage":"en-US","url":"https:\/\/secure.gravatar.com\/avatar\/085c03e75ffb51af2509c1cfad9c7d78b30236d43a008db2e46f96e2f40c67fc?s=96&d=wavatar&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/085c03e75ffb51af2509c1cfad9c7d78b30236d43a008db2e46f96e2f40c67fc?s=96&d=wavatar&r=g","caption":"Mahesh"},"url":"https:\/\/asha24.net\/blog\/author\/mahesh\/"}]}},"_links":{"self":[{"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/posts\/48040","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/comments?post=48040"}],"version-history":[{"count":11,"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/posts\/48040\/revisions"}],"predecessor-version":[{"id":52187,"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/posts\/48040\/revisions\/52187"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/media\/48045"}],"wp:attachment":[{"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/media?parent=48040"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/categories?post=48040"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/asha24.net\/blog\/wp-json\/wp\/v2\/tags?post=48040"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}