{"id":13984,"date":"2021-08-31T17:26:51","date_gmt":"2021-08-31T11:56:51","guid":{"rendered":"http:\/\/ismiletechnologies.com\/?p=13984"},"modified":"2022-10-06T17:07:25","modified_gmt":"2022-10-06T11:37:25","slug":"hadoop-migration-service","status":"publish","type":"post","link":"https:\/\/ismiletechnologies.com\/en_us\/cloud-services\/hadoop-migration-service\/","title":{"rendered":"HADOOP MIGRATION SERVICE"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">In this paper,&nbsp;we will go through the basic scenario of migration&nbsp;from Hadoop on-premises to Microsoft Azure.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Firstly, let\u2019s see, the benefits of migrating to Azure HDInsight:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Secure and Complaint:&nbsp;Azure HDInsight implements encryption and integrates with Azure Active Directory to protect enterprise data assets.&nbsp;<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li>Easily Scalable:\u00a0You can scale HDInsight workloads up and down on Azure. In addition to redistributing data, the platform rebalances workloads without interrupting work.\u00a0<\/li><li>Integration with other Azure services:\u00a0Integration with other Azure services is easy with HDInsight, such as Azure Data Lake Storage Gen2, Azure SQL Database, and Azure Data Factory (ADF).\u00a0<\/li><li>Self-Healing processes and components:\u00a0HDInsight monitors open-source components and infrastructure around the clock. Additionally, it automatically recovers critical failures such as unavailability of open-source components and nodes.\u00a0<\/li><li>Simplified version management:\u00a0Azure HDInsight keeps Hadoop ecosystem components updated.\u00a0<\/li><li>Extensibility with custom tools:\u00a0It is possible to extend the clusters of HDInsight with installed components. Furthermore, they\u00a0are capable of integrating\u00a0with other big data solutions.\u00a0<\/li><\/ul>\n\n\n\t\t<div data-elementor-type=\"section\" data-elementor-id=\"33253\" class=\"elementor elementor-33253\" data-elementor-post-type=\"elementor_library\">\n\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-1f27ef0 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"1f27ef0\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;gradient&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c65c317\" data-id=\"c65c317\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-dc3c880 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"dc3c880\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-3248be5\" data-id=\"3248be5\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-5886bd6 elementor-widget elementor-widget-heading\" data-id=\"5886bd6\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Ready to experience the full power of cloud technology?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1f56c4e elementor-widget elementor-widget-text-editor\" data-id=\"1f56c4e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p><span class=\"TextRun Highlight SCXW216552313 BCX0\" lang=\"EN-IN\" xml:lang=\"EN-IN\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW216552313 BCX0\">Our cloud experts will speed up cloud deployment, and make your business more efficient.\u00a0<\/span><\/span><span class=\"EOP SCXW216552313 BCX0\" data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-4b7dd61\" data-id=\"4b7dd61\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-81f081a elementor-align-center elementor-widget elementor-widget-lottie\" data-id=\"81f081a\" data-element_type=\"widget\" data-e-type=\"widget\" data-settings=\"{&quot;source&quot;:&quot;external_url&quot;,&quot;source_external_url&quot;:{&quot;url&quot;:&quot;https:\\\/\\\/assets4.lottiefiles.com\\\/packages\\\/lf20_kkesf8mx.json&quot;,&quot;is_external&quot;:&quot;&quot;,&quot;nofollow&quot;:&quot;&quot;,&quot;custom_attributes&quot;:&quot;&quot;},&quot;loop&quot;:&quot;yes&quot;,&quot;play_speed&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:0.9,&quot;sizes&quot;:[]},&quot;lazyload&quot;:&quot;yes&quot;,&quot;link_to&quot;:&quot;none&quot;,&quot;trigger&quot;:&quot;arriving_to_viewport&quot;,&quot;viewport&quot;:{&quot;unit&quot;:&quot;%&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:{&quot;start&quot;:0,&quot;end&quot;:100}},&quot;start_point&quot;:{&quot;unit&quot;:&quot;%&quot;,&quot;size&quot;:0,&quot;sizes&quot;:[]},&quot;end_point&quot;:{&quot;unit&quot;:&quot;%&quot;,&quot;size&quot;:100,&quot;sizes&quot;:[]},&quot;renderer&quot;:&quot;svg&quot;}\" data-widget_type=\"lottie.default\">\n\t\t\t\t\t<div class=\"e-lottie__container\"><div class=\"e-lottie__animation\"><\/div><\/div>\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"elementor-column elementor-col-33 elementor-inner-column elementor-element elementor-element-ac6262c\" data-id=\"ac6262c\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-e19f719 elementor-align-right elementor-mobile-align-left elementor-tablet-align-right elementor-widget elementor-widget-button\" data-id=\"e19f719\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t\t\t\t\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-xs\" href=\"#elementor-action%3Aaction%3Dpopup%3Aopen%26settings%3DeyJpZCI6IjI5OTEiLCJ0b2dnbGUiOnRydWV9\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t\t\t\t<span class=\"elementor-button-text\">Start Today !<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Steps to Migrate<\/strong>&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There are two main routes to moving your Hadoop data to Azure HDInsight:&nbsp;<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Transfer data over network with TLS&nbsp;<\/li><\/ol>\n\n\n\n<ul class=\"wp-block-list\"><li>Over Internet:&nbsp;Through the Internet, you can easily upload large amounts of data to Azure cloud storage. The Azure Storage Explorer, AzCopy, Azure&nbsp;PowerShell, etc. are all tools that can aid in this process.&nbsp;&nbsp;&nbsp;<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li>Express Route:&nbsp;In the Azure ExpressRoute service, users can create private connections between Microsoft data&nbsp;centres&nbsp;and infrastructure on premise or in a colocation facility.&nbsp;In addition to offering better security, reliability, and speeds than traditional Internet connections, ExpressRoute connections don&#8217;t go over the public Internet at all.&nbsp;<\/li><li>Data Box Online Data Transfer:&nbsp;Both Data Box Gateway and Data Box Edge serve as online data transfer products and network storage gateways. The Data Box Edge is an on-premises network device that can transfer data to and from the Azure cloud using artificial intelligence-enabled edge compute. Data Box Gateway is a virtual appliance with storage gateway functionality.&nbsp;<\/li><\/ul>\n\n\n\n<ol class=\"wp-block-list\" start=\"2\"><li>Shipping data offline:&nbsp;Devices such as Data Box Disk and Data Box Heavy let you transfer large data volumes to Azure when you cannot use a network.&nbsp; Devices for offline data transfers are shipped between the Azure&nbsp;datacentre&nbsp;and your organization. To help protect your data in transit, they use AES encryption. After upload, they perform a thorough sanitization process to remove your data from the device.&nbsp;<\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Type of data movement can be decided, depending on below factors:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Data Size:&nbsp;Data size to be transferred&nbsp;<\/li><li>Network:&nbsp;The amount of bandwidth available for data transfer in your environment.&nbsp;<\/li><li>Transfer Frequency:&nbsp;Periodic or one-time data ingestion&nbsp;<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Based on bandwidth, data size, and transfer frequency, this image illustrates the guidelines for choosing Azure data transfer tools.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"621\" src=\"http:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/08\/image-28-1024x621.png\" alt=\"\" class=\"wp-image-13987\" srcset=\"https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/08\/image-28-1024x621.png 1024w, https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/08\/image-28-300x182.png 300w, https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/08\/image-28-768x466.png 768w, https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/08\/image-28.png 1054w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Based on data volume and bandwidth, the following table shows data transfer duration. Consider using a Data box if the migration is expected to take longer than three weeks.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"999\" height=\"766\" src=\"http:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/08\/image-27.png\" alt=\"\" class=\"wp-image-13986\" srcset=\"https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/08\/image-27.png 999w, https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/08\/image-27-300x230.png 300w, https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/08\/image-27-768x589.png 768w\" sizes=\"(max-width: 999px) 100vw, 999px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">A number of&nbsp;native Azure tools, such as Apache Hadoop&nbsp;DistCp, Azure Data Factory, and&nbsp;AzureCp, are available for data transfers.&nbsp;WANDisco&nbsp;is another tool that can be used for the same purpose. On-premises data can be transferred to Azure storage systems using Apache Kafka&nbsp;Mirrormaker&nbsp;and Apache Sqoop.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Apache Hadoop&nbsp;DistCp&nbsp;performance considerations&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The&nbsp;DistCP&nbsp;project uses MapReduce Map jobs to handle data transfer, handle errors and recover from errors. For each Map task, Source files are assigned. It then copies all the files it has been assigned to the destination. You can improve the performance of&nbsp;DistCp&nbsp;using&nbsp;a number of&nbsp;techniques.&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Increase the number of Mappers:&nbsp;&nbsp;<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">In&nbsp;DistCp, tasks are created so that each one copies roughly the same amount of data. In&nbsp;DistCp, 20 mappers are used by default. With&nbsp;Distcp&nbsp;Mapper (&#8216;m&#8217; parameter at command line), data transfers can be parallelized, reducing their duration. While increasing the number of mappers, however, there are two factors to keep in mind:&nbsp;<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>The lowest level of granularity of&nbsp;DistCp&nbsp;is a single file. By specifying more Mappers than source files, you are wasting cluster resources&nbsp;<\/li><\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"2\"><li>Mappers should be determined by considering the available Yarn memory on the cluster. A Yarn container is launched for every Map task. If no other heavy workloads are running on the cluster, the number of Mappers can be determined as follows: m = (number of worker nodes * YARN memory per worker node) \/ YARN container size.&nbsp;DistCp&nbsp;jobs can be limited to just a portion of YARN memory if other applications are using memory&nbsp;<\/li><\/ol>\n\n\n\n<ul class=\"wp-block-list\"><li>Use more than one&nbsp;DistCp&nbsp;job:&nbsp;<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Use multiple&nbsp;DistCp&nbsp;jobs if the dataset to be moved is larger than 1 TB. By using multiple jobs, failures can be reduced. When a job fails, you need to restart only that&nbsp;particular job&nbsp;and not all of them.&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Consider splitting files:&nbsp;<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">When there are a few large files, consider splitting them into 256-MB chunks to increase concurrency with more Mappers.&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Increase the number of threads:&nbsp;<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Check if increasing the -numListstatusThreads&nbsp;Parameters improve performance. The maximum value for this parameter is 40, and it controls the number of threads for building file listings.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this paper,&nbsp;we will go through the basic scenario of migration&nbsp;from Hadoop on-premises to Microsoft Azure.&nbsp; Firstly, let\u2019s see, the benefits of migrating to Azure HDInsight:&nbsp; Secure and Complaint:&nbsp;Azure HDInsight implements encryption and integrates with Azure Active Directory to protect enterprise data assets.&nbsp; Easily Scalable:\u00a0You can scale HDInsight workloads up and down on Azure. In [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":14188,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[81],"tags":[],"class_list":["post-13984","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-services"],"_links":{"self":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts\/13984","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/comments?post=13984"}],"version-history":[{"count":2,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts\/13984\/revisions"}],"predecessor-version":[{"id":33645,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts\/13984\/revisions\/33645"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/media\/14188"}],"wp:attachment":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/media?parent=13984"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/categories?post=13984"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/tags?post=13984"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}