{"id":16688,"date":"2021-09-22T21:25:27","date_gmt":"2021-09-22T15:55:27","guid":{"rendered":"http:\/\/ismiletechnologies.com\/?p=16688"},"modified":"2022-12-10T02:01:05","modified_gmt":"2022-12-09T20:31:05","slug":"data-migration-from-oracle-on-premises-database-to-big-query","status":"publish","type":"post","link":"https:\/\/ismiletechnologies.com\/en_us\/dataops\/data-migration-from-oracle-on-premises-database-to-big-query\/","title":{"rendered":"Data Migration from Oracle on premises Database to Big Query"},"content":{"rendered":"\r\n<p class=\"wp-block-paragraph\">There can be numerous ways to migrate data from a database to Big Query like\u00a0<strong>exporting to CSV, Parquet files, direct DB connection<\/strong>, etc.\u00a0So\u00a0in this article, we will focus on two ways-\u00a0<\/p>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li>Using\u00a0Fivetran\u00a0connector\u00a0<\/li>\r\n<li>Using a python script\u00a0\u00a0<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\"><strong>Fivetran\u00a0connector<\/strong>\u00a0<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-full is-resized\"><img decoding=\"async\" class=\"wp-image-16694\" src=\"http:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-101.png\" alt=\"Data Migration from Oracle on premises Database to Big Query\" width=\"283\" height=\"142\" srcset=\"https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-101.png 768w, https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-101-300x150.png 300w\" sizes=\"(max-width: 283px) 100vw, 283px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">GCP provides numerous connectors for database connection\/migration, one of which is\u00a0Fivetran.\u00a0<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">\u00a0Using\u00a0Fivetran\u00a0we can establish a\u00a0<strong>direct connection between Big Query and Oracle database<\/strong>\u00a0for this we need to just configure it by\u00a0<strong>providing the live hosting address of our database and its credentials<\/strong>,\u00a0\u00a0<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Once\u00a0verified then it\u2019s good to go for data migration.\u00a0<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">But using\u00a0Fivetran\u00a0connector is\u00a0<strong>not free, it&#8217;s chargeable.<\/strong>\u00a0<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\"><strong>Using Python script<\/strong>\u00a0<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"328\" height=\"120\" class=\"wp-image-16697\" src=\"http:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-104.png\" alt=\"Data Migration from Oracle on premises Database to Big Query\" srcset=\"https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-104.png 328w, https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-104-300x110.png 300w\" sizes=\"(max-width: 328px) 100vw, 328px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">This method is the best if the Oracle database\u00a0<strong>is at the local system<\/strong>\u00a0and does\u00a0<strong>not have a hosting address<\/strong>.\u00a0\u00a0<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">In this using python script, we will establish a type of connection between the Oracle database and Big Query, for that, we need to connect the python script with the oracle database\u00a0<strong>using proper connection string\u00a0<\/strong>and same way using\u00a0<strong>the credential key provided by GCP\u00a0<\/strong>we can establish a connection with Big Query.\u00a0<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">\u00a0After connection establishment, we need to get the data from the Oracle database for which we perform a SQL query inside python script\u00a0<strong>using\u00a0Sqlalchemy\u00a0and Pandas<\/strong>\u00a0and the result will be the\u00a0<strong>desired data stored in\u00a0Dataframe<\/strong>.\u00a0<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Using <strong>Panda\u2019s \u201cto_gbq\u201d method<\/strong>\u00a0we can easily send our data from the\u00a0<strong>data frame to the Big Query table<\/strong>.\u00a0\u00a0<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\"><strong>We can make separate text files for the connection string, SQL query, GCP Big Query credential and read them in\u00a0python script for easy use. This makes it easier to transfer data; change the SQL script and run the python script, and your data will be transferred.\u00a0\u00a0<\/strong><\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-full\"><img fetchpriority=\"high\" decoding=\"async\" width=\"386\" height=\"212\" class=\"wp-image-16696\" src=\"http:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-103.png\" alt=\"Data Migration from Oracle on premises Database to Big Query\" srcset=\"https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-103.png 386w, https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-103-300x165.png 300w\" sizes=\"(max-width: 386px) 100vw, 386px\" \/><\/figure>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"424\" height=\"222\" class=\"wp-image-16693\" src=\"http:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-100.png\" alt=\"Data Migration from Oracle on premises Database to Big Query\" srcset=\"https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-100.png 424w, https:\/\/ismiletechnologies.com\/wp-content\/uploads\/2021\/09\/image-100-300x157.png 300w\" sizes=\"(max-width: 424px) 100vw, 424px\" \/><\/figure>\r\n","protected":false},"excerpt":{"rendered":"<p>There can be numerous ways to migrate data from a database to Big Query like\u00a0exporting to CSV, Parquet files, direct DB connection, etc.\u00a0So\u00a0in this article, we will focus on two ways-\u00a0 Using\u00a0Fivetran\u00a0connector\u00a0 Using a python script\u00a0\u00a0 Fivetran\u00a0connector\u00a0 GCP provides numerous connectors for database connection\/migration, one of which is\u00a0Fivetran.\u00a0 \u00a0Using\u00a0Fivetran\u00a0we can establish a\u00a0direct connection between Big [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":16790,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[249],"tags":[],"class_list":["post-16688","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dataops"],"_links":{"self":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts\/16688","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/comments?post=16688"}],"version-history":[{"count":6,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts\/16688\/revisions"}],"predecessor-version":[{"id":36166,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts\/16688\/revisions\/36166"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/media\/16790"}],"wp:attachment":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/media?parent=16688"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/categories?post=16688"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/tags?post=16688"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}