Skip to main content

Datawarehouse Bigdata Integration - Proof of Concept

The objective of this proof of concept project is to evaluate the feasibility of converting a traditional ETL architecture for data warehouse load into a hybrid approach with bigdata integration.
 
Refer the following post for architectural details.
 

  • Proof of Concept - Project Plan

 The POC project has a timeline of 4 weeks.
Following activities planned during this period.
  1. Define Business goals and corresponding use cases.
  2. Setup and Configuration.
  3. Architecture and Design.
  4. Development.
  5. Evaluation and recommendations.
 
 

  • System Hardware Architecture

Minimal hardware investment planned for the POC project. Cluster configured with 1 master nodes and 3 slave nodes.
 

 

  • System Software Architecture

 

  • Application Use Cases

Application use cases include the following.
  1. Data ingestion from OLTP to HDFS using Sqoop.
  2. Load Facts using mapreduce jobs.
  3. Create Aggregates using Hive QLs.
  4. Export processed Aggregates from from HDFS to DW using Sqoop.

  • Performance Use Cases

 

 

Comments

Post a Comment

Popular posts from this blog

DW Architecture - Traditional vs Bigdata Approach

DW Flow Architecture - Traditional             Using ETL tools like Informatica and Reporting tools like OBIEE.   Source OLTP to Stage data load using ETL process. Load Dimensions using ETL process. Cache dimension keys. Load Facts using ETL process. Load Aggregates using ETL process. OBIEE connect to DW for reporting.  
35 comments
Read more

CentralBins ChatGPT: Revolutionizing Data Dialogue

  Introduction to CentralBins ChatGPT CentralBins ChatGPT is an innovative AI-powered chatbot designed to revolutionize data dialogue. It leverages the cutting-edge technology of OpenAI's GPT-3 to transform the way data professionals, educators, and business leaders interact with big data and analytics. This advanced chatbot is developed by CentralBins, a leading provider of AI solutions in the field of big data analytics, data warehousing, and business intelligence. With CentralBins ChatGPT, users can engage in natural language conversations to gain insights, seek solutions, and explore the depths of data-driven decision-making. CentralBins ChatGPT is designed to streamline the process of querying, analyzing, and interpreting data, making it an indispensable tool for unlocking the potential of data assets across various domains and industries. Understanding the Role of ChatGPT in Data Dialogue ChatGPT plays a pivotal role in enhancing data dialogue by enabling seamless and intuiti...
Post a Comment
Read more

Cloudera QuickStart virtual machines (VMs) Installation

Cloudera Distribution including Apache Hadoop ( CDH ) is the most popular Hadoop distribution currently available. CDH is 100% open source. Cloudera quick start VMs include everything that is needed to tryout basic package based CDH installation. This is useful to create initial deployments for proof of concept (POC) or development.
Post a Comment
Read more