Datawarehouse Bigdata Integration - Proof of Concept

The objective of this proof of concept project is to evaluate the feasibility of converting a traditional ETL architecture for data warehouse load into a hybrid approach with bigdata integration.

 

Refer the following post for architectural details.

 

 http://www.bigdatadwbi.com/2015/08/dw-architecture-traditional-vs-bigdata.html

• Proof of Concept - Project Plan

 The POC project has a timeline of 4 weeks.

Following activities planned during this period.

1. Define Business goals and corresponding use cases.
2. Setup and Configuration.
3. Architecture and Design.
4. Development.
5. Evaluation and recommendations.

 

 

• System Hardware Architecture

Minimal hardware investment planned for the POC project. Cluster configured with 1 master nodes and 3 slave nodes.

 

 

• System Software Architecture

 

• Application Use Cases

Application use cases include the following.

1. Data ingestion from OLTP to HDFS using Sqoop.
2. Load Facts using mapreduce jobs.
3. Create Aggregates using Hive QLs.
4. Export processed Aggregates from from HDFS to DW using Sqoop.

• Performance Use Cases