Have you ever seen a data plant? No? Then it's about time to visit one. And as luck would have it, Uptown Process is home to the first such data assembly line.
Uptown Process is a place where you can discover lean and efficient process frameworks. Our top choice is PMI Disciplined Agile, offering a goal-centric approach with various delivery lifecycle options. These include iteration-based lifecycles like Scrum and flow-based lifecycles like Kanban. Furthermore, there are two types of development organizations:
- Generalizing Specialists Organization: In this setup, most tasks for implementing a macro end-to-end increment (from data source to data warehouse and final data product) can be executed by a single, versatile specialist.
- Assembly Line Organization: In this arrangement, work on a macro end-to-end increment is divided into several workstations where specialists handle micro end-to-end increments.
A typical data assembly line in the BI/DWH domain consists of five workstations:
- Rapid Prototyping: Create a rapid prototype of the data product and underlying data model, connecting directly to the source system with analysis tools like Power BI.
- Data Lake & Persistent Staging Area (PSA): Implement source system connectivity, loading necessary data into the data lake and PSA. Initially, load only a few months ofdata for quick loading and testing. Provide a simple user interface (e.g., an Excel-based report) for product owners or business users to validate the new increment.
- Data Warehouse & Data Mart: Load new source data from the PSA into the Data Warehouse and data mart layers, including data harmonization in the Data Warehouse and measure logic implementation in the data mart layer. Report this data in a basic Excel report without complex design.
- Final Data Product: Create the data product by taking data from the underlying Data Mart, focusing on meaningful visualization and user experience.
- Deployment, Integration Testing & Data Volume Extension: Deploy the macro end-to-end incre-ment to a test environ-ment, loading it with the full amount of available source data to add more value compared to increments built with a subset of data.