About Agile Data Warehousing and Business Intelligence

Data warehouses and databases don’t differ to staggering levels, they are familiar systems, are both relational data systems, but were built to serve different purposes. A data warehouse is generally built to store staggeringly large quantities of historical data and also process simple and complex queries to be made all across those datasets. This is often made possible using online analytical processing tools and algorithms. 

Here's where a Data Warehouse comes to business use:

Data warehouse use cases focus on providing high-level reporting and analysis that lead to more informed business decisions. They are used for the following purposes:

1. Carrying out data mining to gain new insights from the information held in many large databases. Mining is done at a frequent scale and extrudes a lot of data to bring up the right information. 
2. Conducting market research by analyzing large volumes of data in-depth. The warehouses serve as a hoard for data to be accessed by applications and individuals for learning. 
3. An online business analyzing user behavior to make business decisions. Analysing user behavior requires a lot of to-and-fro between the database and the accessing user. The data warehouse makes it easier to access enormous quantities of data, verify and put it to good use as and when required. 

A disciplined (agile) approach to data warehouse development 

Inception Phase : What should be done during this period 

This is sort of the zero hours, during the inception phase the team strives to perform discussions and devise a direction for themselves to head in. This process shouldn't consume more than 2-5 days. What items should be covered during this inception period ?

• Team formation - An initial team has to be formed with each member contributing to the best of their abilities. It is generally tough to make sure every team member is on the same page about a few things but that’s what your leadership skill has to get done.
• Workout a common resort or vision for the business intelligence project. 
• Bring your vision in parallel with enterprise's direction. 
• Formulate an initial technical development, execution and resource engagement strategy.
• Lay-down a first draft of the Release Plan.
• Get secured on funds. 
• Form a collaborative and enthusiastic work environment. 
• Identify potential risks.

The Construction Phase

The construction phase is the actual build phase, where the team tries to meet goals. 

• Produce a Potentially Consumable Solution.
• Address Changing Stakeholder Needs.
• Move Closer to a Deployable Release.
• Improve Quality.
• Prove Architecture Early.

The Transition Phase

This phase is all about making the phase consumable, and finding the right time to deploy the best possible solutions and even finding workarounds to already happened hurdles. During the Transition phase the team strives to ensure that the solution is consumable, and when it is deployed the solution. 

Employ continuous testing 

Some testing may slip into the Transition phase. Ideally all testing should occur during Construction, other than one last run of your regression test suite to ensure you're ready to ship. 

Document deliverables

Some teams will let documentation slip, or at least some documentation slip, to the end of the life cycle. This is a practice called Document Late, although I prefer a continuous documentation approach described earlier.

Deploy changes to database schema if need be 

Part of your overall deployment efforts will be to deploy database schema changes. If you've been taking a database refactoring approach then this will be very straightforward as your change scripts will already be running and fully testing. Before making the schema changes you should consider creating a backup of the database.

Migrate production data to new schema

The only potential secondary activity is to finalize any secondary documentation, such as your logical data model (LDM) or your meta-data documentation, that you believe will add real value in the future. 

Best practises to get you going 

Analyse Change Requirement 

A changed requirement late in the lifecycle is a competitive advantage as long as you can act on it. Instead of adopting a strict change management process which for the most part is based on change prevention, you can instead adopt a more agile approach to change management where your stakeholders can easily change their minds as the progress progresses. 

Get inputs from stakeholders before making iterations 

Following short iterations, perhaps a few weeks in length, and providing working software at the end of each iteration, often results in stakeholders who are far more interested in getting more software than they are in getting more specifications.

Figure iterations continuously, keep making it better 

Iterations of this length provide more opportunity to govern the project effectively due to the greater feedback provided by regular delivery of working software. 

Bring lean data governance into practise 

Traditional, command-and-control approaches to data governance appear to work very poorly in practice. 

The 2006 DDJ survey into the current state of data management practices showed that 66% of development teams will choose to "work around" their organization's data group, and when they do so that 75% of the time it is because they find the data group too difficult to work with, too slow to respond, or that the data group doesn't provide sufficient value to justify the effort of working with them. 

Remainder

Data warehousing and business intelligence have come a long way and they need to be able to taken parallelly in this era of technological advancements. This is making sure they run hand in hand and processes and teams function seamlessly to deliver world class business intelligence application.