Data warehouses (DWs) built using waterfall methodologies are not doomed to failure. Despite the headlines we see of how often DWs fail, many of them do succeed. I worked on several DWs earlier in my career that used a traditional waterfall software development lifecycle (SDLC). In my experience, the projects were massively over schedule and over budget, but they did succeed in the end and some of them are still in use today.

But we could have done better. The companies hired consultants who knew little about the data and were given limited access to our business users or technical subject matter experts, because they were too busy. We made assumptions about what the users wanted and tried to ensure that the DW could handle these situations, thinking we were brilliant when we came up with a possible use case for the data that nobody else had thought of. For example, we decided to track history for every field in the system, because somebody somewhere might need it someday.

These stories aren’t shared to make me look bad. Mistakes like this are common, and we learn from them. I believe that using agile methodologies alleviates many of the situations that led to the mistakes I’ve experienced. And I’m not alone, TDWI’s 2014 BI Benchmark report determined that ‘agile is the most effective development methodology in terms of BI value’:

TDWI does not elaborate on how the define ‘hybrid’ in the benchmark report but I interpret it to mean companies use BOTH agile and waterfall depending on the project (rather than that they take what they like from both methodologies and blend them together).