Agile Data Warehousing

In my ScrumMaster certification course, the instructor started the day with a group exercise.  He broke the class into groups of two with each team having a ‘manager’ and a ’developer’.  Our first task was to navigate around a small section of the room with the manager giving direction (start, stop, left turn, right turn) and the developer following those directions, counting how many times we could circle the area together in the allocated time period.   The second time around, the goal was the same but the developers decided when and where to move and the managers just came along for the ride.

The first attempt resulted in very little progress with many near-collisions, complaints about management, and laughter.  The second resulted in us gradually merging together and forming a circle that went round and round the room in sync – more than twice as many times as the first try.  A silly exercise yes, but it got the point across – one of the primary tenets of the Agile Manifesto is that the best architecture, requirements and designs emerge from self-organizing teams.

It is easy to get tired of the hype surrounding Big Data, Data Science, Self-Service, Cloud, Mobile BI, Agile BI, etc.  I’ve worked with several data warehousing teams who have said that agile didn’t work for them, that agile doesn’t work for data warehousing or that they are just plain tired of hearing the word.  My experience has been:

• Data warehousing teams who feel agile doesn’t work for them are trying to ‘do agile’ rather than ‘be agile’ and can succeed if they re-assess their practices.
• Agile is an excellent methodology for data warehouses if you account for the differences between data warehousing and software development.
• ‘Agile Fatigue’ isn’t going away any time soon — it’s the natural result of people (like me) feeling so rejuvenated by the benefits of agile that they can’t stop using it as the solution to all of the problems of the universe.