Tuesday, July 05, 2011

Data modeling and agile BI

One of the advantage that some analytical tools such as QlikView Spotfire or Tableau claims to offer over the products they call "Traditional BI" is that they can be used without data modeling. According to this claim, data modeling is a major obstacle to agile business intelligence and not needed anyway.

Is it true that data modeling is dead? Has technology found a workaround?
The need for data modeling depends upon the application. Products that promise user friendly analysis without any data analysis are usually intended for a specific type of analysis that does not require any previously specified structure.

A good example of data does not require modeling that retailers gather about their customers. This data comes in big flat tables with many columns, and the whole point to the analysis is to find unexpected patterns in this unstructured data. In this case adding a model is adding assumptions that may actually hinder the analysis process.

However, some types of analyses only make sense with at least some modeling. Time intelligence is an example of a type of analysis that is supported by a data model. Also analyzing predefined internal structures such as cost accounts or complex sales channels is usually more convenient based on predefined structures. The alternative method of discovering the structures in the raw data may not be possible.

Planning is a common area of agile BI, and planning is rarely possible without predefined structures. It is no coincidence that the tools that promise analysis without data modeling do not offer planning features. Planning requires adding new data to an existing data set. In some cases, this includes adding new master data, for example when new products are being planned. Furthermore, there is often a good deal of custom business logic in a planning application that cannot be defined automatically. Most financial planning processes, and the analysis and simulation that goes along with them cannot be carried out on simple table.

In my view the new generation columnar databases are a welcome addition to agile BI. But I also think that their marketing is sometimes a little over the top when it comes to dismissing existing BI solution in this area.

8 comments:

Erica Driver said...

Barney,

Thanks for writing up this article about data modeling and BI. It sparked a lively conversation here at QlikTech.

I wanted to point out a subtlety that we should probably do a better job of explaining. We don’t purport that QlikView can be used “without data modeling.” In fact, QlikView includes a built-in visual data modeling tool that helps the developer (or business user, with appropriate skills) connect together tables extracted from various sources, linked on key fields, so QlikView can work its magic—which is associating the data so users can explore the data, asking and answering their own questions and pursuing their own path to insight. Ease of data modeling is one of the beauties of QlikView.

Analyzing internal structures such as cost accounts or complex sales channels absolutely *can* be done, quite effectively, using a Business Discovery platform. QlikView isn’t limited to data that exists just in a simple table, as you describe. QlikView can handle billions of rows of data extracted from dozens of different data sources—and it automatically builds associations in the data so the user can click on, or select with the mouse, any data in the data set and instantly see what is associated with it—and what is not associated. When analyzing cost accounts or complex sales channels, for example, the user can select a time period and instantly see what expenses were *not* incurred in that period. They can compare the sales from one channel with sales from another channel, by product line or category and by time.

If I misunderstood you and what you mean, instead, is that QlikView and other Business Discovery tools are not optimally suited to use cases that require pre-determined drill paths, static reports, and preconfigured queries and reports, then yes, you are correct. We fully recognize that the need for these capabilities exists, and will continue to exist, for some use cases. But in our view, Business Discovery is the approach that ultimately fulfills the promise of BI: to enable decision makers at all levels in an organization to leverage data for business advantage—to explore data, draw insights and meaningful conclusions, and make better decisions.

Erica Driver, QlikTech

Barney said...
This comment has been removed by the author.
Barney said...

Erica,
Thanks for your comment. There is a fine line between data import and data modeling, I suppose. To me the visual modeling tool you mention is a data import tool. The final result is just a flat table. But you could argue that it is a kind of data modeling as well. It is true that the data can comes from disparate sources.

For my purposes here, I was thinking of modeling as defining pre-determined drill paths. The point I was making here is that it is sometimes, but not always needed, so I think we agree.

I am planning a few more posts on this topic, and I intend to get back to the issue of data acquisition in agile BI in the next post, because I according to our research it is two thirds of the effort in BI projects.

Ralf Becher said...

Barney,
you're right there are some lacks of modelling support on modern BI tools.
But, I don't know if you're familiar with QlikView. The data modelling in QlikView - there is no visual data modelling tool but a script language - leads into a ER like data model (which can be visualized of course) and not a flat table.
Ralf Becher, TIQ Solutions

Barney said...

Ralf,
There is some basic modeling without a script -- the tool can recognize some structures in a rlational database automatically. But you are right that what I would call data import, modeling requires a lot of scripting. I'm not quite sure why QlikTech does not change this. I promise to get back to data import and agile BI soon.

Anonymous said...

Hi there, I check your blogs like every week. Your writing style is
awesome, keep it up!

Here is my web blog - http://kellykemnitz.com/mediawiki/index.php?title=User:UCWLucie

Anonymous said...

If some one wishes to be updated with newest technologies afterward he must be pay a visit this site and be up to date
every day.

Feel free to visit my web page ... Buy Garcinia Cambogia

Anonymous said...

What a information of un-ambiguity and preserveness
of precious knowledge regarding unpredicted emotions.


Look into my webpage; Vapor reviews

Post a Comment

Please leave a comment