ERP industry moving from project to process, but what about reporting?

Back in the 1980’s, most ERP systems had a life expectancy of 15 years or longer. A system developed in 1980 on AS400/Mainframe, was expected to run until at least 1995. In the 90’s, expectancy was quite high still, but with the introduction of internet based communication and technology, the recycling curve ramped.

Early 2000 (after Y2K) it was generally accepted that customers would change system every 5 to 7 years. Today, it has become common to upgrade every three years, sometimes less. The large ERP vendors SAP, Oracle and Microsoft release more often – sometimes using in between release number to mark the evolution. 

Going forward, On-line (cloud-based) ERP and several Business ‘Apps’ have changed the way Businesses consume IT power. Commoditisation and consumerisation having been driving trends.

With such short-lived expectancy, it is far less likely that you will be able to succeed your ERP by starting up full projects every time. You likely need to embrace a more continuous approach, integrating acquired businesses, allowing for some silo application, hybrid cloud architecture and rapidly changing sources.

Has ERP begun to transform from a project (discrete) to a process (continuous) game?

This has an impact on how data flows in the organisation, especially with regards to management reporting. It also puts limits on our ability to capture essential information.

As most companies derive data out of more than one system, it seems like at least part of the data model is changing on a monthly basis, if not faster.

Hence, it has similarly become harder to address Management Information (or Business Intelligence) issues based on a project approach. Especially so when your Data Warehouse project takes longer to implement than the average lifetime of the source(s) it has been derived from. If it takes you 6 months to deliver a new Dashboard, and main data contribution to that dashboard changes every month, you will likely be facing some serious challenges.

This has had a couple of major implications: during the previous decade, we have marked the rise of Data Discovery tools such as QlikView orTableau – on the BI firmament. They seemed to be able to respond faster to the data challenges. As these tools are capable of connecting directly into the data model and deliver real-time reporting, they seemed to have cut out the middle man, solely based on time to market performance. 

However, they have proven far less capable of handling Data Model change underneath. As these systems start spreading throughout the organisation, people responsible of maintaining an ever growing tightly nit reporting system will have difficulty to keep up with change. Any change that is. 

As the data model grows, time to market for including new Business requirements can explode. Suppose management decides to implement a new reporting structure and KPI’s at the end of last Fiscal Year. If it takes the IT team 6 months or longer to implement, that would mean navigating blind for half of the fiscal, which is never a good option. 

Real-time reports which started lean and mean, now seem to take for ever to refresh. If you need to wait for 15 minutes for a Dashboard to refresh, what is then its real-time value? Most end-users have long gone sip some coffee and moved on to other topics.

Despite the in-memory capabilities, increased server and bandwidth, the downsides of the direct-to-data approach may rapidly overtakes initial advantages. People get discouraged and tools kicked overboard. Likely so when the next ERP upgrade cycle hits the organisation. 

But are the reporting tools to blame?

I believe Business Intelligence is likely moving from Project to Process. And the best fit agent to do so seems to be the Data Warehouse. Though this conclusion may surprise, the explanation might be quite simple:

Data Warehousing has always had the advantage of offering superior Data Governance. By this, I mean the ability to follow and verify data streams. Warehouse data has always been easier to validate, the room for erroneous data can be handled much more precisely than any other method to date. Data validation, history tracking, incremental load procedures certainly speak in favour of this proven path.

Biggest Data Warehouse turn-downs have been update frequency (real-timeness) and time to market (cycle time).

The real-timeness has been handled by adding real time load capabilities. All major platform vendors (Oracle, Microsoft, IBM) have incorporated tools and technologies to accomodate for this. So – yes, the data may be as much as 5′ to 10′ older than the reading on your computer clock, at least end-users are saved the frustration of the forced 15′ coffee-break approach their direct-to-data counterparts imply.

So what bout cycle time, and time to market?

Up until recently, the technical nature of the Data Warehouse environment implies the intervention of scarce (outside) resources. It often took time to liberate these from other projects, through expensive consulting firms. For each of them you would need to spend time to have them understand the business finesse of the existing set-up and data model.

When result were ready to be validated by the Business, they may have become obsolete.

So here is the catch 22 question: is it a question of choosing the lesser evil?

• either you spend a lot of time and work to arrive at a result the Business does not care for or need any more, or
• ou get to rapid results which may be erroneous, take too much time to refresh,
  or cannot be maintained over any period of time

Setting up slowly changing dimensions, incremental loads and other shenanigans of Data Warehouse bliss cannot be considered if the foundation itself is not in place.

By itself, these topics are not complicated:

1. set up the data to only the changes will get tracked and handled, allowing for some time overlap
2. Automatically capture and upload last minute adjustments when and where it makes sense to do so

Automatic being the card up the sleeve & Data Warehouse Automation (DWA) being the key

Will this emerging field be the key in order to move the practice of Business Intelligence from project to process?

Basically the concept is simple:

• when you inform me you that our fiscal year starts on February 01 – I can automatically derive Fiscal Quarters, Financial calendar etc.
• if I tell you I only want to capture relevant change, than we can agree on what time window we need to allow for change data capture (always reload the last 5 minutes intra day and load the full day over night).
• If we agree transactions need to be captured from a legacy system until d-1, and from the new cloud based ERP as of d-0, we both immediately understand the most obvious implications and restrictions.

Tough simple by concept, the way to implement this today has not always been so clear. If you need to hand code this, like 99,xx % of the current systems, you will either be facing one or the other catch 22 predicament. If you find a way to capture change, expected change and – to some extent – unexpected change by automating code, we may be able to tackle both.

One approach is to use code-based automation. Using specific coding languages such as BIML (Business Inteligence Markup Language) offers great automation capabilities. However, in reality they move the complexity from SQL knowledge to Object Oriented Programming knowledge, preferring people who are skilled at both. Let’s just say that these folks will not be found round every corner. A strong BIML team will usually outperform any ETL crew, experienced as it may be.

Another method can be to use black-box (appliance based) automation. ERP vendor SAP has been one to pioneer this approach. These systems, often in combination with AAA hardware will focus on throughput and performance, but will have but one option to offer when they reach the limit: buy the new version of said appliance. Still, the hardware performance boost can solve time to market issues, in the short run.

A more enticing alternative may be found in Automation tools (such as Timextender, BIReady and Wherescape). These software vendors offer the ability to generate code based on the Data Warehouse team thinking logic. They often add authoring, documentation and collaboration features – which are usually missing from the Database Engine used in the classical approach. Embracing iterative (or Agile) approach to Data Warehouse development, they often help customers move from a Project based to a more continuous, process approach. Rather than replacing the ETL toolkit, they all choose to add to the stack. And, by god, they can be fast.

Some customers report between 50% and 80% time cut on iterations.

• Is this what it will take to step up the game and satisfy the ever growing needs of information hungry end users?
• Will Data Warehouse Automation offer the prospect of supporting users with differing Business needs?

From my seat at least, it looks like a promising path, worthy of further discovery.

But then again, I may be biassed.