Wednesday, December 8, 2010
Thursday, August 7, 2008
In a former post on this blog I argued why Mr. Ferguson, Managing Director of Intelligent Business Strategies Limited, might have recommended a dangerous path for business intelligence, the path of Automated Decisions. However, at the network dinner later that day I ended up at the same table as Mr. Ferguson and thus had the opportunity to let him elaborate on the subject.
Mr. Ferguson argues that some business decisions in their nature take so long to carry out manually that it may inflict losses or otherwise put the organization at risk. For example, in a bank it may be determined in one department that a customer is unable to pay interest on his real estate loan. Thus, the bank should check its other engagements with this customer to make sure that his money shortage is taken into consideration when evaluating the overall risk. The customer may for instance have a credit card that must be blocked or somehow hindered from being further credited.
In one particular bank this evaluation process took six weeks. Thus, the bank would allow a customer to increase his debts several weeks after it was known that the customer might be unable to pay his debts.
Mike Ferguson at the Network Dinner in restaurant Påfuglen in Tivoli, Copenhagen
Advanced statistical algorithms
In that same discussion, Mr. Ferguson pointed out that by using a range of information about the customer such as historic transactions, current account statuses and demographics it is possible - thanks to advanced statistical algorithms - to make computers calculate probabilities of various scenarios or outcomes of the relationship. The algorithm will calculate the probability of a certain hypothesis, so if for example an employee in the bank needs to investigate whether certain conditions are met on a customer meaning his credit account should be blocked, then the algorithm can make a calculation of the probability that the employee will come to one conclusion or another - given the information available to the algorithm.
For example the algorithm could find that there is a 96% probability that a certain customer meets these criteria and the account should be blocked. Thus, it makes sense to allow the computer to make that decision if it can say with a certain confidence that the criteria are met.
Obviously there is a small risk that the criteria are not met and that the computer makes a wrong decision but this should be compared with the risk of human error. In fact, if an organization can document the level of human error then the algorithm can be programmed to make better decisions than humans. It is just a choice of the confidence level of the algorithm.
In cases where the algorithm cannot make a decision with the required confidence level the case will be left for manual handling by an employee.
Leave to computers what they can do better
Mr. Ferguson's point is that we should allow computers to make decisions that do not require human attention or that computers can do with a higher quality/faster than humans. When it comes to manufacturing it has long been accepted that machines usually produce a better quality than humans but maybe we yet have to accept that this idea can also be applied to decision making. A lot of science fiction has dealt with this idea and any one of us can probably mention at least one movie where this scenario runs out of control - think Terminator or Matrix.
But it is really ourselves who control what we leave to computers to decide. And in our constant race for improving production efficiency and service levels while maintaining and improving our quality of living it is important that we take the stress off humans by letting computers and other machines to do their part.
Automation and Lean BI
So, is automation of decisions "LEAN"? The immediate answer to this is "yes" because automation relieves humans of trivial work. Well, not trivial in the sense that it is easy to carry out manually but trivial in the sense that it can be automated and carried out by machines more efficiently than by humans.
However, there are also arguments against automation as a LEAN principle. If the number of decisions to be made is low then the effort of programming decision algorithms may not be justified by the benefits. This is fairly obvious. What may not be so obvious is that any type of automation makes the organization more "stiff" or fixed in its operations for two reasons:
- automation takes longer to implement and change than manual processes
- automation lowers the organizational awareness of the processes that are handled by machines. If all instances of a process are handled by a machine then there will be few humans who are familiar with the process. Thus, if the process breaks down or needs to be modified then there may be very few humans who actually know how to handle the processes manually. This will prolong the change process which means the organisation will react more slowly.
Thursday, July 3, 2008
This is basically how I got into the world of blogging. Except for that important part: Prepare.
I pretty much jumped right into the execution phase (sounds familiar?) by setting up a basic blog account and starting to write.
The charm of this was - simplicity.
However, at a certain point one reaches the phase of follow up and this is where you will eventually realize whether your goals and preparation were properly in place. You ask yourself: Is this project a success or not? The rest you can figure out.
I realized that I wasn't able to measure the term "success" because I didn't set up mechanisms for measuring Key Performance Indicators in my project.
Hey, isn't that the sound of business intelligence?
No coincidence really, because BI is a formalization of activities that successful organizations have always done: Set goals, measure critical success factors and follow up, analyze discrepancies and opportunities and make informed decisions.
Writing a blog is no different although it may not be as mission critical as running a business.
Those of you considering a blog on your own may want to consider these things in your preparation:
- You want to measure the number of readers of your blog. To do this you must install tracking scripts in your blog, e.g. from Google Analytics.
- Readers can subscribe to your blog and thus read it passively, i.e. they do not visit your blog site. To measure this activity you may want to use Feedburner to track number of subscribers etc.
- And not least: Read some of all the excellent blog literature out there
Now I am back to the execution phase, where all the fun is. Writing blog articles!
Thursday, June 19, 2008
Wednesday, June 18, 2008
This entry was held by Mr. Jørgen Steines and represents his view of what is hip in the world of Business Intelligence, or more specifically – the hip technologies, existing as well as emerging ones.
According to Mr. Steines, all companies today have one or two datawarehouses. Terabyte datawarehouses are more the standard than an exception today.
When talking about what is popular in the BI landscape we have to look at what the primary drivers are for the businesses that employ business intelligence systems to improve their profits and competitive abilities.
Driver #1: Business process integration
BI is moving out of the analytic departments into all divisions and operations of businesses and thus needs to become integrated into the existing operational systems so all users can get transparent access to BI information.
Driver #2: Flexibility
Users need to combine data in new ways to get the right context for decision making. Therefore there is a great need for flexibility in the tools that provide BI information.
Driver #3: Real-time data
In many business processes it is required for data to be real-time because critical business decisions must be made on a minute-to-minute basis. Examples of this could be logistic planning processes.
Driver #4: Growing number of users
BI will become pervasive for growing numbers of users and systems.
Driver #5: Growing data volumes
Data volumes are growing and growing and the number of source systems seems to grow as well.
Mr. Steines added another interesting perspective: The hefty consolidation taking place in the BI market with the acquisitions of Business Objects, Hyperion and Cognos being some of the more prominent, could change the BI landscape dramatically. Mr. Steines hopes that SAP will hold true its promise to let BO be an “independent” vendor of BI tools due to its breadth of tools and technologies.
So, according to Mr. Steines, these technologies are the answers to the business requirements:
Hip #1: SOA (Service Oriented Architecture)
Data must be decoupled from the applications and be provided through open data services that can be combined much the same way as LEGO bricks. The layer of services will make it transparent where data comes from whether it is a datawarehouse, ODS, Master Data or even data hosted at partners.
The SOA angle is the approach preferred by most of the companies that have acquired BI companies with products that now need to be integrated into complete product suites.
Hip #2: EII (
Much like traditional DW architecture but the main difference is that the common integrated data model is a virtual view of the data sources. Sources are mapped to common views and then dynamically extracted to applications, reports, portals, message queues etc.
The concept was made popular by IBM’s Web Sphere Federator. The key aspect of EII from a BI perspective is the ability to join data from the data sources. (Now, this has a remarkable resemblance with the concept of “Virtual datawarehouse” developed by no less than Acinta, doesn’t it?). To keep up performance an EII system must employ advanced caching mechanisms provided by specaialized services in the data sources.
An example of this is to combine data from the datawarehouse with live data from the operational systems.
Hip #3: Datawarehouse appliances
First there was the monolithic mainframes that were used for everything. Later appeared the OLAP dedicated servers in various flavors.
Lately we have seen column-based databases, in-memory databases and now datawarehouse appliances: Everything needed for a datawarehouse built in to one combined product. The operating system, the hardware, disk systems etc. are all built into one box that has very low maintenance costs and increased performance. This is achieved by designing systems for the sole purpose of DW – everything is tuned from the factory and it will not work well with other uses.
Hip #4: In- memory tools
All data in memory – this is especially interesting after the introduction of 64–bit systems that allow for very large memory address spaces.
Currently there are two types of tools with this technology:
- In memory BI applications (e.g. ClikTech)
- Caching of traditional RDBMS (e.g. Oracle Times Ten, SAP BI accelerator)
According to Gartner: “By 2012, 70% of Global 1000 org. will load detailed data into memory as the primary method to optimize BI application performance”
Hip #5: Open source BI tools
The primary drivers for using open source tools would include that they are cheap, may offer special functions/features, are vendor independent, allow access to source code and are mostly standard-based.
Survey: 43% are already using OS. 24% are uses OS BI. 31 consider OS BI.
Hip #6: Mash-up technologies
Mashups started in the music industry by sampling parts of different songs and combine it into new songs. The idea is the same for IT: Various services are mixed into dedicated applications. E.g. combine BBC news with a Google map to see where the news event took place.
In BI this translates to integrating internal and external data sources. But what is the difference between Mashup and SOA? Mainly, SOA is very formalized and contains a large framework of standards, whereas mashups are “rule-less” - there are some basic techniques for connecting things. So they can be made without expert assistance.
Hip #7: BI Search
BI data must be searchable the same way as in Google. Google already provides certain BI features in its OneBox enterprise search product by integrating with the metadata repositories of various BI vendors.
Summing it all up
- Business process integration: Is already happening. Many companies require this by default
- SOA: Interesting but doesn’t solve the data quality problem
- EII: Can be useful in certain scenarios especially when live data is combined with datawarehouse data
- Data Warehouse appliances: Yes, yes, yes!
- In memory tools: Are special purpose tools. Good for some, not for others.
- Open Source BI tools: The question is, will consultants be more expensive when software is free?
- Mashup: It will get inside companies due to flexibility and user-friendliness
- BI search: Try it.
Remember: Don’t try to prohibit emerging technologies. Instead remember that technology is the easy part and should never be the first consideration. For example business requirements and data quality must be considered first.
Question from the audience: “What about security?” It can be applied at multiple levels, for example the database and/or the application interface. BI systems are not special from all other IT systems.
Mr. Steines provided a good overview of what is going on in the BI landscape. However, one of the proposed technologies is questionable in value mostly because it is a fault engineering of a good idea. EII in one of its corner stones builds on the idea of virtually coupling data sources with applications. The primary motivation for doing such a thing would be to keep system complexity down, but this is contradicted by requiring such a system to be able to handle and integrate all sorts of data from all sorts of sources, require source systems to provide special interfaces and services and so forth.
Thus, as a Danish saying has it: “Den person kan med rette kalde sig ekspert, som tager det lette og gør det svært”. (Translation: That person can truly call himself an expert, who can take the simple and make it hard).
- Ad-hoc agents. Everyone must be able to create an agent, not just administrators and super users. Reacting fast to actual data is easier and more powerful than creating forecasts. Not that one should not make budgets and so on but fast reaction is more powerful.
- Hyper-relations. Any click on data will let the user view relevant, related data.
- Intelligent analysis. Suggestions by the system for how to analyze certain KPIs. A Smiley can show you trends in the underlying data, so you can click the Smiley to find out what is wrong.
- Search in unstructured data.
Tuesday, June 17, 2008
“Steering wheels” exist at all levels of the organization. It is important that everyone steers in the same direction or the business will be torn apart,” as Mike Ferguson put it
To make that possible it is necessary that the systems reflect the way the corporation works while giving the managers control downwards and getting data from below. At the same time the infrastructure needs to be able to sustain changes continuously without exposing it to errors or vulnerabilities.
And with the explosion of data, sooner or later it will be impossible to give people all the information they need to make decisions on all events for the business. This makes way for the need of automated decisions, claims Mike Ferguson.
Still, today, most of the world’s larger companies, in one way or the other, have room for improvement, even with basic stuff like common data names and definitions. A data quality firewall is necessary at every point of data entry, to prevent bad data from entering the systems. And even when that is taken care of, there is still the almost universal challenge of getting unstructured data into the BI-systems.
The goal is to place BI in the centre of the enterprise and connect it with all of the enterprise and it needs to be available wherever it’s needed, whenever it’s needed. Mike Ferguson calls this Right Time Business Optimization. To do this it is absolutely necessary to react before the data reaches storage – even in memory.
This is the challenge for the modern enterprise and no one has solved it yet, but Mike Ferguson predicts that automated decisions will start to make an impact already in 2009-2012.
I think this is highly unlikely that there will be any impact what so ever, since automated decisions already take place in most enterprises and since they really have nothing to do with BI in particular. Intelligent business systems will continue to evolve and more automated decisions will enter any enterprise.
If Mr. Ferguson is referring to automation of strategic decisions then this could – however compelling it may sound – pose a significant threat to a business that follows this path. Because any automated decision and structure can be mapped and reverse engineered by competitors who can then take advantage of the rigid structure inherent in automated systems against the business. Automation of strategic decisions will lock corporations into fixed reaction patterns making them much easier to predict by competitors who will not hold back.
The woes of BI in the coming information explosion can be much more easily handled by democratization of BI throughout the enterprise.
The main drivers in the BI-evolution, is the constant need for even faster business?? and a market that seems to get more volatile every day. And on top of that, there’s a new drive to make information infrastructure business oriented. This means changes and heavy investments for large enterprises.
SAS delivers solutions to some of the worlds largest and most competitive companies, but the arguments that make these major players go even further into BI, are the exact same arguments that will drive BI deeper into the smaller corporations as well.
Business Intelligence based on expensive data warehouses and major upgrades is out of reach for many successful companies of all sizes, but LEAN BI will level the playing field and give small companies the same benefits as their largest competitor, instantly.
With leaner organizations, fewer data and fewer applications, most of the hassle of BI will disappear and today BI can be delivered in less than a day at a fraction of the cost.
LEAN BI works with your live data, eliminating the need to copy data from your existing systems into data warehouses or data marts before you can use them in your BI-system. Since a lot of companies are in fact using similar systems there is a broad range of experience in the market place and this allows LEAN BI systems to be implemented extremely fast, even when tailoring data models to existing data structures.
The bottom line is that the advantage that larger companies had before with extensive BI, may now turn into a disadvantage, since smaller and leaner competitors can now implement the same capabilities simpler, faster and much cheaper.
All this will slow down the largest enterprises while the SMB-market will pick up the best of classical BI in faster, cheaper and trustworthy LEAN BI applications.
Online tools are prevalent in the worlds most successful enterprises with almost 60 percent of managers in World Class enterprises, while less than 40 percent of more average “peer-to-peer” companies can access tools immediately where ever they are. This analysis was made by Hacknet.
As Platon, Microsoft wants to focus on the people in the processes. To do this, solutions needs to be focused on the results and the way to these results. Data is still important, technology is important, but it’s how people leverage data and technology that sets companies apart.
While Microsoft focuses on large enterprises Acinta uses Microsoft technology to deliver the results speedy, in real time and without the usual heavy investments in data warehouse and hardware upgrades. Acinta exploits SQL Server to deliver swift and on-time BI to companies with millions of rows of data at a fraction of the cost of other solutions.
Dashboards are vital in all solutions that handle large amounts of data. They are easy to understand and immediately usable. As Michael Borges mentioned in his keynote this morning, we need to measure how well we do, to make sure that we maintain focus.
Microsoft looks at BI in three different ways:
- Personal BI: Analyze your data and use it to work smarter. It is usable in small companies, but as soon as you need to share your data, you need other solutions.
- Team BI: Lets your team access data, the relevant dashboards, report data and maybe even run their own analysis
- Corporate BI: How does your company do in the market? Corporate BI paints the large picture and should ideally give you a direct reading of your companies pulse. But, special needs arise as well, because Corporate BI needs to reach the end-users in the organization. To do this and benefit from it, you need a thorough Information Strategy
Lean BI is faster, simpler and better. Faster because LEAN BI accesses your data directly in the system where they belong, for instance in Microsoft Dynamics. It’s simpler because LEAN BI contains ready to use data models that will get you started doing BI within hours. And it’s better because there are fewer possibilities of errors in the simpler architecture.
With Microsoft and Acinta, you get a BI-solution in as little as half a day at a fraction of the cost of data warehouse BI. If you already have MS SQL-server and Microsoft Dynamics, you just need Acinta to get into lean BI
IM2008 started with a blast from the past and a short video of Borges fighting his way from the office to Scandic Hotel in the beautiful sunny Copenhagen.
Information strategy is the issue and more than 50 experts from Nordic businesses will cover their perspective on managing the flow of information and the development of the business.
The basic message of Michael Borges’ keynote speech was to stop and think. BI is developing extremely fast and today we need to think more holistic about our business. This goes for IM as well. We don’t need more isles of data and fragmented strategies will not be able to transform the company to a real information driven entity.
And the way to do it is to focus on the information strategy and make it a continuum: No surprises, no abrupt changes, but at a steady development of the company’s information and the use thereof.
Michael summarized the current issues in a short list of the 7 deadly sins within IM:
Seven Deadly sins of the Information strategy continuum
Sin #1. Wait for the silver bullet
Waiting for the final solution that will suddenly appear out of nowhere – IT WILL NOT HAPPEN. The need is there now and the challenge will only increase with time.
Nobody will provide a ”killer-app” that will fix all you problems just like that.
Sin # 2. Bite off more than you can chew
Focus ! But remember there are synergies that can be harvested through BI. And about 80 percent of data in any company is unstructured
Sin # 3. Choose a technology focus
Strategy is NOT choosing between Microsoft or SAS, and still 9 out of 10 think this choice is strategy. But it’s important to involve the business, and it may seem like a hassle, but this way we open up for the vast knowledge inherent to the company. But, this too have challenges because the existing organization does not work with the information strategy. Based on that companies can continue to establish the information strategy that comprises policies about how the company wants things to work.
Sin # 4. Allow a process vacuum
You need a transition strategy to get from where you are to where you want to be; it’s clearly not enough to define your goals – you need to think about how you get there. BI never ends; data grows and changes and we need to adapt our business and BI-systems accordingly.
Sin # 5. Ignore execution
“Vision without execution is day dreaming” as bill gates once said. You need to DO, thinking and wondering is clearly not enough. We need strategy yes, but we need to act on it. TIP: If you measure it, it will happen. This is very basic but if you don’t measure your progress, how will you know how to go forward: you don’t even know where you are. Build the program, govern the program, review the program, modify and then start over
Sin # 6. Make nobody accountable
You need someone to take charge of this, because your company needs the focus and this is NOT easy. Also, your company probably sports loads of different opinions on what is good and not so good and there will be wildly differing ideas on the right way for this to work. Make one person accountable – that is the way to push the company forward. And let formal power follow the accountability
Sin # 7. Make information the “be all – end all”
Information is never enough in it self; to get the value of information you need to process it and use it where its useful. And even tough data is extremely important, other parts ot the organization might not even think about the data. You need to be able to understand how the information is being used and what the people with the information prioritizes.
Have you committed a sin? Mr Borges invited anyone who had committed a sin to come to him and make a confession – under a wow of silence naturally.
/Nikolaj Henrichsen, Acinta.dk
Friday, May 16, 2008
As for my self the motivation is a little more - shall we call it - therapeutic. I do write in our business intelligence web site and in my electronic newsletter but both of these media have their scope and some thoughts simple fall outside. So I want to write those thougts here and I truly hope that they can be of any inspiration to you.
Feel free to write to me with your thougts and comments.