Chief BIAS Officer

MapReduce Redux or MapReduce and Flow-based Programming (reprised)

(On request from a reader, this is a reprise of a blog post I did years ago, but was lost in the coming and going of corporate blog content. It is a bit dated, but I believe mostly still relevant, so I am reposting it here.)

Lots of discussion online about MapReduce these days. Stephen Swoyer's piece is quite a good one.

The confusion about what MapReduce is and isn't about actually goes back a ways.

The arguments within computerdom tend to generally be fairly religious in tone. That's because people really like computing and tend to be passionate about it, and when people think they've something new, their ego interest is in showing how new and innovative it is, and the ego-interest of the creators of prior-generation innovations is in casting it as old news.

I am biased, but not working in scalable computing at the moment, so I've had a little bit of time to develop some perspective here I'd like to share.

Scalable computing via parallel processing is big business these days. There's heavy data lifting at the transaction processing level, at the data analysis level, for web search indexing, for enterprise search indexing, scientific computing, data mining for target marketing, affinity advertising. There are many places where data processing of really large amounts of data has commercial benefits, and that means parallel processing adds lots of value.

At the smaller end of the parallel processing scale there's actually a crisis of sorts - we need parallelism to use up all these multi-core CPUs that the chip makers can now create easily, yet there is no mainstream accepted programming idiom that creates the multi-threaded programs yet is easy to use.

MapReduce was designed initially for the other end of the spectrum - huge data, Internet-scale data. Much bigger than enterprise data for even the largest enterprise. This is where clearly you want to use lots of computers all at once. Advocates will tell you it's also great for that downward scale to multi-core CPU thing - and I concur. But that wasn't the first intention for it.

The other big scalable computing systems of commercial import are the relational database engines, and the ETL tools that transform data, and sometimes also support data mining. In common to both of these: commercial data, commercial data types, lots of structured data in the mix, of course with gradually improving capabilities to handle unstructured data. Databases are of course data storage tools, where ETL systems are algorithmic systems. For example, in a database when you talk about how to break up data into partitions you are talking about how it is stored. In the ETL world, partitioning data is about how it is moving, and incidently it can be stored that way also. This latter is very close to the way MapReduce folks talk about data even though I don't think ETL folks and MapReduce folks drink in the same bars.

What all three of these kinds of systems, MapReduce included, have in common is that they are all about scalable Flow-based programming (FBP for short) to some degree. Not everyone agrees on this term flow-based programming, preferring dataflow, or streaming, or flow-graph or some other term. RDBMS systems have their query graphs at their query processing cores - they transform SQL into these. A query graph is a FBP. ETL tools let you draw these graphs (and script them also sometimes). They allow the graphs to have multiple outputs, unlike a query graph, but otherwise are fairly similar. If you use the Apache Pig (so NOT a commericial software system!) system to create MapReduce programs, it cascades the map-reduce operations together into a flow graph. This was created because so many map-reduce applications found themselves wanting to connect one mapreduce back to back into another that someone wisely observed that this flow composition was FBP and that was really part of the idiom.

The primary source of parallelism in all the above is called partitioning. Basically it is the principle that if you have a large piece of data, or large collection of data items, you can divide-and-conquer it, and then put the pieces back together (if necessary). The divided processing is the "map" part, the putting back together is the "reduce" part. You might map and map and map and then reduce, or alternate. Do this kind of thing over and over as you process the data, flowing the output of one such operation to the input of another. 

So here's one major point of total agreement: There's a agreement across many different areas of computerdom that flow-based programming is a very good way to go if you want parallelism. I think people would also say the idiom is pretty easy to learn, adequately expressive for many problems, and so forth.

That's very important common ground. If you are a software developer and can say you've done object-oriented programming and you've used threads or actor libraries or some such, you may feel relatively accomplished at your trade. If you are not directly familiar with this FBP idiom it is something you should learn about, because you will likely find it very valuable, and it will change the way you think about solving problems.

FBP principles are where the similarities end among these kinds of data processing systems though. There are huge differences about what these systems are designed to be used for.

Now, advocates of each seem to have some "hammer vs. nail" mentality - By that I mean if you have a big enough hammer, *everything* starts to look like a nail. Example: extending an RDBMS to do what it does, but also handle spatial data, or blobs of image data, or real-time streaming input data, or transactions from a queue, or sensor data feeds. You pick it. Extending a RDBMS hammer to hit these kinds of nails, this is a very attractive option to those familiar with RDBMS technology, and particularly if they have access to the source code for one. Same for ETL tools. I was chief architect for a major commercial ETL processing engine, and we tried to extend it in all kinds of stretch directions. MapReduce advocates would look at these same problems and recognize the maps and reduces that are being cascaded to form the solutions to them.

So, what separates these technologies?

I claim the first thing is maximum scale. But what is a big enough scale to start telling these apart?

Ok, here's a problem suitable only for MapReduce today: Let's play where's Waldo and find every video frame containing my face, in every video CCTV surveilance capture everywhere on earth for the last year. You have 60 seconds to find the answer.

That is the kind of problem map-reduce can scale up to. The data is petabytes or exabytes of data. Now, the actual where's Waldo search might be very cleverly indexed, so I probably do not have to stream all these exabytes of video frame by frame to look for my face. But at some point every frame was processed to create whatever index answers this problem. That processing scale is truly vast.

None of the other technologies were built with anything like this scale in mind. Yes there are FBP principles of operation which are in common. The distinction is what frameworks have been built to actually do.

What about ETL or RDBMS technology? What's suited to them that crisply distinguishes them from MapReduce frameworks. Answer: processing structured data records repeatedly. I'll characterize what I mean by this, but first let me talk about some of the design limitations of RDBMS and ETL technologies.

First the problems are ones at more traditional enterprise scale. These days that's multiple terabytes of information, maybe even 100 terabytes now adays. One assumption common to these systems is that the whole computing cluster across which the work is spread is assumed to be up and working. Failure of any part of the cluster is expected to stop the processing. There are various checkpoint/recovery schemes which are pretty much the cutting edge of these systems these days. These always assume a failure is a relatively rare event. Most executions will complete without failure.

There's a thing I call a time/complexity trade off at work here. Which would you rather have? A system that is slower, and more complex, but which takes twice as long to compute something thereby doubling the likelyhood of encountering a failure during a run? Or would you prefer a faster system without the complexity? Computers are fairly reliable these days, so at the scale of a cluster of computers small enough where I can reasonably expect them all to all be running properly for hours at a time... Well I know my own preference here and that of many "enterprise" customers matches my own - simple wins.

MapReduce makes no such assumptions about failures. The implemented scale is assumed to include computers that will fail during the execution of a flow, and the computation must continue and mask the failure.

So here's a real and concrete question: Does your processing problem have the vastness to it that means the compute cluster will have failures in it most of the time, or is the processing scale such that the cluster is very unlikely to have a failure in it that interrupts processing? If the former, MapReduce is the only viable technology today. If the latter, there are still reasons you might want to use a MapReduce-like means.

A big difference between MapReduce systems I've looked at and the enterprise RDBMS and ETL world is efficiency of data access vs. complexity.

RDBMS and ETL are optimized for handling business-oriented data. Quite literally, I worked on engineering an ETL engine so that when a processing component accessed an integer from a data record, the C++-based implementation would generate code that would compile into a base+offset-indirection - really our goal, and we got very close, was for this to be a single CPU instruction.  The assumption here: you are accessing billions of rows of well behaved small data that has been imported into a well behaved representation, bringing it in once from whatever it was before you started processing it. By the way, getting this super low level efficiency introduces lots of complexity. Many would rightfully say this was "wicked over-engineered" - mea culpa. Databases go even further and try to apply cost-based dynamic programming optimizers, move things around in the flows to reduce work done. Very cool stuff. Actually one of the first applications of ETL tools was actually a backlash to the unpredictability of SQL database optimizers. People wanted to draw a flow and have it run that way dammit! That was 1995. The optimizers have improved, but ETL is important for other more fundamental capabilities like superior expressive power. ETL tools can produce multiple outputs from a single input in one pass. SQL is a query language and is incapable of expressing this directly even though the underlying engines could easily implement this.

Anyway, all this low-level data access optimization and high-level flow-graph optimization, this kind of thing is just not considered particularly important by the MapReduce advocates. You process data in the form it was found, which is assumed to be an HTML document, XML data file, or perhaps JSON. Perhaps other forms that you happen to have some code to handle. Really it's a text-intensive world. These formats are all very verbose textual formats full of tag syntax. Verbose by comparison with carefully layed out binary data records is what I mean.  It takes thousands, of machine instructions to extract a piece of data like an integer from such representations because you have to find it first (parse and/or search for tag), then convert text to integer.

But there is absolutely nothing wrong with that overhead. Because accessing small numeric fields is just not what the MapReduce world is about. Consider this: in RDBMS and ETL worlds, if you access a string, there's low overhead to get to the start of the string, and then you're processing the string data using the usual libraries available in the programming language of your implementation.

In MapReduce, there's high overhead to get to the start of the string you care about, but then you're processing the string data using the usual libraries available in the programming language of your implementation.

Get my point? The only place where the RDBMS/ETL world has advantage is if the strings are really small and must be repeatedly accessed. In many many very interesting problems, the strings are not at all small, and many problems are "one-pass" over the data by each algorithm.

So the conclusion should not be surprising: it's "horses for courses" - none of these technologies is superior to the other. They all share flow-based programming principles - they jury is back, flow-based programming is a GOOD IDEA whose time has come and it is being used all over the place. Beyond that, what kind of processing are you doing? What kind of data is at its core, and at what scale?

Tags: BI, Business Intelligence, data integration, data warehouse, MapReduce, Scalable Computing, Parallel Processing, Parallelism, multi-core

Daffodil: An Open Source DFDL Implementation

Well this blog went entirely silent because I got heads down into Daffodil, an open source implementation of the Data Format Description Language (DFDL), while working full time for Tresys Technology.

Update: Daffodil has migrated from roots at NCSA to becoming Apache Daffodil.

This means there is now a commercial DFDL implementation from IBM which is part of their impressive WebSphere Message Broker product, as well as now an open source implementation. This is important because the standards organization of which the DFDL effort is part, the Open Grid Forum, requires 2 implementations in order to move forward with the standardization process.

IBM's product has been in the marketplace for several years now. The Daffodil open source implementation is now at release 2.1.0. It doesn't have every bit of DFDL in it, as it is a large specification and of necessity is being implemented in stages, but it will be plenty rich enough for real applications and difficult and complex data formats. DFDL schemas for many formats are published on the DFDLSchemas site on github. 

The open source Daffodil DFDL implementation is written in a very cool programming language called Scala which runs on the Java virtual machine and shares the Java technology stack. I'm a huge fan of Scala now. I am much more productive in this language than any time since I last worked in Lisp, which was ...egad... back in the 1980's.  Scala has substantial advantages over Lisp, in that it is strongly typed. Yet it remains flexible and easy to use. 

The Daffodil code base is beyond 300,000 lines of Scala code and assocaited tests. (Here's what OpenHub has to say about the code base.) It includes a large suite of conformance tests. It's a serious code base with devoted developers and QA/testers working on it. 

For a quick introduction to DFDL, what it is, and its purpose, you can read Intro to DFDL post, or check out the Wikipedia article on DFDL, or go to the Open Grid Forum DFDL site.

 

DFDL Accelerated - and GMTFD!

I co-chair this standard about Data Formats, describing them to insure interoperability and such. I posted about this previously.

I just wanted to advise folks that DFDL (Data Format Description Language) efforts are being accelerated by major vendors of late.

Update (Jan 2011): DFDL is now an official proposed standard. See www.ogf.org/dfdl.

So this standard is no longer languishing in the slow efforts of devoted committee members working extra hours outside their day jobs. People are really being paid now to make this reality.

I am frankly thrilled to see this finally happen. DFDL is important to interoperability, high performance computing, and bridging the very rich legacy of computing to the future where the data will be everywhere providing new value to businesses!

Just extracting data from many systems is HARD. That's where DFDL comes in. 

Reality: most businesses need to give data over to outside organizations who will give back to them information providing valuable business insight. Many organizations simply have failure of imagination - you can't anticipate everything that can be learned from your data.

Warning/Caveats/Alert: Keep data privacy rules and security in mind....but outside of that, let the data go!

GMTFD is my new acronym. By analogy to the now legendary (at least on T-shirts) acronym RTFM,  my acronym stands for "Give Me the Freakin' Data".

Given data, smart outside organizations can provide businesses with just incredible insights that would simply be unavailable if they tried to do this work in-house. (www.oco-inc.com)

Open Data Standards - Relay to New Blog

I remain the Chief Bias Officer, biased because I believe in what I do, but unbiased in that my opinions are not strictly corporate or self-serving. I keep my clients interest at heart. That said, there are contexts where my opinion is clearly that of my company. Where both my Bias officer opinion and my opinion as Oco CTO are aligned. In those cases I'll post on the Oco IT blog. Please check out this post: 

Open Standards, SaaS Business Intelligence, SQL, and Cloud Computing

It's about the whole "lock in" fear of SaaS, recently popularized by The Economist . This fear is not really justified so long as you ask your providers to support open data standards.

Predictions for 2009 for Business Intelligence

Time to predict the future a bit. The trends we see in the evolution of Business Intelligence are going to be amplified by the current economic climate.

1.     Growth of Complementary BI Solutions

Companies have long attempted to standardize using a single application or approach to address all of their technology needs in a specific area.  However, BI in particular, cannot be effectively implemented using a single standardized approach.  There are too many different capabilities, vertical-specific requirements and deployment models to be restricted to one standard BI tool from a single vendor.  Driven by diverse organizational needs, a growing number of companies will recognize that one size does not fit all, and turn to a set of distinctive BI solutions with varying capabilities from different vendors to address a range of BI requirements. 

2.     SaaS Goes Upmarket

Software-as-a-Service (SaaS) has provided an attractive proposition to small and mid-sized companies, yet large, multinational enterprises are also realizing the benefits of rapidly deployed solutions that require minimal ongoing IT support.  These large firms are turning to SaaS BI solutions to complement their existing BI infrastructure.  The SaaS-based BI approach will significantly proliferate among large, multi-billion-dollar companies in 2009 and beyond.

3.     Information Silos Will Tumble

As the different functions within an organization become more interrelated, the insight needed to make effective business decisions doesn’t come from one single area.    Analytical decision-making requires that information be gathered from many different functions across an enterprise, which is a difficult task when data ownership resides in individual systems.  As decision-making continues to become more cross-functional, data silos will break down.  For example, to gain an understanding of at-risk customer revenue, companies may need to pull actual customer revenue from finance, forecast revenue from sales, customer satisfaction metrics from marketing, and on-time shipment information from supply chain management. 

4.     Collaborative BI Throughout the Extended Enterprise

As analytic solutions become richer and more powerful, they will reach outside the firewall to a company’s customers, suppliers and service providers.  This extended enterprise will both contribute data to the solution, and also use the analytics that it offers.  Reaching into the extended enterprise has been a priority for years—now measuring and making business decisions based on the same set of analytics will become pervasive.   These solutions will be significantly enriched in 2009, driving value throughout the ecosystem. 

5.     Corporate Hot Spots Will Drive BI

In the recession, companies will choose IT initiatives that address specific business “hot spots” and offer a quick payback. Initiatives focusing on increasing revenue, reducing costs or improving customer satisfaction will reign.  BI will provide organizations with visibility into opportunities to cross-sell into existing customers, ensure at-risk revenue is secured, identify new opportunities, rationalize costs and inventory, and meet goals set out by leadership.  Companies are targeting a 3x ROI and payback in 90 days or less. 

Solution is to Tool as Business Genomics is to Relational Databases

I struggle to explain to people what it is that is unique about Oco and what we do that is different from all the other "SaaS BI" companies, which are rushing into the playing field.

I always try to say something which briefly summarized is "It's the business semantics stupid!". I've harped on Oco does solutions, everyone else is doing just tools.

Ok, that's a bit polemical, but the point here is that what we do is VERY different, ... but what is the difference between a solution and a tool anyway?

Here goes....

Oco was founded (back in 1999) by a gentleman named George O'Connor (hence the name. ... yup. I'm naming my next company after myself too.)

George was in a very interesting situation which allowed him to make a rather unique insight which is just blindingly obvious in hindsight, but is really quite profound. So in order to completely slap you in the face with it, let me digress.

Suppose you are at the 50,000 foot level. You are IBM, or Oracle. You are trying to decide how to understand all the data in businesses. If you are looking across dozens or hundreds or thousands of businesses, then you would have to adopt a computer science kind of posture and say to yourself "I think we need a relational database system here. All these businesses can represent everything they're doing in that common framework and then we'll be able to exploit commonalities". You would be right of course. Most businesses today use lots of relational database systems, and they are the tool of choice to represent business information.

Now suppose you are looking at just a single business, trying to decide how to understand all its data. You would adopt a very precise focus and say "I think our orders can be a table with columns for the amount of the order, tax, who sold it, what division they are part of..." and so forth. You would create something very tailored to what you need for just that business.

Ok, now here's the point. George O'Conor worked for a company that owned 10 others that he was responsible for. He had exactly 10 companies to look at. Not one, not hundreds, but ten.

This changes everything. There aren't too many jobs where you are responsible for 10.

His insight was this. Among the 10 companies, there are not an infinite number of different ways they need to represent their orders. There are only 3. Their shipments have only 2 variations actually, their pricing policies - 4 different kinds.

At the perspective of 10, you don't invent something as general as the relational database. You look at the common business behaviors, and you invent software that represents each of the different behaviors. You don't need infinite flexibility, you need limited flexibility.

So far so good, but why was this article title about "genomics" anyway.

Well, we all know all life is based on DNA, and there's a gazillion base pairs organized into genes, and there is the potential there for infinite variability.

Yet, how many different human eye colors are there? Answer: it's not an infinite number of variations, it is 6. They are Amber, Blue, Brown, Gray, Green and Hazel. (Sorry Liz. Wikipedia says violet is just not real!)

Just 6.

The database of DNA supports infinite variability, yet we humans boil down to exatly 6 variations.

Could that be because our environment didn't NEED more than 6 variations. Certainly had it been to evolutionary advantage there'd be many more colors, but 6 proves to be enough for us humans even in our complex social world. In fact having too many variations would be problematic. It is comforting to us to see similarities across people. Not uniformity, but not total diversity either.

And so it is with businesses. In businesses today, how many different pricing strategies are really out there being practiced? How many different ways to manage orders. A relational database can represent, like DNA, an infinite number of variations on business concepts, yet businesses really only do any given business behavior a few different ways. And this is because competition hasn't required more variations. And, if every business did each of these behaviors entirely differently.... well it would be awfully hard to hire experienced help. It's useful for us that there aren't so many variations.

In other words: There are lots of better ways to make money than constantly inventing different pricing strategies for your business. 'nuf said.

Bruce Richardson from AMR coined this idea as "Oco discovers the business genome". It is a very appropriate analogy.

So for pricing, there's some variations, but not so many. For order management, again a few variations. Inventory, again a few. Combining them together there's lots of combined variations, but in each individual area, we could call it each business gene, there's not so many variations.

Just like hair color, eye color, etc.

This is truly compelling and entirely obvious once you see it. Yet people tend to believe that businesses just are full of endless variations of practices and what they do is unique in so many ways. And it simply isn't the case.

We exploit this big time at Oco.

Oh yeah, the ERP companies do also. Funny that.

How do we take advantage at Oco? Well we have a central database design in our product which can represent a few variations on say, pick pricing again. Will we change it if we find another variation? Sure we will, but we don't expect to do that often now. If you can cover the variations we have now, we think it's likely we can already handle many other variations.

Here's the sweet part: All our tools can exploit the fact that this database is quite stable in theme and design. We don't have to transform data from what the business has in hand to whatever database design is created for a particular project. We've got a fixed database design. That whole variable drops out of the equation.

Back to Solution vs. Tool.

A tool is like a relational database. It does whatever you want to do with it.

A solution has more of the genome built up in it. It's closer to a living thing, a living business.

To tell them apart, well if you are considering a BI system, ask the vendor what they know about pricing for your industry? Do they recommend managing orders (i.e., what customers buy from you) and purchases (what you buy from others) the same way, or differently? Heck, can they even comment on these topics? What about transportation logistics? Can they comment on fuel surcharges, or why trucking is different from rail?

If the answer is "however you want to do it", then you are purchasing a tool, not a solution.

 

EII, ETL, ELT, EAI ... Enough!

EII, ETL, ELT, EAI,....This alphabet soup for integration technologies got created by some industry analysts. I've spent much of my career in this Exx soup, and so have become quite expert in what these things are and are not.

EII stands for Enterprise Information Integration, which doesn't say at all what it is.

EII is SQL Database Federation - it is viewing the problem of data integration using the SQL perspective. That is, what if we could make all data look like it was in one big RDBMS, even though it isn't. Then we'd have a uniform way to query it.

This idea is great really. Viewing all data through the lens of SQL. It is great for query and analysis types of applications. Not so great for transactional processing though there are products in the market which federate and implement distributed two-phase commit even. But nobody would claim it's a high-performance system.

The problems with EII start with expressiveness - SQL's type system doesn't handle hierarchical data. Or rather, standard SQL doesn't. Unfortunately, an industry consensus never formed around nested tables or any of the other ideas for dealing with hierarchical data from SQL.

EII now includes XQuery (which is SQL-like, but does deal with hierarchy), and systems that view all data as XML.

Another problem with EII is incompleteness. EII is about how you access the data once it's been made syntactically compatible, but it doesn't address how that is achieved. You need another integration technology of some sort to turn non-RDBMS or non-XML data sources into ones that an EII environment can manipulate.

Finally there are performance issues with EII. As good as SQL optimization technology is, you still can't ignore that federated databases aren't one database, so joins across two separate databases are going to be very expensive. Optimizing XQuery is a research topic still.

Given how inexpensive databases and storage are these days, a commercially viable EII approach is to gather all data into a central database. That is, forget about the federation aspect and just centralize so as to bring SQL to bear in the environment where it performs best.

Generally, EII is a good example of an adage I like: "If you've got a really big hammer, everything starts looking like a nail" :-)

So much for EII.

Now, ETL - Extract Transform, and Load. This is the technology used to populate databases from non-databases. It is usually batch, but is being adapted to more real-time things.

ETL tools are usually based on a principle called streaming dataflow, as are the SQL-engines inside databases, but ETL tools are more extensible. E.g., in SQL you can aggregate data with a group-by statement and compute sums, counts, averages, etc. But in an ETL tool you can build a best of breed aggregator using your own custom business specific algorithm. E.g., given all the records representing a customer, take fields from one or another so as to get the best record representing the customer. So, consider the gender field in 10 records representing the same  customer, suppose it is blank in a few, Male in a few, Female in a few. The name is "Dana", which doesn't help. So, a business specific heuristic can be defined that says if a particular record comes from say, the brokerage division of a financial house, then that's more likely to have the gender correct because the brokers actually talk to the customers on the phone. Try expressing that in SQL!

The streaming dataflow that ETL tools use as a basis is a natural candidate for parallel processing to increase throughput. This can be on clustered computers, or just a multi-core CPU. This idiom is one candidate for fixing the problem where programmers don't know how to exploit all these cheaply available CPU cores.

ELT is an alternative to ETL. It's more SQL oriented. It stands for Extract, Load, then Transform.  You extract data. Do to it only what has to be done to it to load it into a database, then transform by using a number of SQL-based operations on it. This works when SQL is sufficient to express what needs to be done. In that sense it is like EII.

All of the three so far ETL, EII, ELT, are what we call set-oriented systems. Aggregating a set of multiple separate records together to compute a collective result from the set makes sense. Sorting a set of records makes sense so that you can examine them in a particular order. All of these are suitable for massive-scale processing when data is large.

Now let's look at EAI - Enterprise Application Integration. This is the stuff of enterprise message buses. it's about processing individual messages so as to perform a transaction not only in one place, but in all the places that the information that transaction contains is needed.

EAI systems are primarily about the processing of one message at a time. not necessarily sequentially, but the logic of an EAI system isn't about processing groups of messages together. The idea of taking the set of messages, sorting it, doing best of breed record construction, is not part of the EAI mind set.  Rather, the transformation and routing of a message to the right places and in the right formats is what it is about.

EAI's message by message approach can be scaled up massively also, but can't do the same kinds of things that a set-oriented system can do across records/messages. Or rather it can't do them the same way. To carry out a transformation involving multiple messages in an EAI system you process one record and keep a state in a data store, typically a database. When the next message of the set arrives, you retrieve the data from the database, compute more, write the state back to the database. Repeat until you've combined all the messages. (Figuring out which is the last one is a big issue.) Ultimately the answer comes from the last message and its interaction with the state, which is where the accumulated knowledge sits about the collective set of messages.

But here's the rub with EAI. When you have messages where there is complex hierarchical data e.g., an XML document, then sometimes the transformations on the detail levels of the hierarchy are set-oriented. E.g., given a purchase order, what is the total of all the parts being ordered. This is not an item-by-item operation on each line-item of the purchase order, it is an operation on the whole set of line-items for a purchase order.

So an EAI system needs to be able to transition from the message by message model of transformation and routing into the set-oriented notion for dealing with the hierarchical content of those messages. This set-oriented stuff is much more like what is typically and readily expressed in ETL/EII systems.

Lately some new acronyms have been added to the soup. (They don't begin with "E" thankfully)One is CEP - complex event processing. This is EAI with different means of expressing the processing using rule languages. Another thing (no acronym) is sensor network streams - this is similar to the real-time ETL systems, but emphasizes that the messages are coming from sensors and the input rate is so fast that it might even be ok to lose a message here or there if needed to keep up overall.

So, my conclusion is that the alphabet soup of EII, ETL, EAI, ELT, isn't really very useful anymore. You have SQL-based approaches, and non-SQL approaches. You need more than the expressive power of just SQL in many cases, and you need more than message-by-message processing as well.

An ideal system has to have the expressive power of the ETL systems, but be designed for always-on online applications, not just batch. It must handle hierarchical data naturally. It must be extensible to allow ad-hoc aggregations, not just the fixed library of SQL or XQuery. It must make simple online message by message processing very simple to express.

But... to my knowledge nobody makes one of those. They're all stuck in one of the Exx boxes and provide only what that one Exx box narrowly provides.

Maybe some open source provider, someone who doesn't care about the Exx categories, will fix this someday.

Internet Data Center vs. Your PC - Which is more wasteful?

My prior post Your Computation Mileage May Vary generated some interesting hallway discussion.

I believe my intended point is valid still which is there is no upper bound in the demand for computational services. My article didn't really make its point too well, which is more about the fact that the computation cost is kind of like rent with utilities included. No incentive to conserve.  

But, I need to clarify that it was not my intent to accuse the Internet search companies of being wasteful of resources.  Not at all.

One issue is the matter of what Internet searching replaces. For example, Ebay vs. trucking newsprint around in order to occasionally reach my eyeballs with a classified ad or two. Hauling newspaper isn't exactly smart use of resources either.

So by way of setting the record straight on Internet search energy use, colleagues pointed me to some of the facts about this subject. Here are some notes on estimates for energy consumption per search query:

• Standard server-class machines are generally not used. Instead, much more energy-efficient designs are used (careful power distribution, power supply, motherboard design, etc.
• There is a lot of room to do cooling more efficiently than is usually done.
• The "thousands" of computers involved in serving search queries are not all involved for each query. (This part I already knew.)
• Searches take far less than 1 second.

Clearly, if it took anywhere near the energy my blog post suggested to run a search engine, companies like Yahoo, Google, etc. wouldn't be able to make (much) money through search advertising.

Google has previously published some information about energy efficiency, here are a couple:

  http://services.google.com/blog_resources/PSU_white_paper.pdf

The above points out that every single PC has an incredibly wasteful power supply in it. Collectively, that's a lot of wasted energy. After reading the above I feel good about buying laptops! 

  http://googleblog.blogspot.com/2008/10/saving-electricity-one-data-center-at.html 

The above says "In the time it takes to do a Google search, your own personal computer will use more energy than Google will use to answer your query."

From my understanding of Internet search technology, and looking at what we can find out publicly about Google, this is entirely credible. Really it is great to know that Google has been successful at lowering the impact of all this great computation capability they're providing.
 

Why SaaS is Good for IT vs. Why SaaS Isn't Good for IT

I've heard this argument back and forth from a number of people, so it is worth commenting on, particularly as related to business intelligence systems. That is data warehouses and the like.

Here's the positive themes: cost savings, freeing up resources for other strategic projects, improved quality of BI information and improved distribution of BI information, improved coverage of business units.

Here's the negatives: risk of reduced budget, risk of career disruption/change.

The net is that IT folks have to view SaaS as "good" for them because (a) it is good for the business (b) it is inevitable.

Let me elaborate.

The basic argument for SaaS in general is just the Adam Smith economic division of labor argument, i.e., you should make use of services instead of being self-sufficient for the same reason you buy food at the supermarket instead of farming your own - it is more efficient for a specialized producer to do it than for individuals to each do something they do not have the specialized skills for. That efficiency is returned to us in lowered cost, and increased reliability of supply. For IT systems this argument is presented really well in The Big Switch, which I highly recommend.

Contrary-wise this implies that if you have computing needs for which services aren't available, then by all means you should do those in house. This goes all the way down to writing your own software if software isn't available to do what you need. This is where the "good" part of the SaaS trend for IT comes from. It's the ability to refocus efforts on things that are strategic, not commodity.

For SaaS BI, more specifically, well it is good for the business because it saves cost over doing it in-house, and in most cases we at Oco can do it far better than an in-house project would be able to do. The cost reduction can actually enable BI to be available for the first time at many businesses which could not afford to do it before on the price scales that larger enterprises have been paying.

However, for businesses that replace an in-house effort with SaaS BI, whether it is "good" for IT depends on what is done with the savings. If these are re-invested in other strategic IT projects then it creates ongoing career opportunities for IT folks. However, if those savings are just used to increase profits and/or pay dividends to shareholders, then this good for the business is at best neutral for IT folks working in the trenches, and possibly very negative. Change can be opportunity, but always presents challenges.

If a new IT technology is good for the business, then strategic IT departments have to find a way to embrace it and make it successful for the business. It's untenable for an IT department to push back on important technology for "turf" reasons alone.

The career plans of people that are vested today in in-house creation of BI/DW systems clearly need to evolve to match the new reality. E.g., if you are an IT leader who wants a successful DW/BI deployment on your resume as part of your career plans, well you have to consider whether it would look better to have a SaaS deployment of BI on your resume instead. A SaaS BI deployment should save your organization lots of money, so perhaps you would also be able to feature the other projects you can now afford to do with the budget savings.

If you are a data analyst, then by definition your company already sees the importance of having data analysts, and having a SaaS BI deployment should just let you move up the value chain to deliver the BI information that goes beyond what you can get from your Oco system, or which explores the directions for enhancements of your SaaS-deployed BI solution. 

If you are an IT worker in the trenches, such as a DBA or systems administrator hoping to work on a DW/BI deployment, then there certainly is a risk of a disruption of your plans due to SaaS. I'm not sure I can sweeten this for you. As IT switches over to services and away from in-house deployments, then your job and role is going to change. You really do have to read The Big Switch, and should consider looking for employment on the services side, that is with the service providers. If you stick within your existing business/employer, you should look for something you can work on that is more strategic, i.e., that is related to a system that is very specific to your business, and not common across that business and many others. I can't find the origin of this quote, but the point bluntly is to "get strategic or get outsourced".

There's another way in which an inevitable change can be "good" for IT. The inevitability is basically saying that since it's coming, it's "good" to be a thought leader and embrace the change and get some advantage from it before everyone else has those advantages as well, at which point you are just playing catch-up. This argument is also known as "the best defense is a good offense".

The alternative is to fight the trend, rather than switch, but this is just burying one's head in the sand, to mix in another metaphor.

SaaS BI Going Mainstream: What Business Objects OnDemand + Oco Partnership Means to Me

Today Oco announced a partnership with Business Objects. I'm not going to reiterate what the announcement says since you can read that elsewhere. To me the key point here is that a major trusted BI supplier, Business Objects, is (and has been for a while now) heavily investing in OnDemand/SaaS/hosted deployment, and they've recognized that Oco's solution provides value for their customers through the integrated information content Oco creates accessed by way of the Business Objects OnDemand hosted environment.

To Oco this partnership is great because it legitimizes what we do and really reduces the shock-factor that our prospective customers see. Let's face it. Oco is an innovator (read "small" -- though growing!) company which by itself sells a solution which is delivered "shockingly" quickly (6 to 10 weeks), financial "risk free", delivered via a "radical" new deployment model (SaaS), and it provides only a UI targeted at the business-user. This is all pretty unfamiliar stuff for a conservative BI customer. Keep in mind that the track record on Data Warehouse/BI projects is historically pretty poor - hence many people have become conservative about them.

Business Objects changes the balance considerably: You replace the small company risk-factor with a major trusted supplier like Business Objects. You replace this radical SaaS deployment model with the fact that Business Objects OnDemand represents this SaaS trend going mainstream. Finally, you get what I like to call "design headroom" that the broad suite of Business Objects tools offers which caters to the complete user community that traditionally consumes BI spanning from the business users (who depend on Crystal Reports and Xcelsius dashboards) to data analysts performing ad-hoc analysis (for example via Web Intelligence).

What was shocking, new, and risky, is now a pretty safe bet, and customers can focus on what the new value is that the solution provides, not on all the seemingly risky new-ness of the way it is created and delivered.

My thanks go out to my colleagues at Business Objects who examined Oco, saw the value here, and worked to add us into their partnership sphere. Everyone at Oco and myself will be working hard to make sure we deliver great solutions to our joint customers.