Chief BIAS Officer

Solution is to Tool as Business Genomics is to Relational Databases

I struggle to explain to people what it is that is unique about Oco and what we do that is different from all the other "SaaS BI" companies, which are rushing into the playing field.

I always try to say something which briefly summarized is "It's the business semantics stupid!". I've harped on Oco does solutions, everyone else is doing just tools.

Ok, that's a bit polemical, but the point here is that what we do is VERY different, ... but what is the difference between a solution and a tool anyway?

Here goes....

Oco was founded (back in 1999) by a gentleman named George O'Connor (hence the name. ... yup. I'm naming my next company after myself too.)

George was in a very interesting situation which allowed him to make a rather unique insight which is just blindingly obvious in hindsight, but is really quite profound. So in order to completely slap you in the face with it, let me digress.

Suppose you are at the 50,000 foot level. You are IBM, or Oracle. You are trying to decide how to understand all the data in businesses. If you are looking across dozens or hundreds or thousands of businesses, then you would have to adopt a computer science kind of posture and say to yourself "I think we need a relational database system here. All these businesses can represent everything they're doing in that common framework and then we'll be able to exploit commonalities". You would be right of course. Most businesses today use lots of relational database systems, and they are the tool of choice to represent business information.

Now suppose you are looking at just a single business, trying to decide how to understand all its data. You would adopt a very precise focus and say "I think our orders can be a table with columns for the amount of the order, tax, who sold it, what division they are part of..." and so forth. You would create something very tailored to what you need for just that business.

Ok, now here's the point. George O'Conor worked for a company that owned 10 others that he was responsible for. He had exactly 10 companies to look at. Not one, not hundreds, but ten.

This changes everything. There aren't too many jobs where you are responsible for 10.

His insight was this. Among the 10 companies, there are not an infinite number of different ways they need to represent their orders. There are only 3. Their shipments have only 2 variations actually, their pricing policies - 4 different kinds.

At the perspective of 10, you don't invent something as general as the relational database. You look at the common business behaviors, and you invent software that represents each of the different behaviors. You don't need infinite flexibility, you need limited flexibility.

So far so good, but why was this article title about "genomics" anyway.

Well, we all know all life is based on DNA, and there's a gazillion base pairs organized into genes, and there is the potential there for infinite variability.

Yet, how many different human eye colors are there? Answer: it's not an infinite number of variations, it is 6. They are Amber, Blue, Brown, Gray, Green and Hazel. (Sorry Liz. Wikipedia says violet is just not real!)

Just 6.

The database of DNA supports infinite variability, yet we humans boil down to exatly 6 variations.

Could that be because our environment didn't NEED more than 6 variations. Certainly had it been to evolutionary advantage there'd be many more colors, but 6 proves to be enough for us humans even in our complex social world. In fact having too many variations would be problematic. It is comforting to us to see similarities across people. Not uniformity, but not total diversity either.

And so it is with businesses. In businesses today, how many different pricing strategies are really out there being practiced? How many different ways to manage orders. A relational database can represent, like DNA, an infinite number of variations on business concepts, yet businesses really only do any given business behavior a few different ways. And this is because competition hasn't required more variations. And, if every business did each of these behaviors entirely differently.... well it would be awfully hard to hire experienced help. It's useful for us that there aren't so many variations.

In other words: There are lots of better ways to make money than constantly inventing different pricing strategies for your business. 'nuf said.

Bruce Richardson from AMR coined this idea as "Oco discovers the business genome". It is a very appropriate analogy.

So for pricing, there's some variations, but not so many. For order management, again a few variations. Inventory, again a few. Combining them together there's lots of combined variations, but in each individual area, we could call it each business gene, there's not so many variations.

Just like hair color, eye color, etc.

This is truly compelling and entirely obvious once you see it. Yet people tend to believe that businesses just are full of endless variations of practices and what they do is unique in so many ways. And it simply isn't the case.

We exploit this big time at Oco.

Oh yeah, the ERP companies do also. Funny that.

How do we take advantage at Oco? Well we have a central database design in our product which can represent a few variations on say, pick pricing again. Will we change it if we find another variation? Sure we will, but we don't expect to do that often now. If you can cover the variations we have now, we think it's likely we can already handle many other variations.

Here's the sweet part: All our tools can exploit the fact that this database is quite stable in theme and design. We don't have to transform data from what the business has in hand to whatever database design is created for a particular project. We've got a fixed database design. That whole variable drops out of the equation.

Back to Solution vs. Tool.

A tool is like a relational database. It does whatever you want to do with it.

A solution has more of the genome built up in it. It's closer to a living thing, a living business.

To tell them apart, well if you are considering a BI system, ask the vendor what they know about pricing for your industry? Do they recommend managing orders (i.e., what customers buy from you) and purchases (what you buy from others) the same way, or differently? Heck, can they even comment on these topics? What about transportation logistics? Can they comment on fuel surcharges, or why trucking is different from rail?

If the answer is "however you want to do it", then you are purchasing a tool, not a solution.

 

EII, ETL, ELT, EAI ... Enough!

EII, ETL, ELT, EAI,....This alphabet soup for integration technologies got created by some industry analysts. I've spent much of my career in this Exx soup, and so have become quite expert in what these things are and are not.

EII stands for Enterprise Information Integration, which doesn't say at all what it is.

EII is SQL Database Federation - it is viewing the problem of data integration using the SQL perspective. That is, what if we could make all data look like it was in one big RDBMS, even though it isn't. Then we'd have a uniform way to query it.

This idea is great really. Viewing all data through the lens of SQL. It is great for query and analysis types of applications. Not so great for transactional processing though there are products in the market which federate and implement distributed two-phase commit even. But nobody would claim it's a high-performance system.

The problems with EII start with expressiveness - SQL's type system doesn't handle hierarchical data. Or rather, standard SQL doesn't. Unfortunately, an industry consensus never formed around nested tables or any of the other ideas for dealing with hierarchical data from SQL.

EII now includes XQuery (which is SQL-like, but does deal with hierarchy), and systems that view all data as XML.

Another problem with EII is incompleteness. EII is about how you access the data once it's been made syntactically compatible, but it doesn't address how that is achieved. You need another integration technology of some sort to turn non-RDBMS or non-XML data sources into ones that an EII environment can manipulate.

Finally there are performance issues with EII. As good as SQL optimization technology is, you still can't ignore that federated databases aren't one database, so joins across two separate databases are going to be very expensive. Optimizing XQuery is a research topic still.

Given how inexpensive databases and storage are these days, a commercially viable EII approach is to gather all data into a central database. That is, forget about the federation aspect and just centralize so as to bring SQL to bear in the environment where it performs best.

Generally, EII is a good example of an adage I like: "If you've got a really big hammer, everything starts looking like a nail" :-)

So much for EII.

Now, ETL - Extract Transform, and Load. This is the technology used to populate databases from non-databases. It is usually batch, but is being adapted to more real-time things.

ETL tools are usually based on a principle called streaming dataflow, as are the SQL-engines inside databases, but ETL tools are more extensible. E.g., in SQL you can aggregate data with a group-by statement and compute sums, counts, averages, etc. But in an ETL tool you can build a best of breed aggregator using your own custom business specific algorithm. E.g., given all the records representing a customer, take fields from one or another so as to get the best record representing the customer. So, consider the gender field in 10 records representing the same  customer, suppose it is blank in a few, Male in a few, Female in a few. The name is "Dana", which doesn't help. So, a business specific heuristic can be defined that says if a particular record comes from say, the brokerage division of a financial house, then that's more likely to have the gender correct because the brokers actually talk to the customers on the phone. Try expressing that in SQL!

The streaming dataflow that ETL tools use as a basis is a natural candidate for parallel processing to increase throughput. This can be on clustered computers, or just a multi-core CPU. This idiom is one candidate for fixing the problem where programmers don't know how to exploit all these cheaply available CPU cores.

ELT is an alternative to ETL. It's more SQL oriented. It stands for Extract, Load, then Transform.  You extract data. Do to it only what has to be done to it to load it into a database, then transform by using a number of SQL-based operations on it. This works when SQL is sufficient to express what needs to be done. In that sense it is like EII.

All of the three so far ETL, EII, ELT, are what we call set-oriented systems. Aggregating a set of multiple separate records together to compute a collective result from the set makes sense. Sorting a set of records makes sense so that you can examine them in a particular order. All of these are suitable for massive-scale processing when data is large.

Now let's look at EAI - Enterprise Application Integration. This is the stuff of enterprise message buses. it's about processing individual messages so as to perform a transaction not only in one place, but in all the places that the information that transaction contains is needed.

EAI systems are primarily about the processing of one message at a time. not necessarily sequentially, but the logic of an EAI system isn't about processing groups of messages together. The idea of taking the set of messages, sorting it, doing best of breed record construction, is not part of the EAI mind set.  Rather, the transformation and routing of a message to the right places and in the right formats is what it is about.

EAI's message by message approach can be scaled up massively also, but can't do the same kinds of things that a set-oriented system can do across records/messages. Or rather it can't do them the same way. To carry out a transformation involving multiple messages in an EAI system you process one record and keep a state in a data store, typically a database. When the next message of the set arrives, you retrieve the data from the database, compute more, write the state back to the database. Repeat until you've combined all the messages. (Figuring out which is the last one is a big issue.) Ultimately the answer comes from the last message and its interaction with the state, which is where the accumulated knowledge sits about the collective set of messages.

But here's the rub with EAI. When you have messages where there is complex hierarchical data e.g., an XML document, then sometimes the transformations on the detail levels of the hierarchy are set-oriented. E.g., given a purchase order, what is the total of all the parts being ordered. This is not an item-by-item operation on each line-item of the purchase order, it is an operation on the whole set of line-items for a purchase order.

So an EAI system needs to be able to transition from the message by message model of transformation and routing into the set-oriented notion for dealing with the hierarchical content of those messages. This set-oriented stuff is much more like what is typically and readily expressed in ETL/EII systems.

Lately some new acronyms have been added to the soup. (They don't begin with "E" thankfully)One is CEP - complex event processing. This is EAI with different means of expressing the processing using rule languages. Another thing (no acronym) is sensor network streams - this is similar to the real-time ETL systems, but emphasizes that the messages are coming from sensors and the input rate is so fast that it might even be ok to lose a message here or there if needed to keep up overall.

So, my conclusion is that the alphabet soup of EII, ETL, EAI, ELT, isn't really very useful anymore. You have SQL-based approaches, and non-SQL approaches. You need more than the expressive power of just SQL in many cases, and you need more than message-by-message processing as well.

An ideal system has to have the expressive power of the ETL systems, but be designed for always-on online applications, not just batch. It must handle hierarchical data naturally. It must be extensible to allow ad-hoc aggregations, not just the fixed library of SQL or XQuery. It must make simple online message by message processing very simple to express.

But... to my knowledge nobody makes one of those. They're all stuck in one of the Exx boxes and provide only what that one Exx box narrowly provides.

Maybe some open source provider, someone who doesn't care about the Exx categories, will fix this someday.

Internet Data Center vs. Your PC - Which is more wasteful?

My prior post Your Computation Mileage May Vary generated some interesting hallway discussion.

I believe my intended point is valid still which is there is no upper bound in the demand for computational services. My article didn't really make its point too well, which is more about the fact that the computation cost is kind of like rent with utilities included. No incentive to conserve.  

But, I need to clarify that it was not my intent to accuse the Internet search companies of being wasteful of resources.  Not at all.

One issue is the matter of what Internet searching replaces. For example, Ebay vs. trucking newsprint around in order to occasionally reach my eyeballs with a classified ad or two. Hauling newspaper isn't exactly smart use of resources either.

So by way of setting the record straight on Internet search energy use, colleagues pointed me to some of the facts about this subject. Here are some notes on estimates for energy consumption per search query:

• Standard server-class machines are generally not used. Instead, much more energy-efficient designs are used (careful power distribution, power supply, motherboard design, etc.
• There is a lot of room to do cooling more efficiently than is usually done.
• The "thousands" of computers involved in serving search queries are not all involved for each query. (This part I already knew.)
• Searches take far less than 1 second.

Clearly, if it took anywhere near the energy my blog post suggested to run a search engine, companies like Yahoo, Google, etc. wouldn't be able to make (much) money through search advertising.

Google has previously published some information about energy efficiency, here are a couple:

  http://services.google.com/blog_resources/PSU_white_paper.pdf

The above points out that every single PC has an incredibly wasteful power supply in it. Collectively, that's a lot of wasted energy. After reading the above I feel good about buying laptops! 

  http://googleblog.blogspot.com/2008/10/saving-electricity-one-data-center-at.html 

The above says "In the time it takes to do a Google search, your own personal computer will use more energy than Google will use to answer your query."

From my understanding of Internet search technology, and looking at what we can find out publicly about Google, this is entirely credible. Really it is great to know that Google has been successful at lowering the impact of all this great computation capability they're providing.