Rick Barton, managing director of expressor partner Emunio Consulting has just launched a new blog on the BeyeNETWORK.
Should make for interesting reading, as Emunio is the UK’s leading DI/ETL consultancy, and Rick has been working with the leading data integration tools for more than a decade. Click here to read Rick’s first post, where he makes the case for “green” DI.
- Steve Casey, marketing
The independent industry analysis firm Bloor Research has published an “InDetail Report” on expressor software, its place in the industry and the capabilities of the expressor semantic data integration system.
Here’s a small excerpt from the report’s conclusion: “Anyone needing to process high volumes of data and/or needing to support complex processes should immediately put expressor on their short list of data integration products to consider.”
Click here to download the entire document.
- Steve Casey, marketing
News release today: expressor software has signed Sila Solutions Group as a systems integrator partner to target Fortune 500 companies, bringing the total of expressor’s technology, SI and reseller partners to more than 20. Click here to read the full announcement.
- Steve Casey, marketing
Interesting blog from Philip Howard of Bloor comparing expressor’s value proposition vs. Ab Initio to that of Netezza vs. Teradata. We like the comparison…
http://www.it-director.com/technology/data_mgmt/content.php?cid=11366
- Steve Casey, marketing
So there I was sitting with a trainee in a class the other day. He showed me what he had done in a competitor’s product and indicated that there was “no code”. What he had developed was a data flow with 30 or so steps. Each step did something fairly simple such as appending a new field to the output. I asked him to open a step up so we could take a look. Each step had a SQL statement, some fairly simple and some complex.
So let me digress for a second here. Is SQL not code? Sure a simple select statement might not be considered code. Often vendor specific SQL is used creating portability issues. When it can’t be done with “simple” SQL, and this is a real hoot, a stored procedure is created. My perception is that SQL is code.
Developers often know how to do things very efficiently using SQL. Developers know SQL so well that they don’t even consider it code, it’s SQL. And therein lays the problem. What about managers, data analysts, directors, C level executives and auditors? If you are a public company and therefore need to comply with the Sarbanes-Oxley Act, you have created a huge problem. You cannot have accountability without traceability. Traceability can only be accomplished with standards and best practices.
No matter which ETL / data integration tool you use you will be writing code to implement the necessary business and transformation logic. Some of this code will be generated for you automatically; some of the code will have to be written by hand. So the question is not whether SQL is the right scripting language for DI. The question is how DI implementations can be more streamlined.
To assemble a customer’s full name in SQL one might do this:
customer_full_name = first_name || “ “ || last_name
Sorry, not very apparent what is going on to a typical business user or perhaps a C level executive. How about this:
customer_full_name = concatenate(first_name, “ “, last_name)
Most technical staff will argue that SQL, or any other language, is well known and so therefore the implementation turnaround will be much better. So ask the business user why is takes 3 months to add a few columns to a data warehouse table. Maybe it is because the business user has absolutely no insight into the process?
DI implementations can be more streamlined by actually providing real insight into the process. I believe that we at expressor have taken the right approach by providing a scripting language that is flexible, powerful, platform independent and most important, easy to read by all users.
BTW, the application mentioned above only needs one operator, no SQL, and zero lines of “code” with expressor.
John Russell, chief scientist, expressor
What are the differences between your production and development systems? Your production system usually has more computing power. Probably has a naming convention that indicates it is production rather than development. And maybe tighter security. But real differences? None at all.
Think about it for a second. Production and development are implied concepts on every system on the planet. Management by permission, if even present, and naming conventions are all that are available in software lifecycle management, until now.
Imagine a system that on installation asked on what environment it is being installed. And if during installation you choose a production environment, the installation enforces a mutually exclusive nature and allows no other production environment to be executed from the software itself. Likewise if you indicate an install into development, system, integration, or a readiness environment, production applications are not allowed.
Let us take this a bit further and think about an application development system that simply will not allow changes in any environment except development because the system itself is lifecycle aware. That translates into a complete and traceable governance model with total accountability built into the system itself — not an afterthought or checklist that needs to be complete, but part of the development and execution management of the application.
What if this “new” system took it a bit further and accommodated roles? In the old paradigm, project managers or architects have the authority to indicate when a project has completed development and is ready for migration to integration or production stages. Should they have permission to do so? Not likely, because this capability is outside the scope of their job functions. I have always advocated for a different role, a deployer, who may have no responsibility in development whatsoever but is completely responsible for migration to other environments.
Total lifecycle management dictates total governance and accountability in every single environment. With onshore and offshore models becoming more and more prevalent, with government regulations and theft of company data becoming common place, can you really afford to stick with what was once perceived as “good enough?”
By the way, this subject is point #4 in our new “top 10 reasons to choose expressor.” Click the link to read more.
- john russell, chief-scientist and co-founder
expressor’s not a typical software product and I am not your typical technology founder. Sporting a mop of wild hair, having tattoos and riding a Harley are unusual enough, but being from Omaha and attending the University of Nebraska at Omaha like our most famous native Warren Buffet really puts me on the fringe.
In fact, I was reading an article about Buffet’s address to the 2009 Berkshire Hathaway shareholders’ meeting recently, and found it interesting that Buffet was so disdainful of complex financial models, at one point saying, “If you need to use a computer or a calculator to make the calculation, you shouldn’t buy it.” This got me thinking about the over-engineering I’ve seen in too many software products and applications.
Which is why I used a technique that I classify as logic reduction to not simply optimize but to actually remove unneeded logic or processing in expressor. Two useful tools are “Proof by Contrapositive” and “Model Order Reduction.” These are absolutely invaluable tools to eliminate unneeded complexity. Google them and see if you can find ways to utilize these tools.
Does it work? Here are some highlights.
A custom consumer matching process that once took 250 CPU hours with a leading, “high-performance” DI tool is reduced to 1.33 CPU hours using expressor.
A business subsidiary parent/child process that once took 168 CPU hours is reduced to 1 minute.
There is an old saying that “the devil is in the details.” While this is still very true a new one might be “over-engineering in IT is purgatory.”
- john russell, chief-scientist and co-founder
We’ve taken the wraps off the next major update to expressor – which features new reporting and bulk-loading tools, and increased connectivity and platform support.
Click here to read the announcement or check out our expanded product pages for more details on the entire expressor semantic data integration system.
- Steve Casey, marketing manager
Have you ever considered migrating an existing data integration application to another ETL platform – but stopped because you couldn’t justify the required expense and time?
Then join us next Thursday to learn how you can cut your ETL migration cost and time by 80% with an end-to-end software and services solution from expressor software and BitWise.
We’re hosting a webinar at 11:00 am EDT on May 14th that will provide analysis of the technologies and business practices driving down DI costs, an explanation of why and when to migrate to expressor and a demonstration of a tool that converts ETL created on other platforms to run on expressor – and enables you to benefit more quickly from expressor’s high-performance, affordable data integration system.
Click here to read more and to register.
Steve Casey, marketing manager
I have been working in the data integration space for about 15 years now and let me tell you this stuff is expensive. I have always found it odd that the current leading integration software vendors are not bigger financially. Think about it… There are many software vendors that are much younger and have a lower price point and yet yield more profit than the DI vendors. How is that possible?
Well the first reason is that the truth is that most data integration software is not much better than writing custom code. If you are familiar with the space, there is always a stored procedure or some shell script that need to be run due to holes in the product architecture. Let’s face it, these products were first designed, and never rearchitected, when data volumes were much smaller and everything was pretty much relational data. XML and standard message formats like SWIFT and EDI were an afterthought. That deceptive product demo shows a simple GUI, but when the rubber hits the road, the wheels fall off.
The real problem is the cost of the software. Every single add-on cost extra: connectivity, 64-bit, real-time, versioning. Why are these extra when they are fundamental to data integration? Even the speed of the hardware is factored into the cost. Do you have any idea how painful it is to decide to upgrade a server because you know it will also increase your software license costs? And this insult is compounded by the fact that the software already costs more than the hardware in the first place.
The real problem is that software is a commodity market. The old software sales practices from the ‘90’s no longer apply and should not. Back then, many vendors were selling software for outrageous prices because it could be done. Today, that does not work. Look at the technology debacle of 2000. But when you have a large installed base or even worse, are already public, you can’t very well go dropping your prices to accommodate market changes.
Need some validation? Take a look at the SEC filings for your technology vendor. See how much business they are really doing. What you will see if that even though they are adding N new customers per quarter, the installed base is not growing by the same percentage. The reason is that old customers are leaving just about as fast as new ones are joining.
I wonder why that is happening?
- john russell, chief-scientist and co-founder
