2. What Solutions are Available?

2.1. Hand Coding:

Prior to the development of specialist ETL tools , solutions to transformation problems were primarily developed as custom solutions by in-house or consulting programming teams.

Organisations have increasingly found (as in other areas) that building by hand can initially offer a rapid low cost solution, but so often turns into a costly, skill intensive and time-consuming task. A consequence of adopting this approach is that important details and business critical knowledge are very often hidden in lowlevel code and this creates difficulty, especially down the line, in maintaining and updating the code. Indeed, this problem is compounded by the continual need to update the solution each time models or requirements change, often without the original team who developed the code.

Since the mid 90s a number of third party products appeared on the market, many from very large established vendors , some specialising in data integration or ETL. The specialist ETL vendors were initially dominated by what the Data Warehousing Institute1 described as engine based tools, although a number of new products on the market generated code which could easily be incorporated into client architectures. More recently, database vendors have added ETL features and these are beginning to provide a reasonable level of ETL capability.

Surprisingly, even though ETL tools have been developed for over 10 years and are now mature products, hand coding still persists as a significant contribution for solving transformation problems.

ENGINE BASED ETL DATA INTEGRATION SOLUTION