2.2. Engine Based ETL Data Integration Solution
The specialist ETL vendors provide tools employing proprietary scripting languages running within a generally rigid centralised ETL server. These ETL engines use language interpreters to process ETL workflows at runtime. The ETL workflows defined by developers in the graphical environment are stored in a meta-data repository, which the engine reads at run-time to determine how to process incoming data. The intent was to abstract away some of the implementation issues, make data mapping more graphically orientated and introduce more automated ETL processes.
Although this interpretive approach more tightly unified design and execution environments, it rarely eliminated the need to custom code. Every organisation has its own unique and/or complex requirements, and invariably developers or high cost vendor consultants then have to resort to coding custom routines and exits that the tool then accessed at run-time . The result is often the complexity and cost of maintaining and updating that is found in the purely hand-coded option.
Another significant characteristic of an engine based approach is that all processing takes place in the engine, not on source systems . The engine typically runs on a Windows® or UNIX® machine and establishes direct connections to source systems. This has increasingly been a source of frustration, as projects need a great deal of flexibility in their architecture, and freedom to deploy transformations and other data integration components without having to revolve around an often heavyweight and costly centralized server.
