Our data transformation expertise

Today's post is all about data transformation, and how Qbase does it differently. Sure, there are a whole lot of commercial transformation engines out there, but what makes Qbase better? Well, it really comes down to the technological advancements we have developed for our transformation engine: Qbase Data Transformer™. These advancements are in the following areas: in our "non-blocking" data flow architecture; in our use of late binding based on the requirements of the data; in our leveraging of deliberate error processing (which allows processing to continue when certain known errors are encountered); and, finally, in our use of distributed HTC, where jobs run across multiple servers. Interested in the details on these differentiators? Read on...

• Data flow architecture — Qbase Data Transformer™ has been designed to run "non-blocking" (at least as much as the data flows allow). This allows large data sets to be writing finished results to output files even while original data sets are still being read.
• Late binding — Rather than making rigid requirements regarding which fields are required and in what order, we've designed QDT™ for maximum flexibility with respect to data source changes. We did this by basing data field requirements on a runtime binding against the requirements of the data flow. So, as long as the fields required by the transformation are present in some order, the job will run successfully. Any additional data not required by the flow is passed on unchanged.
• Deliberate error processing — By "deliberate error processing," we mean that each component in a QDT™ job is "aware" of explicit error types that allow bad data to continue to flow through the system, retaining its original form and error status. Many systems kick out records and fields as errors occur, but QDT™ can let these continue to flow through the system allowing their corresponding records to participate in subsequent processing (where it still makes sense, that is).
• Distributed High Throughout architecture — QDT™ jobs are able to run with individual components distributed across multiple servers in an HTC setup. This provides greater parallel processing and allows the job to move processing to systems which may have specific resources unavailable on other servers. When combined with the data flow architecture, this results in quite a boon to overall throughput.

Find out how we can put these superior technologies to work for you today.