Data Pipeline 2, Data Transformation Toolkit for Java Released

Data Pipeline makes it easy to convert, process, and transform data in Java applications.

The new release includes...
* Open source (GPL) and commercial licenses
* New data readers, writers, and transformers
* Java 5 support

Features
* Readers: CSV, fixed-width, Excel, database, weblogs, custom
* Writers: CSV, fixed-width, Excel, database, PDF, Word, XML, custom
* Operations: validate, filter, sort, lookup, remove duplicates, convert, throttle, calculate, custom, and more
* Run-time expression language for filters, validation, and calculated fields

A typical scenario might be to:
1. read a CSV file
2. remove duplicate records
3. add calculated field
4. remove unused columns
5. save to database

Code:

---

    public static void main(String[] args) throws Throwable {
        DataReader reader = new CSVReader(new File("credit-balance.csv"))
            .setFieldNamesInFirstRow(true);
       
        // Use only the "Rating" and "CreditLimit" fields in duplicate test
        reader = new RemoveDuplicatesReader(reader, new FieldList("Rating", "CreditLimit"));       
       
        // Add AvailableCredit field, remove "CreditLimit", "Balance" fields
        reader = new TransformingReader(reader)
            .add(new SetCalculatedField("AvailableCredit", "parseDouble(CreditLimit) - parseDouble(Balance)"))
            .add(new ExcludeFields("CreditLimit", "Balance"));
       
        DataWriter writer = new  JdbcWriter(getJdbcConnection(), "dp_credit_balance")
            .setAutoCloseConnection(true);
       
        JobTemplate.DEFAULT.transfer(reader, writer);
    }

---

We look forward to hearing your feedback.

Downloads:
http://northconcepts.com/downloads/

Examples:
http://northconcepts.com/data-pipeline/examples/

Getting Started:
http://northconcepts.com/data-pipeline/getting-started/

Forums:
http://northconcepts.com/forums/


Dele Taylor
North Concepts Inc.