SSIS Delimited File Source

An extremely robust flat file source for Integration Services which handles quoted text and provides full data auditing. Based on the work of Microsoft's Bob Bojanic (Delimited File Reader Source Sample).

Features

  • Easy to use interface based on the exisiting File Source
  • Handles quoted text and extra/missing columns
  • Support for auditing all read and format actions/errors
  • International data (all encodings) ready
  • Clean Model-View-Presenter design
  • Automated package creation based integration tests
  • Suite of unit tests

Notes I'll upload the SSIS 2005 branch shortly.

Documentation

This initial release is a bit rough around the edges, but over the next few weeks I will create a blog series detailing the design decisions and concepts that go into creating an SSIS custom component. Hopefully I'll also have a chance to clean up the source code as well!

For a more detailed dive into the DFS please see http://www.quickandagile.co.uk/post/Introduction-to-SSIS-Delimited-File-Source.aspx

History

The Delimited File Source (DFS) was originally created to handle quoted strings within CSV files as part of The Database Group ETL process. The first version was based on the CSV Reader by Sebastien Lorion. Whilst this provided outstanding performance it lacked in robustness and extensibility. The overall design of the project was very central to the reader which resulted in a hard to maintain and test solution.

For these reasons the release version (the one released here) was completely rearchitected and based off the sample created by Bob Bojanic. This introduced a better clean design which allowed unit and integration testing to be performed with ease.

Links
The Database Group http://www.databasegroup.co.uk
CSV Reader http://www.codeproject.com/KB/database/CsvReader.aspx
SSIS Community Samples http://www.codeplex.com/SQLSrvIntegrationSrv

Roadmap

Although adding more integration test features is high on my agender (and presenter unit tests), the key feature missing is the performance that CSV Reader provided. I hope with the next major release to take some of the great work Sebastien has done on his CSV Reader and incorporate it to once again allow the DFS to be blisteringly fast.

Last edited Sep 15, 2009 at 2:58 PM by roblad, version 20