User Defined Data Loader



Pete Chanis



Allow data warehouse system users to load data that can be linked to existing tables.



My project will allow a user to load data local to his computer into a data warehouse table through an http based interface. The user will then be able to query his data in conjunction with existing data warehouse tables. The user will be able to uniquely identify the data set. The user will receive confirmation that the data was loaded correctly. The user will also be able to manage the loaded date, (e.g., edit, delete update existing data sets).  I am anticipating the application being servlet/jsp based.



Many of our users generate custom lists of customers for various marketing related purposes. With our web based reporting tools there is currently no mechanism for users to generate results filtered by such a custom list. Users must query all customers and then filter the results locally, typically in a spreadsheet. This is inefficient, frustrating and tome-consuming.



The project will require a servlet engine and web server. It seems likely that I will need to use some xml based technology to generate the data for the web service which will actually load the data. Perhaps this could be MS Excel based and use VBA XML libraries to generate the data for the data load web service. I will use xslt to format html status messages.



My biggest question is does this make sense from a technology standpoint? I like the idea of the flexibility and potential re-usability of a web service to load data for this purpose but I may be off track here. Also, assuming this is a good technical approach, where should the processing be done. Can I and should I leverage Excel to process the data  and communicate with the web service from the client pc? In which case Excel essentially becomes the front end interface, at lease for loading data. Or should the data files be processed on a server?


I am certainly open to help, thoughts and feedback. I also have other project ideas if you do not feel that this is a good project.