First, setup four package level variables:
FilePath – string
SSISFolderFilePath – string
SSISFolderName – string
SSISFolders – object

FilePath should be set to the location where you want to store the packages. Include the trailing slash, and ensure that the location exists.
SSISFolderFilePath should be set to evaluate an expression (EvaluateAsExpression = TRUE), and its expression should be:
@[User::FilePath] + @[User::SSISFolderName]

Then create a connection manager object that points to your MSDB database where you want to extract from. If you want this to be dynamic, perhaps to archive ALL SSIS packages in your company, then this could be made a dynamic connection manager object (by using expressions) and you iterate over each server. For now, we’re only going to use one server.

Once these are setup, the rest is point and click, pretty much. Add an Execute SQL Task to the control flow, and configure it so that it returns a full resultset. See below.

The query for the Execute SQL Task is:

SELECT foldername
  FROM dbo.sysssispackagefolders folders
 WHERE EXISTS (SELECT NULL
                 FROM dbo.sysssispackages pkg
                WHERE pkg.folderid = folders.folderid)

*Note – the above SQL only assumes a flat directory structure. If you have subdirectories in MSDB, you can use a recursive CTE to create a hierarchy resultset.

With the Execute SQL Task set up, you’ll have to shred the results using a Foreach Loop Container. Inside the Foreach Loop Container, we will want to use a File System Task to create directories for our packages if they do not already exist. The configuration for that is below:

Next, we have to create a data flow and attach it to the Foreach Loop Container. The full control flow should resemble the next screenshot.

In the data flow, we have three components – OLE DB Source, Derived Column, and the Export Column transformation. Connect the OLE DB source to your connection manager object, and use the following query against MSDB:

SELECT folder.foldername,
       pkg.name,
       pkg.packagedata
  FROM dbo.sysssispackages pkg
  JOIN dbo.sysssispackagefolders folder
    ON pkg.folderid = folder.folderid

Then, configure the Derived Column component so that it creates one new column, FullFilePath. The following is the expression to use:

(DT_WSTR,500)(@[User::FilePath] + foldername + "\\" + name + ".dtsx")

Then configure the Export Column transformation as the following screenshot illustrates:

The final data flow should look like the following:

The only thing left to do is execute the package. Navigate to the location defined by the FilePath package variable and you should see a directory structure resembling that in MSDB with all of the SSIS packages saved.

If you wish to download the full SSIS package, here you go: http://www.ssistalk.com/ExportSSISPackages.dtsx

These are common questions I see when dealing with “slow” performance of an SSIS package. Sometimes though, the developer of the package does not even understand which component is taking so long to process say, thousands of records in their data flow, so it is understandable why this component may proliferate in many shops. (Sometimes it is needed, of course, but you need to always ask the question – do I need the updates to happen RIGHT NOW?)

If you have a need to update data in another table, while still pushing data from a source to a destination, the OLE DB Command is often used because of the thought that it is part of the data flow, so why not use it.

Read on for more….

The trouble with the OLE DB Command component is that it executes whatever command you have provided for each and every row that passes through it. The data flow is designed to operate on buffers of data, which contain by default around 10,000 rows. If you are simply moving data from one table to another, this concept works really well and quite fast too, I might add.

However, if you need to update data, should you use the OLE DB Command component in that data flow? Are there other options? The answers of course are “maybe” and “yes.”

If you simply need to update data in a table and you don’t care when it happens, then I would recommend not using an OLE DB Command component at all in your data flow. Instead, I recommend you build a staging table on your destination database server, and populate that table with your “update” data. Then, in the control flow, after the data flow successfully executes, you can issue an Execute SQL Task to perform a batch update.

This recommendation is built on the fact that using an OLE DB Command component ensures that your update operations are done row-by-row, rather than in a batch operation.

Why? How much faster is the batch operation compared to the OLE DB Command operation? Well, the answer depends a bit, but in my simple test below, the time to update 5,000 records took 15 minutes using the OLE DB Command component and 4 seconds (yup – SECONDS) to populate a staging table and then perform a batch update.

Here is the setup I used:

CREATE TABLE numbers (
  number int not null
)
GO

insert into numbers (number)
select number from
(
select row_number() over (order by s1.name) as number
  from sysobjects s1
 cross join sysobjects s2
 cross join sysobjects s3
 cross join sysobjects s4
 cross join sysobjects s5
) as numbers
where number between 1 and 1000000;
GO

select number
  from numbers
 where number < 5003;
GO

/*
number
1
2
3
4
5
...
4998
4999
5000
5001
5002
*/

Here is a picture of the first SSIS package.

I used the following SQL for the OLE DB Source component:

SELECT number
  FROM numbers
 WHERE number < 5001
 ORDER BY number DESC

I used the following expression in the Derived Column component to create a new column, new_number:

number + 1

I used the following SQL in the OLE DB Command component:

update numbers
  set number = ?
 where number = ?

In the OLE DB Command "Column Mappings" tab, I mapped the incoming column, new_number, to the Param_0 parameter, and the incoming column, number, to the parameter, Param_1.

I then executed the package. Time to execute: 15 minutes, 43 seconds.

Next I tried the batch update approach. Use the same setup as above, except replace the OLE DB Command component with an OLE DB Destination, and push the two data flow columns to a staging table.

Here is the data flow layout:

Here is the staging table DDL:

CREATE TABLE staging_numbers (
  orig_number int not null,
  new_number int not null
)

Map the data flow column, number, to the orig_number field in the staging table. Map the data flow column, new_number, to the new_number field in the staging table, as shown below:

Then, in the control flow, hook up an Execute SQL Task to the new Data Flow task, so that the Execute SQL Task executes after the Data Flow, as shown below:

The Execute SQL Task SQL is below:

update numbers
  set number = staging_numbers.new_number
from numbers, staging_numbers
where numbers.number = staging_numbers.orig_number

Upon executing the package, the execution time shrank to under five seconds:

So, the moral of this story is that whenever you think you need to use an OLE DB Command component, ask yourself, "Do I *really* need to use this component, or should I use a different approach?"

If you decide to use an ORDER BY in your source and want to tell the SSIS Data Flow that your data is sorted, follow the below steps:

First, ensure that your data is being sorted with the ORDER BY clause. Make note of which column(s) are in the sort and their positions.

Second, open the Advanced Editor for the OLE DB Source component by right-clicking on the OLE DB Source and selecting “Show Advanced Editor…”

Then, click on the “Input and Output Properties” tab.

Then highlight the “OLE DB Source Output Properties” branch and set the “IsSorted” property to True.

Next, expand the “OLE DB Source Output Properties” branch. From here, select the first column under “Output Columns” that corresponds to the first column in your ORDER BY clause. Set its “SortKeyPosition” property to the number 1.

Repeat the last step for each column in your ORDER BY clause, advancing the SortKeyPosition by one each time. (1,2,3,…)

That’s it. Now the SSIS Data Flow will understand and acknowledge that the incoming data is sorted and will not require a Sort Component between the source and a Merge Join component, for instance.

For this post, use the following schema and data:

create table fact_sales
(id int identity(1,1),
 sales_rep_id int,
 sales_dollars decimal(18,2)
)

create table dim_sales_rep
( id int identity(1,1),
  first_name varchar(30),
  last_name varchar(50)
  )

insert into fact_sales (sales_rep_id, sales_dollars) values (1,120.99);
insert into fact_sales (sales_rep_id, sales_dollars) values (2,24.87);
insert into fact_sales (sales_rep_id, sales_dollars) values (3,98.11);
insert into fact_sales (sales_rep_id, sales_dollars) values (4,70.64);
insert into fact_sales (sales_rep_id, sales_dollars) values (4,114.19);
insert into fact_sales (sales_rep_id, sales_dollars) values (4,37.00);
insert into fact_sales (sales_rep_id, sales_dollars) values (5,401.50);

insert into dim_sales_rep (first_name, last_name) values ('John','Doe');
insert into dim_sales_rep (first_name, last_name) values ('Jane','Doe');
insert into dim_sales_rep (first_name, last_name) values ('Larry','White');
insert into dim_sales_rep (first_name, last_name) values ('Carrie','Green');
insert into dim_sales_rep (first_name, last_name) values ('Adam','Smith');

FULL Cache Mode
First, it is always advisable to build a query for the lookup, instead of choosing a table in the Table/View drop-down. The primary reason is so that you can limit the resultset to only the columns needed to perform the lookup as well as return any columns needed downstream, and to have the ability to add a WHERE clause if needed.

The full cache mode will run the specified query (or its own depending on how you assigned the lookup table) and attempt to cache all of the results. It will execute this query very early on in the package execution to ensure that the first set of rows coming out of the source(s) are cached. If SSIS runs out of memory on the machine though, the data flow will fail as the lookup component will not spool its memory overflow to disk. Be cautious of this fact. Once the data is cached, the lookup component will not go back to the database to retrieve its records, so long as the data flow is not restarted. (In SQL Server 2008, you can now reuse lookup caches.)

Using SQL Profiler, you can see that only one database call is made:

declare @p1 int
set @p1=1
exec sp_prepare @p1 output,NULL,N'select sales_rep_id, sales_dollars
 from fact_sales',1
select @p1
go
exec sp_execute 1
go
SET NO_BROWSETABLE ON
go
declare @p1 int
set @p1=1
exec sp_prepare @p1 output,NULL,N'select id, first_name, last_name
from dim_sales_rep',1
select @p1
go
exec sp_execute 1
go
exec sp_unprepare 1
go
exec sp_unprepare 1
go

PARTIAL Cache Mode
Partial cache mode will not execute a query immediately at package execution. Instead, it will wait until its first input row arrives. Once the row arrives, whatever lookup value (in this case, sales_rep_id) is being passed in, will get substituted for a parameter, and then SSIS will send the query to the database for retrieval. At this point, all of the data returned will be cached for future lookups. If a new sales_rep_id is encountered, then the query will have to be re-executed, and the new resultset will get added to the lookup cache.

In other words, in the above data, if my source is “select sales_rep_id, sales_dollars from fact_sales”, we should have five database calls made by the lookup component. Even though for sales_rep_id = 4 we have three entries, in partial cache mode the first time we retrieve the lookup records for sales_rep_id = 4, the results will be cached, allowing future occurrences of sales_rep_id = 4 to be retrieved from cache.

This is illustrated in the SQL Profiler data:

exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',1
go
exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',2
go
exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',3
go
exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',4
go
exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',5
go
exec sp_unprepare 1
go

In the above data, you can see at the end each sales_rep_id being passed in. Note that we only have one line for sales_rep_id = 4. That’s because the remaining two records were bounced against the lookup cache, avoiding a trip to the database.

NO Cache Mode
Using the NO Cache Mode will essentially tell SSIS that you want each incoming row (from fact_sales in this case) to be bounced against the database. Since we have seven fact_sales rows, we will see seven calls to the database – MOST of the time. It is important to note that even though we are telling the lookup component to avoid caching rows, it will keep the last match in memory and use it for the next comparison. If the next comparison’s key value matches the value still in memory, a database call is avoided, and the value is carried forward.

In our example data above, if we sort by sales_rep_id, we will still only have five calls to the database because after we lookup our first value of sales_rep_id = 4, it will be reused for the subsequent lookups for sales_rep_id = 4. If we sort our data by sales_dollars, we will have six database calls, because only two sales_rep_id = 4 records are together and hence the first lookup is only used once.

Here is a simple table illustrating each no cache example mentioned above:

SALES_REP_ID, SALES_DOLLARS, LOOKUP DATABASE CALL Y or N
1       120.99    Y
2       24.87     Y
3       98.11     Y
4       70.64     Y
4       114.19    N
4       37.00     N
5       401.50    Y

SALES_REP_ID, SALES_DOLLARS, LOOKUP DATABASE CALL Y or N
2      24.87      Y
4      37.00      Y
4      70.64      N
3      98.11      Y
4      114.19     Y
1      120.99     Y
5      401.50     Y

The SQL Profiler data for the second example above is here:

exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',2
go
exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',4
go
exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',3
go
exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',4
go
exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',1
go
exec sp_executesql N'select * from (select id, first_name, last_name
from dim_sales_rep) [refTable]
where [refTable].[id] = @P1',N'@P1 int',5
go

http://blogs.msdn.com/sqlperf/archive/2009/03/03/an-etl-world-record-revealed-finally.aspx

If you’d at least like to see CaSE INsensitive searches, please visit my submission and lets see if we can get this feature added to SSIS in SQL Server 2008.

One workaround for you, should you want to use it, is to use the Fuzzy Lookup Component instead. This has its own problems, as I’ve made mention to in the above feature request submission.

Edit: Jamie Thomson submitted an identical request before me but I’ve made reference to that submission in mine. The idea was to get it in SSIS 2008. Never-the-less, Michael Entin claims this feature request has been dropped due to lack of time.

• Ensure that there is a final select statement at the end of the stored procedure to ensure the correct resultset is returned
• At the top of the stored procedure as part of the procedure code, add the following line: SET NOCOUNT ON
• At the top of the OLE DB Source SQL statement and before you issue the EXEC YourSproc line, add the following: SET FMTONLY OFF.

This should ensure that when you click on the column mapping tab, the columns are correctly displayed.

Also another approach is to use a table variable in the stored proc. Then at the end, select from that table variable.

See them here: http://blogs.conchango.com/jamiethomson/archive/2007/08/21/SSIS_3A00_-Big-improvements-to-Lookup-in-SQL-Server-2008.aspx

First things first. Add a script component to the data flow. When prompted, select “Source” as the component type. (click on images for full sized version)

Next, for clarity’s sake, I’ll rename the default output. (An output simply contains a set of columns and will correlate to the green arrow on the data flow. The more outputs you have, the more green arrows you can use.) I’ll rename the default output, “Output 0″ to “MyOutput.”

Next, I’ll add an output column called, “MyColumn” and its type will be an integer. Be sure to change the type accordingly.

From here, we can edit the script by clicking on the script section on the left of the above window. From there click on the “Design Script…” button. Below is the setup:

Notice how when I type, I am automatically prompted for a list of available items to choose from. One of them is the column I added previously. Let’s pick the column and assign a value of 123 to it. Full script below:

For SSIS 2005:

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper

Public Class ScriptMain
    Inherits UserComponent

    Public Overrides Sub CreateNewOutputRows()
        MyOutputBuffer.AddRow()
        MyOutputBuffer.MyColumn = 123
    End Sub

End Class

For SSIS 2008, it’s a bit different:
Visual Basic:

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper

<Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute> _
<CLSCompliant(False)> _
Public Class ScriptMain
    Inherits UserComponent

    Public Overrides Sub CreateNewOutputRows()
        Output0Buffer.AddRow()
        Output0Buffer.Column = 123
    End Sub

End Class

C#:

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{

  public override void CreateNewOutputRows()
    {
        Output0Buffer.AddRow();
        Output0Buffer.Column = 123;
    }

}

You can close the script and then click OK on the script component dialog box. Now when you hook this source up to another component, you'll end up with one column with one row with a value of 123.

Let’s say we have the following data file with two rows in it. This data file contains three columns:

06 000960{ 00000{
07 000471C 00074}

What do we have here? The first column represents a key, perhaps. The second and third columns could be dollar amounts, perhaps. In normal terms the fields above are really representing:

06 +9600 +00
07 +4713 -740

The problem with zoned decimals, and it’s not really a big deal, is that the last byte is overlaid with a sign representation. The last byte indicates the sign AND the numerical value. One other thing to note is that the data does not contain decimal point placement information. That is handled outside of the storage and is really a presentation layer feature.

So, how do we work with this in SSIS? Actually, it’s pretty easy with some simple tricks. (I’m going to move away from the file processing and just pretend that I only have one field containing a zoned decimal called “CurrencyValue.”)

Basically we only need three transformations in the data flow to convert these values: two derived columns and a lookup transformation.

In the first derived column transformation, we do two things. The first is to grab the right-most character of our field. This character contains the sign for the entire number and the numerical value of the last digit. We also check to ensure that we don’t have nulls or blanks coming in, if we do, assign a default value of “0{” just to passify the later transformations.

As you can see above (click on the image to expand it), we added a column “SignChar” and replaced the CurrencyValue column with a default value if nothing existed there. Pretty simple and straightforward stuff.

Next, we add a simple lookup table. This “table” is really just an SQL statement to perform our conversion. We’ll output the sign and the numerical value from this lookup. Use the SQL below to build the lookup table:

select 1 as sign, '{' as lookupvalue, 0 as returnvalue
union all
select 1 as sign, 'A' as lookupvalue, 1 as returnvalue
union all
select 1 as sign, 'B' as lookupvalue, 2 as returnvalue
union all
select 1 as sign, 'C' as lookupvalue, 3 as returnvalue
union all
select 1 as sign, 'D' as lookupvalue, 4 as returnvalue
union all
select 1 as sign, 'E' as lookupvalue, 5 as returnvalue
union all
select 1 as sign, 'F' as lookupvalue, 6 as returnvalue
union all
select 1 as sign, 'G' as lookupvalue, 7 as returnvalue
union all
select 1 as sign, 'H' as lookupvalue, 8 as returnvalue
union all
select 1 as sign, 'I' as lookupvalue, 9 as returnvalue
union all
select -1 as sign, '}' as lookupvalue, 0 as returnvalue
union all
select -1 as sign, 'J' as lookupvalue, 1 as returnvalue
union all
select -1 as sign, 'K' as lookupvalue, 2 as returnvalue
union all
select -1 as sign, 'L' as lookupvalue, 3 as returnvalue
union all
select -1 as sign, 'M' as lookupvalue, 4 as returnvalue
union all
select -1 as sign, 'N' as lookupvalue, 5 as returnvalue
union all
select -1 as sign, 'O' as lookupvalue, 6 as returnvalue
union all
select -1 as sign, 'P' as lookupvalue, 7 as returnvalue
union all
select -1 as sign, 'Q' as lookupvalue, 8 as returnvalue
union all
select -1 as sign, 'R' as lookupvalue, 9 as returnvalue

Next, set up the mappings:

We map our input, SignChar to “lookupvalue” and return “returnvalue” and “sign.”

The last step is to calculate the actual value of the zoned decimal. We use the second derived column to do this and basically we replace the last character of the zoned decimal with “returnvalue” and then multiply that value by “sign.”

(In my work I chose to have the output of this be character data, but you could cast it to a decimal, integer, whatever if you desire.)

Here’s the expression:

(DT_STR,17,1252)(sign * (DT_I8)REPLACE(CurrencyValue, RIGHT(CurrencyValue,1), (DT_WSTR,1)returnvalue))

And that’s pretty much it. NewCurrencyValue will contain the converted number from Zoned Decimal to a “normal” representation to be able to be used in calculations and such.

NOTE: There is one gotcha with this. The example I provided above works well for ONE column. If you have many, you’d better look into unpivoting the records first before going into the first derived column transformation, and then pivot them back when done with the last derived column. That way, you only have to build this once, versus one for each column.