SQL Server – Business Intelligence Development (BIDS) – Data Flow Task – Lookup Transformation – Introduction

What is Lookup Transformation?

Lookup Transformation is exactly what it sounds like.

Using columns that you designate from your Source Data Source, it draws a line to the Secondary Data Source and project columns.

Here is Microsoft’s own definition:

Link
“The Lookup transformation performs lookups by joining data in input columns with columns in a reference dataset. You use the lookup to access additional information in a related table that is based on values in common columns”.

Back End Database Platforms Supported

There are some restrictions, such as:

  1. OLE DB Provider
    • The Only 3 providers that are supported are
      • SQL Server
      • Oracle
      • DB2

 

Join Type

  • It is an Equijoin, an inner join so to speak
    • Only Source Records that have corresponding entries in the Secondary Table are returned
    • On each Source Record that have more than one Secondary Record, only one matching record is returned
  • Both singleton ( Single Column ) and Composite ( Multiple Columns ) joins  are supported
  • Comparison is usually case-sensitive, except if a Cache Manager is used to cache the dataset
    • To guide against mismatches due to case and space differences, please use
      • Conversion Functions
        • upper, lower
      • Trimming Functions
        • ltrim, rtrim, trim
    • To facilitate the adjustments to data mentioned above, please use
      • Character Map Transform
      • Derived Column Transform
  • When matching records are not found in the Lookup Table, an error is triggered.
    • This error is considered to be  a Row Level Error
    • Error can be handled using a few choices
      • Failure ( Fail Component )
      • Ignored ( Ignore failure )
      • Captured ( Redirect Row )
  • When records are found, we choose the columns to merge into the Source Table
    • This error is Column Level Error
    • Errors are handleable based on the same choices available to Row Level Error Handling

 

Business Case

OLTP tables are usually built to be very slim.

OLAP tables on the other hand are built to be fat.

 

Data Model

Drawn below are three tables.

DatabaseDiagram

 

The OLTP table is LTEDRKV.callLog, the lookup table is dbo.internationalCallingCode, and the Data Warehouse Table is LTEDRKV.callLogDW.

Our need is very preliminary and it is also track the country being called in the LTEDRKV.callLogDW.

 

Data

Data Definition Language

Function

Function – [dbo].[ufn_getPhoneNumberCountryCode]

 
if object_id('[dbo].[ufn_getPhoneNumberCountryCode]') is null
begin
 
    exec('create function [dbo].[ufn_getPhoneNumberCountryCode]() 
          returns varchar(60)
          as
          begin
            return 1
          end
          ')
 
end
go
 
alter function [dbo].[ufn_getPhoneNumberCountryCode]
(
    @phoneNumber varchar(100)
) 
returns varchar(60)
with schemabinding
as
begin
 
    return
    (
 
        case
                when (@phoneNumber is null) then null
                when charindex('-', @phoneNumber) = 0 then null
                else left(
                              @phoneNumber
                            , charindex('-', @phoneNumber) -1
                         )
        end
 
    )
 
end    
 
go
 
grant execute [dbo].[ufn_getPhoneNumberCountryCode] on [public]
go
 

 

Table

Table – [dbo].[internationalCallingCode]

 

 
set noexec off
go
 
if object_id('[dbo].[internationalCallingCode]') is not null
begin
    set noexec on
end
go
 
create table [dbo].[internationalCallingCode]
(
 
      [country]                varchar(60) not null
    , [countryCallingCode]  varchar(6)  not null
 
    , [dateAdded]   datetime not null
                        default getdate()
 
    , constraint [PK_InternationalCallingCode]
            primary key
            (
                  [country]    
                , [countryCallingCode]
            )
 
)
go
 
set noexec off
go
 

 

 

 

Table – [LTEDRKV].[callLog]

 

 

 
set noexec off
go
 
if schema_id('LTEDRKV') is null
begin 
 
    exec('create schema [LTEDRKV] authorization [dbo]') 
 
end 
go
 
if object_id('[LTEDRKV].[callLog]') is not null
begin
    set noexec on
end
go
 
create table [LTEDRKV].[callLog]
(
 
      [callID] bigint not null
            identity(1,1)
 
    , [phoneNumber] varchar(60) not null
 
    , [dateAdded]   datetime not null
                        default getdate()
 
    , constraint [PK_CallLog]
            primary key
            (
                  [callID]    
            )
 
)
go
 
set noexec off
go
 
if not exists
    (
        select 1
        from   sys.columns tblSC
        where  tblSC.[object_id] = object_id('[LTEDRKV].[callLog]')
        and    tblSC.[name] = 'phoneNumberCountryCode'
    )
begin
 
    alter table [LTEDRKV].[callLog]
        add [phoneNumberCountryCode] 
            as [dbo].[ufn_getPhoneNumberCountryCode]
            (
                [phoneNumber]
            )
            persisted
end
go
 

 

Table – [LTEDRKV].[callLogDW]

 

 

 
use [DBLab]
go
 
/*
 
    drop table [LTEDRKV].[callLogDW]
 
*/
 
set noexec off
go
 
if schema_id('LTEDRKV') is null
begin 
 
    exec('create schema [LTEDRKV] authorization [dbo]') 
 
end 
go
 
if object_id('[LTEDRKV].[callLogDW]') is not null
begin
    set noexec on
end
go
 

 
create table [LTEDRKV].[callLogDW]
(
      [callID]                   bigint not null
            
    , [phoneNumber]               varchar(60) not null
 
    , [phoneNumberCountryCode] varchar(6) not null
    , [country]                   varchar(60) not null
 
    , [dateAdded]               datetime not null
 
    , constraint [PK_CallLogDW]
            primary key
            (
                  [callID]    
            )
 
                        
)
go
 
set noexec off
go

 

 

Business Intelligence Development (BIDS)

Designer

Lookup

OLE DB Source Editor

The OLE DB Source Editor has 3 tabs. And, those are the Connection Manager, Columns, and Error Output.

Here is a brief outline of the purpose of each tab.

  1. Connection Manager
    • Reference or shall we say a pointer to one of the Connection Managers defined for the Project
    • Data Access mode
      • Our choices are
        • Table or view
          • Name of the table or view
        • SQL
      • In our case it is LTEDRKV.callLog
  2. Columns
    • Available External Columns
      • All the Columns defined in the External Columns are listed
      • Optimize performance by unchecking uneeded columns
      • In our case we choose all the columns ( callID,  phoneNumber, dateAdded, phoneNumberCountryCode )
  3. Error Output
    • Excception Handling Block
      • Data read from Source / External into Internal Storage
        • If truncation error occurs, how should it be handled
          • Fail?
OLE DB Source Editor – Connection Manager

OLEDBSourceEditor-ConnectionManager

OLE DB Source Editor – Columns

OLEDBSourceEditor-Columns

 

OLE DB Source Editor – Error Output

OLEDBSourceEditor-ErrorOuput

 

 

Lookup Transformation Editor

In the Lookup Transformation Editor we perform an inner join against our Lookup Table ( dbo.InternationalCallingCode )

This Editor has 3 Tabs, as well.

The Tabs are named – Reference Table,  Columns, and Advanced.

Here is a brief outline of the purpose of each tab.

  1. Reference Table
    • Details on the Referenced Table
      • The Connection Manager that houses it
      • And, the Table\View or SQL Statement
      • In our case it is dbo.InternationalCallingCode
  2. Columns
    • Define the Joining Columns as well as the Columns to Project from the Referenced Table
      • Review Data Model and see if Foreign Key Relationships are defined
      • Again, optimize by unchecking columns from the Referenced Table
      • In our case we are joining on the columns below:
        • LTEDRKV.callLog.phoneNumberCountryCode <> dbo.InternationalCallingCode countryCallingCode
    • Projected Columns
      • dbo.InternationalCallingCode.country
  3. Advanced
    • Enable Memory Restriction

 

LookupTransformationEditor-ReferenceTable

 

Columns

LookupColumns

 

Advanced

LookupTransformationEditor-Advanced

 

 

Lookup Transformation Editor – Is Data Captured in Data WareHouse Table

In the Lookup Transformation Editor we perform an inner join against our Destination Table ( LTEDRKV.callLogDW )

In cases where corresponding records are not found in the Lookup Table, we redirect them.

This Editor has 3 Tabs, as well.

The Tabs are named – Reference Table,  Columns, and Advanced.

Here is a brief outline of the purpose of each tab.

  1. Reference Table
    • Details on the Referenced Table
      • The Connection Manager that houses it
      • And, the Table\View or SQL Statement
      • In our case it is LTEDRKV.callLogDW
  2. Columns
    • Define the Joining Columns as well as the Columns to Project from the Referenced Table
      • Review Data Model and see if Foreign Key Relationships are defined
      • Again, optimize by unchecking columns from the Referenced Table
      • In our case it is LTEDRKV.CallLog.CallID <->LTEDRKV.callLogDW.CallID
    • Projected Columns
      • Nothing Defined
  3. Advanced
    • Enable Memory Restriction
  4. Configure Error Output
    • Input Output :- Lookup Output
    • Column :- N/A
    • Error :-
      • Ignore Failure
      • Redirect row
      • Failure Component
      • In our case, we chose “Redirect Row
    • Truncation :- N/A

 

Reference Table

ReferenceTable

 

 

Columns

ReferenceTable-Columns

 

Advanced

ReferenceTable-Advanced

 

Configure Error Output

ConfigureErrorOutput-20161029-0805AM

 

 

Lookup Error Output

In the preceding we join LTEDRKV.callLog against LTEDRKV.callLogDW.

When matching data is not found, they are placed in an ERROR bucket.

 

 

 

Data Flow Path Editor

This Editor has 3 Tabs, as well.

The Tabs are named – Reference Table,  Columns, and Advanced.

Here is a brief outline of the purpose of each tab.

  1. General
    • SourceName :- In our case it is Lookup Error Output
    • DestinationName :- In our case it is OLE DB Destination Input
  2. Metadata
    • Summary for the columns
      • Column Name
      • Datatype
      • Length
      • Source Component

 

General

DataFlowPathEditor-CallLogDW-General

Metadata

DataFlowPathEditor-CallLogDW-Metadata

 

 

 

OLE DB Destination Editor

The OLE DB Destination Editor has 3 tabs. And, those are the Connection Manager, Columns, and Error Output.

Here is a brief outline of the purpose of each tab.

  1. Connection Manger
    • Reference or shall we say a pointer to one of the Connection Managers defined for the Project
    • Data Access mode
      • Our choices are
        • Table or view ( Fast Load )
        • Name of the table or view
        • SQL
      • In our case it is LTEDRKV.callLogDW
  2. Columns
    • Available External Columns
      • All the Columns defined in the External Columns are listed
      • Optimize performance by unchecking uneeded columns
      • In our case we choose all the columns ( callID,  phoneNumber, dateAdded, phoneNumberCountryCode )
  3. Error Output
    • Exception Handling Block
      • Data read from Source / External into Internal Storage
        • If truncation error occurs, how should it be handled
          • Fail?

 

 

Connection Manager

ConnectionManager

 

Mappings

Map each Source Column unto the Destination Column

Mappings

 

 

Data Flow Path Editor

dataflowtask-calllog

Summary

Our sample SSIS Project is pretty simple.

We retrieve transactional data from callLog table and join that table against our Lookup table.

We attempt a join against our Lookup table.

When data is not find, we insert the missing rows into the callLogDW.

 

Reference

  1. Lookup Transformation
    • Integration Services > Data Flow > Integration Services Transformations
      Link
  2. SSIS Team Blog
    • Matt Masson
      • Lookup Pattern: Case Insensitive
        Link
      • Lookup – Using the cache connection manager
        Link

SQL Server Reporting Services [SSRS] – Error – Null is not declared. ‘Null’ constants is no longer supported; use ‘System.DBNull’ instead

Background

It is a little bit of work to use null-able arguments in Sql Server Reporting Services (SSRS).

 

In DataSet Properties, Tried using “=(Null)”

For the dataset, when we set the parameter value to =Null

DatasetProperties-Parameters-(Null)

 

 Textual:

Error - Null is not declared.  'Null' constants is no longer supported; use 'System.DBNull' instead
Image:

NullConstantIsNoLongerSupported

 

In Dataset Properties, tried using “System.DBNull”

In Dataset properties, When we set the parameter value to =System.DBNull

DatasetProperties-Parameters-System.DBNull

Text
Error: [BC30691] 'DBNull' is a type in 'System' and cannot be used as an expression.

Image:
DBNullIsATypeInSystemAndCannotBeUsedInAnExpression

In Dataset Properties, tried using “=Nothing”

Tried setting the parameter value to:

  • =Nothing
  • =System.Convert.DBNull

 

DatasetProperties-Parameters-Nothing

 

We get an error.

Error

In Designer, when trying to preview the report.

Textual:

One or more parameters required to run the report have not been specified

 

Image:

OneOrMoreParametersRequiredToRunTheReportHaveNotBeenSpecified


 

Report Parameter – Default Value – Setting Parameter Value

  • Access the parameter
  • Access the “Default Values” panel
  • In the “Default Values” panel
    • In “Select from one of the following options” radio group, choose the “Specify values” option
    • In “Value” set “(Null)

ReportParameter-DefaultValues-SpecifyValues-Value-(Null)

 

Does not help

Dataset Properties – Report Parameter

Let us remove the parameter

Original:

Here are the dual parameters.

DatasetProperties-Parameter-Null-Included

Revised:

We have removed the second parameter.

DatasetProperties-Parameter-Null-Excluded

Error Message:

OneOrMoreParametersRequiredToRunTheReportHaveNotBeenSpecified (2nd Parameter removed)

Report Parameter – Set Value to Empty

Let us go set the parameter’s value to “” or System.String.Empty.

DatasetProperties-Parameter-SetToEmptyString

Things are good.

SQL Server Profiler

Here is what we see in SQL Server Profiler:

SQLServerProfiler-ParameterValueToEmpty

Transact SQL

In Transact SQL, make sure of the following:

    • The arguments is nullable — (variable [datatype] = null )
    • That you have same code line for when the variable is null and empty

 Sample Code:


if object_id('[dbo].[usp_city]') is null
begin

    exec('create procedure [dbo].[usp_city] as select 1/0 as [shell] ')

end
go

alter procedure [dbo].[usp_city]
(
      -- make sure @countryCode is nullable
      @countryCode varchar(255) = null

     --make sure that @country is nullable
    , @country     varchar(255) = null
)
as

    select
              vc.country
            , vc.countryCode
            , vc.city

    from   [dbo].[vw_city] vc

    where  (

                (
                    vc.countryCode
                                = case
                                        when (@countryCode is null) then vc.countryCode
                                        when (@countryCode = '') then vc.countryCode
                                        else @countryCode
                                  end
                )

                and
                (
                    vc.country
                                = case
                                        when (@country is null) then vc.country
                                        when (@country = '') then vc.country
                                        else @country
                                  end
                )

           ) 

    order by
                  vc.country asc
                , vc.city

go

 

Conclusion

It appears that SSRS does not really support the setting of parameters to Null, but it supports empty string.

 

References

SSRS

Technical: Microsoft – SQL Server – v2008/R2 – Business Intelligence Studio – Cube Browsing – Error dragging measures\attributes to view pane

Background

When Browsing Cube, one of the errors one might encounter as you add measures and attributes to the pane is one that reads:


Retrieving the COM class factory for component with CLSID 
{C966837E-970C-4F4F-A91D-D81E21ABBBDB} failed due to the following error: 80040154. 
(Microsoft Visual Studio)

Full Error Message

Error Text


===================================

Retrieving the COM class factory for component with CLSID {C966837E-970C-4F4F-A91D-D81E21ABBBDB} failed due to the following error: 80040154. (Microsoft Visual Studio)

------------------------------
Program Location:

   at Microsoft.AnalysisServices.Controls.MiscUtilities.GetMixedDataObjectForFilterAndPivotTable(DataObject dataObjectForFilter, PivotTableDataObject pivotTableDataObject)
   at Microsoft.AnalysisServices.Controls.PivotTableBoundMetadataBrowser.GetDataObject(TreeNode node)
   at Microsoft.AnalysisServices.Controls.MetadataTreeView.OnItemDrag(ItemDragEventArgs e)
   at System.Windows.Forms.TreeView.TvnBeginDrag(MouseButtons buttons, NMTREEVIEW* nmtv)
   at System.Windows.Forms.TreeView.WmNotify(Message& m)
   at System.Windows.Forms.TreeView.WndProc(Message& m)
   at Microsoft.AnalysisServices.Controls.MetadataTreeView.WndProc(Message& msg)
   at Microsoft.AnalysisServices.Browse.CubeBrowser.CubeBrowserMetadataTreeView.WndProc(Message& msg)



Error Image

 

RetrievingTheComClassFactory

 

 

Resolution

There are correlating errors on the .Net and prominent fixes includes:

Repair / Install Microsoft Office 2003 Web Components

 

The OWC track did not work for us.

Inspect MS Windows registry

Searched the Windows registry for C966837E-970C-4F4F-A91D-D81E21ABBBDB and found the stub, but could not find CLASS (CLS) branch.

VSIntegrationNativeHelpers

 

 

Tried fiddling with MS Windows Registry

Tried fiddling with MS Windows Registry, but could not find good starting data on the Internet; note that data has to match SQL Server Version, etc.

Install latest Product patch

Downloaded and Installed MS SQL Server v2008/R2 SP2.

BTW, patch is available @ http://www.microsoft.com/en-us/download/details.aspx?id=30437

 

Possible Workaround

Registry Tweaking

BTW, if you want to try to tweak your registry and see if that might work and you avoid installing\re-installing a Service Pack, here is a working version:



Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{C966837E-970C-4f4f-A91D-D81E21ABBBDB}]
@="MarshalledToIStreamDataObject Class"
"AppID"="{B2463DC8-B3FA-4BEC-945E-60219DCC6FD8}"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{C966837E-970C-4f4f-A91D-D81E21ABBBDB}\InprocServer32]
@="c:\\Program Files (x86)\\Microsoft SQL Server\\100\\Tools\\Bin\\Microsoft.DataWarehouse.VsIntegration.Helpers.dll"
"ThreadingModel"="Apartment"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{C966837E-970C-4f4f-A91D-D81E21ABBBDB}\ProgID]
@="VsIntergrationNativeHelpers.Marshalle.2"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{C966837E-970C-4f4f-A91D-D81E21ABBBDB}\TypeLib]
@="{84F2933D-1F4E-43D8-9006-372E64998B36}"



Please keep in mind that the registry entry pasted above is for MS Windows 2008/R2 SP2. Other versions of SQL Server will invariable have version specific entries.

 

Listening To

Listening to Brett Eldredge – Raymond (Video) [http://www.youtube.com/watch?v=txCUwSKo1kg]

 

 

Technical: Microsoft – SQL Server – Analysis Services / BIDS – Error – Message – DegenerateMeasureGroupDimension [xxx] : Granularity is not defined.

Technical: Microsoft – SQL Server – Analysis Services / BIDS – Error – Message – DegenerateMeasureGroupDimension <Cube> : Granularity is not defined.

This error is covered very sparsely. It refers to a problem that can arise from removing\renaming dimensions or attributes.

To correct remove the referenced cube and re-create it.

SQL Server – Analysis Services / BIDS – Error – Message – Dimension [xxx] : No key attribute is defined

Introduction

Mistakenly removed an Attribute, now when I try to build the SSAS project, I am getting an error message stating: “Dimension [xxx]: No Key attribute is defined”.

Background

Ordinarily the message should not be difficult to fix, but the Business Intelligence Development (BIDS) environment does not allow use to set the Dimension’s key.

Set-Table Dimension Properties

Interestingly enough, BIDS hides and shows various settable attributes based on which Dimensions is selected.

Dimension Properties – Time

DimensionTime

Dimension Properties – Fact

DimensionFactUsage

Dimension Properties

From the screen shots above, we can quickly see that based on which Dimension we have selected, the properties we can set varies.

Here is a quick summary as to which properties are enabled.

Item Name Item Value Explanation
AttributeAllMemberName
Collation
CurrentStorageMode Molap
DependsOnDimension Specifies the dimension that this Dimension depends on
ErrorConfiguration
ID
Language
Name
ProactiveCaching ROLAP/HOLAP/MOLAP
ProcessingGroup ByAttribute
ByTable
ProcessingMode Regular
LazyAggregations
ProcessingPriority
ProcessingRecommendation
ProcessingState
Source
StorageMode Molap Molap
Rolap
InMemory
Type Regular
Time
Geography
Organization
BillofMaterials
Accounts
Customers
Products
Scenario
Quantitative
Utility
UnknownMember Visible
Hidden
None
AutomaticNull
UnknownMemberName If not specified, the default value is “Unknown” is used
WriteEnabled False False
True

Fix

From our list above, we see that none of the properties contain the Key setting.

So how do we set a Dimension’s key:

  • Access the “Solution Explorer” panel
  • Navigate to the Dimensions tree
  • Choose the Dimension
  • Double-click on your selection
  • The Dimension Structure panel is activated
  • Access the “Dimension Structure” tab
  • Select the specific Attribute that you should be the Key Attribute
  • Right click on your selection
  • And, from the drop-down menu select “Set Attribute Usage”\Key

By the way, the options available from the “Set Attribute Usage” option are:

  • Regular
  • Key
  • Parent

DimensionKeyAttribute

Crediting

Crediting Vinuthan from Aditi Technologies

Happy

Happy that I am able to capture drop-down menu options from the free version of WinSnap.

Free version of WinSpan ( 1.1.10) available @ Link

Documentation available @ Link.

References

  • Error 1 Dimension [Product] : No key attribute is defined
    Link

 

Microsoft – SQL Server – Integration Services – Designer – Derived Column

Microsoft – SQL Server – Integration Services – Designer – Derived Column

IsNull

Checks whether a column is null, if so replaces it with something more ammenable

Sample

  1. ISNULL(hrs_client_dob) ? (DT_DBTIMESTAMP)”1753-01-01 00:00:00.000″ hrs_client_dob   (SQLServerCentral.com forum <manmaria> –> http://www.tek-tips.com/viewthread.cfm?qid=1283075&page=1)